Rolling bearing life prediction method based on secondary attention mechanism and improved time convolution network

Abstract

The application discloses a rolling bearing life prediction method based on a secondary attention mechanism Transformer encoding layer and an improved time convolution network, acquires bearing full-life vibration signals, and constructs a multi-domain and multi-class normalized degradation feature set; subsequently, the secondary attention mechanism Transformer encoding layer is used to acquire global information of a rolling bearing life process; then, an improved time convolution network with one-dimensional convolution layers in series with different scales is constructed to acquire key local information of a rolling bearing full-life stage, and a smoothing process is performed to obtain a final life prediction result of the rolling bearing. The application can effectively capture the overall change trend of the bearing health state in the form of self-iteration of the attention mechanism, and extract local time features of different scales by the improved time convolution network, so that a more comprehensive time feature expression of the rolling bearing health state is obtained, and the life prediction result precision of the rolling bearing is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of bearing life prediction technology, and in particular relates to a method for predicting the life of rolling bearings based on a Transformer encoding layer with a secondary attention mechanism and an improved temporal convolutional network. Background Technology

[0002] Rolling bearings, as key components of rotating machinery, are widely used in various industrial equipment. Due to the complex working conditions and harsh environments in industrial settings, the performance degradation of rolling bearings can impact equipment production safety and factory economic efficiency, and even cause personnel safety issues. Therefore, accurately predicting bearing life and replacing bearings with warning signs in a timely manner can not only reduce equipment maintenance downtime but also improve equipment reliability and factory production efficiency. Research in this area has become a key challenge and important topic in the engineering field.

[0003] Methods for predicting the remaining useful life (RUL) of rolling bearings are mainly divided into model-driven and data-driven approaches. Model-driven methods suffer from insufficient generalization due to the complexity of actual working conditions and the high randomness of industrial environments, and require a certain level of expert experience, thus lacking broad applicability. Data-driven methods, on the other hand, utilize machine learning to process the complex nonlinear mapping relationship between bearing condition signals and RUL, offering greater flexibility and automation. However, current classic deep learning networks, such as convolutional neural networks and recurrent neural networks and their variants, are not suitable for long-sequence life prediction tasks like rolling bearing fatigue due to their inherent limitations.

[0004] Transformer and TCN are popular models in bearing life prediction in recent years. Transformer excels at handling complex long-sequence tasks with good global performance, while TCN effectively captures local dependencies in sequence data and offers more stable gradient propagation. Combining the advantages of both Transformer and TCN allows for the simultaneous acquisition of global and local features of rolling bearings, improving prediction accuracy. However, the attention mechanism in the Transformer's encoding layer is still somewhat lacking in mining long-term data dependencies, and the causal and dilated convolutional models of TCN have excessively large parameter counts, affecting computational efficiency. In summary, this invention employs a secondary attention mechanism in the Transformer encoding layer and an improved temporal convolutional network, which not only mines deep-level features but also simplifies the network structure and reduces computational resource requirements. Summary of the Invention

[0005] To address the shortcomings of existing deep learning models in bearing life prediction, this invention proposes a rolling bearing life prediction method based on a Transformer encoding layer with a secondary attention mechanism and an improved temporal convolutional network, in order to achieve more efficient and accurate prediction of the remaining life of bearings.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for predicting the life of rolling bearings based on a Transformer encoding layer with a secondary attention mechanism and an improved temporal convolutional network, comprising the following steps: S1. Collect vibration signals from multiple bearings throughout their entire life cycle and divide them into training and test sets; S2. Extract the time domain, frequency domain, time-frequency domain, and entropy features of the whole life cycle vibration signal to form a multi-domain and multi-class feature set; S3. Normalize the multi-domain, multi-class feature sets to obtain the rolling bearing degradation feature set; S4. Construct a Transformer encoding layer with a secondary attention mechanism. The first layer uses residual attention to capture the basic features of the rolling bearing state. The second layer also uses a series structure of residual attention to form a self-iterative attention mechanism to further extract deep state features. S5. An improved temporal convolutional network is constructed by connecting two convolutional layers of different scales to extract local features of the vibration signal at different time scales. The local features at different scales are then fused. The output of step S4 is then used as the input of the improved temporal convolutional network to obtain the initial prediction result of the rolling bearing life. S6. Train the model built above using the divided training set, then use the test set to predict the rolling bearing life, and finally smooth the output results of the proposed model to obtain the final rolling bearing life prediction result.

[0007] The secondary attention mechanism of the Transformer encoding layer in S4 is as follows: it uses a two-layer serially connected residual self-attention mechanism, and adds a Batchnorm layer before each self-attention layer to perform batch normalization of the data to accelerate training and improve model stability. Each self-attention layer is followed by an activation function ELU for non-linear activation.

[0008] The improved temporal convolutional network of S5 consists of a one-dimensional Same convolutional layer with two convolutional kernels of different scales. The Same convolutional layer is followed by a Layernorm layer, and the last Layernorm layer is followed by an activation function ReLU for non-linear activation.

[0009] The present invention has the following beneficial effects: The bearing life prediction method provided in this invention uses a two-layer serially connected residual self-attention mechanism to extract the global dependencies of bearing vibration data. This mechanism enables self-iteration to a certain extent, effectively improving the model's feature extraction capability over long time series. Furthermore, it innovatively combines this method with an improved temporal convolutional network, using stacked convolutional layers with different kernel sizes to extract temporal features at different scales. This captures features across different time ranges in the sequence data, making the network more sensitive to local dependencies within the sequence data. Combining the advantages of both methods, the model possesses powerful local feature extraction and global awareness capabilities, enabling more efficient and accurate bearing life prediction. Attached Figure Description

[0010] Figure 1 Flowchart for predicting the life of rolling bearings; Figure 2 This is a platform schematic diagram of the dataset used in this invention; Figure 3 The prediction results for Experiment 1, Bearing1_4, in the examples are shown. Figure 4 The prediction results are for Experiment 2, Bearing2_4, in the examples. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Example 1 refer to Figure 1 A rolling bearing life prediction method based on a secondary attention mechanism Transformer encoding layer and an improved temporal convolutional network includes the following steps: Step 1: Testing was conducted using the IEEE PHM 2012 public dataset. Two DYTRAN3035B miniature accelerometers were mounted on the PRONOSTIA experimental platform, radially positioned at 90° to each other on the outer ring of the bearing. The test platform was as follows: Figure 2 As shown in the diagram. Two accelerometers sample the bearing vibration signal at a frequency of 25.6 kHz, sampling once every 10 seconds, with each sample acquiring 0.1 seconds of data. The test bench and sensor arrangement are as follows. Figure 2 As shown, each bearing subset contains vibration signals from two channels, horizontal and vertical. In this example, the horizontal vibration signal is selected for analysis.

[0013] The model was validated using data from working conditions 1 and 2. In working condition 1, Bearing1_3 and Bearing1_7 were selected as the training set and Bearing1_4 as the test set. In working condition 2, Bearing2_7 was selected as the training set and Bearing2_4 as the test set.

[0014] The specific training and testing bearings are shown in the table below:

[0015] Step 2: Extract 19 commonly used time-domain features and 7 frequency-domain features. The time-frequency domain features are the energy values ​​of 8 frequency bands generated by the three-level wavelet packet decomposition of the signal. Also extract 8 entropy features (Shannon entropy, sample entropy, approximate entropy, permutation entropy, fuzzy entropy, Renyi entropy, Tsallis entropy, and LogEnergy entropy). After feature extraction from samples obtained at each stage of the entire lifecycle, a multi-domain, multi-class feature set is obtained.

[0016] The time-domain characteristics are as follows, where: This represents the total number of sampling points. For the first The signal sequence obtained from the sampling.

[0017]

[0018] The frequency domain characteristics are as follows, where: This represents the total number of sampling points. For signal frequency, This represents the spectral amplitude.

[0019]

[0020] Step 3: Normalize the feature set and map the features to the [0,1] interval to eliminate the influence of the units, so as to facilitate data analysis and processing, thereby improving the convergence speed of the deep learning model and avoiding the problem of gradient explosion during the prediction process. The normalization formula is as follows: In the formula: The maximum value of this feature. This is the minimum value of the feature. These are the normalized eigenvalues; The remaining service life of a bearing is expressed as a percentage, and the calculation formula is as follows: In the formula: For the current moment, The sample label at the current moment. This is the time of failure.

[0021] Step 4: Connect the Transformer encoding layer with secondary attention mechanism, the improved temporal convolutional network and the regression layer in sequence to build a bearing life prediction model. Train the bearing life prediction model with the training set to obtain the trained bearing life prediction model. Then input the test set to predict the bearing life. Finally, smooth the model output to obtain the bearing life prediction result.

[0022] Specifically, the secondary attention mechanism Transformer encoding layer uses a two-layer serially connected residual self-attention mechanism, and adds a Batchnorm layer before each self-attention layer to batch normalize the data to accelerate training and improve model stability. Each self-attention layer is followed by an activation function ELU for non-linear activation.

[0023] It should be noted that the improved temporal convolutional network is a one-dimensional Same convolutional layer with two convolutional kernels of different scales. The Same convolutional layer is followed by a Layernorm layer, and the last Layernorm layer is followed by a ReLU activation function for non-linear activation.

[0024] Prediction results are as follows Figure 3 and Figure 4 As shown in the figure, the present invention can accurately predict bearing life. The predicted life is basically consistent with the actual life trend, and the life fitting effect is very close, reflecting the degradation trend of the bearing well. It also matches the bearing at the end of its life well, accurately predicting the bearing life, ensuring the bearing utilization rate, reducing the cost of use, and avoiding the risk of bearing failure due to failure to provide timely warning. It is evident that the present invention has the advantages of high prediction accuracy and strong generalization, and has significant practical application value.

[0025] The details of the model structure proposed in this invention are as follows: (1) Input layer This layer is responsible for inputting the previously extracted multi-domain feature set of vibration data into the model. The shape of the input data is (BatchSize, feature length, feature dimension).

[0026] (2) Position encoding layer Positionembed Vibration signals exhibit multi-domain features with temporal sequence characteristics, but models often overlook this temporal order during processing. To address this issue, a position encoding layer adds temporal location information to each time step, helping the model better capture temporal features. The position encoding weight matrix has a size of [missing information]. .

[0027] The purpose of location encoding is to add location information to each time step to preserve the time sequence. The calculation formula is as follows:

[0028]

[0029] In the formula: Indicates the position of the feature vector. The dimension representing the location encoding. Indicates the dimensionality of the feature.

[0030] (3) Addition layer The main function of the additive layer is to superimpose input data with positional encoding information to incorporate positional information into sequence modeling. One of the core features of the Transformer model is the removal of the recurrent structure of traditional RNNs, replacing it with a global self-attention mechanism to process sequence data. However, the self-attention mechanism essentially performs global computation on the contextual information of the entire sequence, failing to directly identify the temporal or positional information of individual elements within the sequence. Therefore, to ensure the model can effectively capture the positional features of each time step in the sequence, explicit positional information must be injected as a supplement.

[0031] (4) Batch Normalization This layer performs batch normalization on the input data, reducing variations in data distribution and thus accelerating model convergence and improving generalization ability. Each batch of data undergoes normalization processing by this layer. Batch normalization (BN) standardizes the input data to reduce feature distribution shifts and improve training stability. The formula for calculating BN is:

[0032]

[0033] In the formula, It is the input feature vector. yes The mean, yes standard deviation The mini-batch size refers to the number of samples processed simultaneously during each training iteration. It is a loop variable used to iterate over each sample in the mini-batch in the formula. To prevent division by zero, the default value is usually 0.001; and These are trainable parameters, representing the scaling factor and the offset factor, respectively.

[0034] (5) Self-Attention Mechanism The self-attention mechanism is the core module of this method, used to capture global dependencies in time series data. In the Transformer architecture, the self-attention mechanism allows the model to establish direct connections between any two positions in the sequence, thus capturing long-distance dependencies. A second self-attention layer is added after the original self-attention layer. This allows the model to deepen feature extraction; the first self-attention layer captures basic features, and the second layer further extracts deeper features based on this. The second self-attention layer can refine the output of the first layer, strengthening key features while suppressing less important features. This more effectively captures and models the complex dynamic behavior of bearings under different operating conditions. The two-layer self-attention mechanism enables the model to better adapt to different data distributions and task requirements, thereby improving the model's generalization ability and enabling it to cope with complex operating conditions in actual industrial production.

[0035] The self-attention mechanism is the core module used to capture global dependencies in time series data. Query, Key, and Value are the three core concepts of self-attention, forming the foundation of the mechanism. The weight matrices for the self-attention module are Query, Key, and Value, with a matrix size of [matrix size missing]. Query represents the query vector at the current time step. Key represents the feature representations of all time steps in the time series. Value represents the feature value at the corresponding time step. Self-attention calculates the attention weights by computing the inner product of Query and Key, and then uses these weights to sum the values ​​in a weighted manner, thereby generating a feature representation containing global information. Multi-head attention further enhances the model's ability to learn different features. The self-attention mechanism is the core module used to capture global dependencies between time steps.

[0036] The input feature matrix is The Query, Key, and Value matrices are generated through three sets of linear transformations: In the formula: This is a trainable parameter matrix.

[0037] Attention weights are calculated using the dot product of the query and the key: In the formula: for The dimensions of the matrix, to prevent If the value is too large, the value of Softmax will be very small, causing the gradient vanishing problem. Softmax converts similarity scores into weights.

[0038] To capture attention for different features, a multi-head mechanism is used for parallel computation: In the formula: For the number of attention heads, To output the transformation matrix, This represents the first in the multi-head attention mechanism. Size.

[0039] (6) ELU activation function ExponentialLinearUnit The advantage of ELU lies in its smoother handling of negative values, thus reducing the gradient vanishing problem and enhancing the model's nonlinear fitting ability. The calculation formula is shown below:

[0040] In the formula, These are the values ​​that need to be processed by the activation function. It is a hyperparameter used to control the shape of the function within the negative interval. In practical applications, It is usually set to 1.

[0041] (7) One-dimensional convolutional layer One of the best techniques for extracting local information is convolution, which extracts local abstract features within a small receptive field. This method uses two one-dimensional Same convolutional layers with different kernels to extract and fuse features at different scales of the bearing vibration signal. One-dimensional convolution is a common convolutional operation, and in this study it is used to process features previously extracted by the self-attention module, extracting and fusing features at different temporal scales. This allows the model to more effectively learn the complex nonlinear relationship between the bearing vibration signal and its lifespan.

[0042] One-dimensional Same convolutions offer more efficient computation when processing sequential data. Same convolutional layers allow for flexible control of the receptive field by adjusting the kernel size and stride, while maintaining low computational cost. This design deepens the network without increasing computational complexity, introducing more non-linearity and enhancing the model's expressive power. Using two Same convolutional layers simplifies the model structure, providing greater flexibility. The kernel size and stride can be independently adjusted to adapt to different sequence lengths and feature extraction requirements. This simplified structure helps reduce model complexity, lowers the risk of overfitting, and improves generalization ability. By stacking two Same convolutional layers, the model can extract higher-level features layer by layer and fuse them, thus enhancing the richness of feature representation. This makes the model more valuable for bearing life prediction tasks. Especially when processing complex vibration signal data, the model can more effectively capture and utilize patterns in the data, thereby improving prediction accuracy.

[0043] The core of one-dimensional convolution is to extract features from a sequence using a one-dimensional sliding window. The calculation formula is shown below: In the formula: For input signal, For convolution kernel, This represents the convolution operation. It is the integration variable, used for shifting in the convolution operation. This is the result of the convolution operation, i.e., the features extracted by the convolution operation.

[0044] (8) Layer Normalization As the number of layers in a neural network increases, updates to the parameters of each layer can alter the distribution of input data in subsequent layers. This phenomenon is called Internal Covariate Shift (ICS). ICS can make network training more difficult because each layer needs to continuously adapt to the new input distribution. Layer Normalization (LN) normalizes the input of each layer, reducing the impact of changes in input distribution between layers and effectively mitigating the effects of ICS. The formula for calculating LN is as follows:

[0045]

[0046] In the formula, It is the input feature vector. yes The mean, yes standard deviation The mini-batch size refers to the number of samples processed simultaneously during each training iteration. It is a loop variable used to iterate over each sample in the mini-batch in the formula. To prevent small values ​​from being divided by zero, the default value is usually 0.001. and are trainable parameters, representing the scaling factor and the offset factor, respectively.

[0047] (9) ReLU activation function Rectified Linear Unit ReLU is one of the most commonly used activation functions. ReLU introduces non-linear characteristics by setting the negative part of the input value to 0 while keeping the positive part unchanged. Therefore, ReLU is simpler to calculate and can effectively alleviate the gradient vanishing problem. Its calculation formula is as follows: In the formula, It is a value that needs to be processed by the activation function.

[0048] (10) Fully Connected Layer The fully connected layer integrates the features extracted earlier, then performs a linear transformation through the weight matrix, and finally the regression layer directly maps the output of the fully connected layer to the predicted value of bearing life.

[0049] This invention proposes a rolling bearing life prediction method based on a Transformer encoding layer with a secondary attention mechanism and an improved temporal convolutional network. By fusing the Transformer encoding layer with a secondary attention mechanism and the improved temporal convolutional network, the accuracy of bearing life prediction is improved. Finally, the method was tested on the PHM2012 public dataset to further verify its reliability. The following conclusions are drawn: (1) The underlying network extracts global features from the time series through a self-attention mechanism. Through the self-iteration of the self-attention mechanism, feature extraction continues based on the previous feature extraction results, which can effectively capture the long-term trend of the time series and provide a comprehensive temporal feature expression for subsequent models.

[0050] (2) The top-level network uses an improved temporal convolutional network to capture local temporal features, extract features at different temporal scales and fuse them, thereby enhancing the model’s feature extraction capabilities.

[0051] (3) The interconnected structure of the upper and lower layers of the network can further enhance the predictive ability of the model. The lower layer network is responsible for extracting multi-level time series features, while the upper layer network uses these features to model complex global dependencies and dynamic patterns. The two achieve a division of labor and cooperation in the processing of time series data, which not only preserves local details but also captures the overall trend of change, thereby significantly improving the accuracy of bearing life prediction.

[0052] (4) Through experiments on the PHM2012 public dataset, the life prediction results of the method proposed in this invention are consistent with the actual life curve and match well at the end of the bearing life, ensuring the bearing utilization rate, reducing the cost of use, and avoiding the risk caused by the failure of bearing due to the inability to provide timely warning.

Claims

1. A method for predicting the life of rolling bearings based on a secondary attention mechanism and an improved temporal convolutional network, characterized in that... Includes the following steps: S1. Collect vibration signals from multiple bearings throughout their entire life cycle and divide them into training and test sets; S2. Extract the time domain, frequency domain, time-frequency domain, and entropy features of the whole life cycle vibration signal to form a multi-domain and multi-class feature set; S3. Normalize the multi-domain, multi-class feature sets to obtain the rolling bearing degradation feature set; S4. Using a two-layer serially connected residual self-attention mechanism, a Transformer encoding layer with a secondary attention mechanism is constructed. The first layer of residual attention captures the basic features of the rolling bearing state, and the second layer also adopts a serial structure of residual attention to form a self-iterative attention mechanism to further extract deep state features. S5. An improved temporal convolutional network is constructed by connecting two convolutional layers of different scales to extract local features of the vibration signal at different time scales. The local features at different scales are then fused. The output of step S4 is then used as the input of the improved temporal convolutional network to obtain the initial prediction result of the rolling bearing life. S6. Train the model built above with the divided training set, then use the test set to predict the rolling bearing life, and finally smooth the output result of the proposed model to obtain the final rolling bearing life prediction result. The secondary attention mechanism Transformer encoding layer of S4 is specifically as follows: a Batchnorm layer is added before each self-attention layer to perform batch normalization of the data, and an activation function ELU is added after each self-attention layer for non-linear activation. The improved temporal convolutional network of S5 consists of a one-dimensional Same convolutional layer with two convolutional kernels of different scales. The Same convolutional layer is followed by a Layernorm layer, and the last Layernorm layer is followed by an activation function ReLU for non-linear activation.

Images