A method for industrial process fault detection based on RATransformer
By combining the RATransformer model with an autoencoder structure, the problem of integrating unsupervised learning and deep learning is solved, enabling efficient feature extraction and real-time monitoring of industrial process fault detection, and improving fault detection performance and model generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2023-11-27
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to effectively combine unsupervised learning with deep learning backbone networks to improve the performance of industrial process fault detection. Furthermore, traditional methods are prone to overfitting when there is insufficient labeled data, making it difficult to extract deep process features.
We employ the RATransformer model, combining linear transformation, multi-head attention mechanism, and residual connections with an autoencoder structure for unsupervised learning to construct an industrial process fault detection method. We utilize the Adam optimizer for gradient descent to construct fault detection thresholds and achieve real-time monitoring.
It improves the feature extraction capability for industrial process fault detection, solves the overfitting problem, enhances the model's generalization ability and learning efficiency, and realizes online real-time fault monitoring.
Smart Images
Figure CN117668451B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of fault detection technology, and in particular relates to an industrial process fault detection method based on RATransformer. Background Technology
[0002] The increasing complexity of modern process testing has led to nonlinear and unstable data collected from many processes. Therefore, effective fault detection methods are crucial for maintaining the smooth operation of industrial processes. The application of digitalization has made process data increasingly available, resulting in the collection of vast amounts of unlabeled historical data. Traditional deep learning methods are generally supervised learning methods, highly dependent on data labels. Unsupervised learning methods can use unlabeled data, but their fault detection performance is mainly affected by the backbone network structure. How to combine unsupervised learning with more effective deep learning backbone networks to improve the performance of industrial process fault detection methods is a pressing issue that needs to be addressed.
[0003] Traditional methods for process fault detection are multivariate statistical process control, mainly including principal component analysis (PCA), partial least squares (PLS), independent component analysis (ICA), and canonical component analysis (CCOM). These methods generally place high demands on the assumptions about the input data distribution and struggle to learn complex process features. Machine learning methods relax these assumptions and mainly include support vector machines (SVMs), random forests, artificial neural networks (ANNs), and Gaussian mixture methods. These methods have simple structures but struggle to extract sufficiently abstract process features. Traditional deep learning methods can extract sufficiently complex features, mainly including feedforward neural networks (FNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs). However, these methods are generally supervised learning methods and are not suitable for situations with insufficient labeled data. Autoencoder unsupervised learning methods are a type of deep learning, mainly including autoencoders, variational autoencoders, denoising autoencoders, and stacked autoencoders. These methods primarily use feedforward neural networks, which are prone to gradient minimization and gradient explosion problems.
[0004] Current fault detection methods based on unsupervised deep learning are mostly based on autoencoder structures, which improve the feature extraction capabilities of neural networks by combining them with other backbone neural network structures. These methods mainly include adversarial autoencoders and one-dimensional convolutional autoencoders. These methods are highly complex and prone to overfitting. Transformer models have achieved remarkable results in natural language processing, machine vision, and other fields, but due to the difficulty in matching their structure with autoencoders, there is a lack of effective methods to combine transformers and autoencoders. Summary of the Invention
[0005] The purpose of this invention is to provide a method for detecting industrial process faults based on RATransformer, characterized by the following steps:
[0006] S1: Collect industrial process data under normal operating conditions and preprocess the data;
[0007] S2: Establish a deep learning network and train the model using fault-free data through unsupervised learning.
[0008] S3: Construct fault detection thresholds using the trained model and training data;
[0009] S4: Real-time online monitoring is achieved through real-time inference from the model and existing thresholds.
[0010] Furthermore, in S1, the maximum-minimum normalization method is used to preprocess the process data collected under normal conditions.
[0011] Furthermore, in S1, the maximum-minimum normalization method is specifically used as follows: first, the training dataset is normalized, and then the extreme values of each variable are retained and the data is preprocessed using the online monitoring data.
[0012] Furthermore, in S2, the training process of the deep learning network specifically includes the following steps:
[0013] S21: Using linear transformation, each input variable is converted into vector data suitable for the transformer model;
[0014] S22: The extracted features are gradually abstracted through multiple transformer modules, and then dimensionality is reduced by using convolutional layers with transformer kernels and max pooling layers to achieve unsupervised learning based on the autoencoder structure.
[0015] S23: Stack multiple transformer modules in each layer of the encoder to increase the feature dimension, thereby reconstructing the input variables. By introducing residual connections between the encoder and decoder, the efficiency of model learning and the generalization ability of the model are improved.
[0016] S24: Use the reconstruction error as the loss function and apply gradient descent to the model parameters using the Adam optimizer.
[0017] Furthermore, in S21, if the input variable is set to x, then the linear transformation of the input variable x into a vector is expressed as:
[0018]
[0019] Where LE stands for linear transformation layer; x iLet i be the i-th input variable; Let r be the coefficient vector used for the i-th linear transformation; e This is the feature matrix output after the linear transformation; by making the linear transformations corresponding to different variables relatively independent, the corresponding process variable information is preserved.
[0020] Furthermore, in S22, bias matrices of different sizes are added to different heads of the multi-head attention mechanism. The i-th head of the modified multi-head attention mechanism is represented as:
[0021]
[0022] in, and These are the coefficients for the query, key, and value of the current attention head, respectively; d k The characteristic matrix r e The k-th feature dimension; b is the bias added to the attention mechanism; h is the number of heads in the attention mechanism; M is the bias coefficient for the i-th head, used to control the magnitude of the bias's influence on that head; i Let be the feature matrix output by the i-th head. Within the same transformer module, since the bias matrices added to different heads are the same, the advantages of transformer and autoencoder can be integrated by controlling the magnitude of the bias coefficients on different heads.
[0023] Furthermore, in S23, the residual connection between the encoder and decoder is implemented through the attention mechanism, which has multiple inputs. The transformer module is represented as follows:
[0024] output = TB(query, key, value)
[0025] TB stands for Transformer module. By using the features extracted from the intermediate layer of the decoder as key and value inputs to the Transformer module in the decoder, and then using the output of the previous layer in the decoder as query, residual connections between features of different dimensions can be realized, thereby accelerating the learning speed of the model and improving the generalization ability of the model.
[0026] Furthermore, in S24, the reconstructed variable of the model output is set as... The loss function is then expressed as:
[0027]
[0028] The training speed of the model is accelerated by using the Adam optimizer, which has an adaptive learning rate adjustment function, to adjust the model parameters in a gradient descent manner.
[0029] Furthermore, in S3, the construction of the fault detection threshold is as follows: First, the squared prediction error of all offline training data after model inference is calculated. Then, the distribution of the squared prediction error is obtained through the kernel density estimation method. Finally, the 99% confidence interval is used as the fault detection threshold.
[0030] Furthermore, S4 specifically involves: inputting the data obtained from online monitoring into the trained model, and the squared error between the predicted value and the input value is the squared prediction error; if the current squared prediction error exceeds the threshold obtained during the training process, it is determined that a fault has occurred in the current process.
[0031] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in:
[0032] 1. This invention considers the integration of transformer and autoencoder to analyze industrial process data and constructs an unsupervised learning method for industrial process fault detection based on transformer. Compared with traditional autoencoder-based methods, it has great advantages, has adaptive feature extraction capability with multi-head attention, and can be well applied to industrial process scenarios.
[0033] 2. This invention uses the transformer method to solve the problem of difficulty in extracting sufficiently deep process features.
[0034] 3. This invention solves the problem of overfitting in transformer models when the training sample size is insufficient by using a residual connection method based on a multi-head attention mechanism. Attached Figure Description
[0035] Figure 1 This is a flowchart of the unsupervised fault detection method for industrial processes according to the present invention.
[0036] Figure 2 This is a schematic diagram of the network structure of the deep learning network of the present invention. Detailed Implementation
[0037] The following will describe in more detail an industrial process fault detection method based on RATransformer according to the present invention with reference to the schematic diagram, which illustrates the preferred embodiments of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the present invention.
[0038] like Figure 1 As shown, an industrial process fault detection method based on RATransformer includes the following steps:
[0039] Step 1: Collect industrial process data under normal operating conditions and preprocess it to balance the weights of different signals. Each data sample in the industrial process data contains 32 observations. 500 samples are used as the training dataset. The preprocessing process is as follows: first, normalize the training dataset; then, retain the extreme values of each variable and perform data preprocessing using the online monitoring data.
[0040] Note: Under normal operating conditions, i.e., no faults (product quality is not a problem, and the process is running smoothly).
[0041] Step 2: Build a deep learning network and train the model using fault-free data in an unsupervised learning manner.
[0042] like Figure 2 As shown, the deep learning network includes two linear transformation layers, an encoder consisting of four transformer modules, and a decoder consisting of four transformer modules connected to the encoder twice via residual connections. Specifically, the linear transformation layers contain multiple independent linear transformations. The encoder includes two multi-transformer modules, one adjustable transformer module, and a max-pooling layer with transformer kernels. The decoder includes four multi-transformer modules. Finally, the Adam optimizer is used to complete the gradient descent process.
[0043] The training process of RATransformer includes the following steps:
[0044] Step 1): Using linear transformations, each input variable is converted into vector data suitable for the transformer model. This process is equivalent to replacing the semantic embedding process in a regular transformer model, saving the process variable information into the corresponding linear transformation. The linear transformation that converts the input variable x into a vector can be expressed as:
[0045]
[0046] Where LE stands for linear transformation layer; x i Let i be the i-th input variable (there are m process variables in total); Let r be the coefficient vector used for the i-th linear transformation; e This is the feature matrix output after the linear transformation; by making the linear transformations corresponding to different variables relatively independent, the corresponding process variable information is preserved.
[0047] Step 2): The extracted features are gradually abstracted through multiple transformer modules, and then feature dimensionality reduction is performed using convolutional layers with transformer kernels and max pooling layers, achieving unsupervised learning based on an autoencoder structure. Bias matrices of different sizes are added to different heads of the multi-head attention mechanism. After modification, the i-th head of the multi-head attention mechanism is represented as:
[0048]
[0049] in, and These are the coefficients for the query, key, and value of the current attention head, respectively; d k The characteristic matrix r e The k-th feature dimension; b is the bias added to the attention mechanism; h is the number of heads in the attention mechanism; M is the bias coefficient for the i-th head, used to control the magnitude of the bias's influence on that head; i This is the feature matrix output by the i-th head. Within the same transformer module, the bias matrices added to different heads are the same. By controlling the magnitude of the bias coefficients on different heads, the advantages of transformer and autoencoder can be integrated.
[0050] Step 3): Stack multiple transformer modules in each layer of the decoder to increase the feature dimensionality and reconstruct the input variables. Residual connections are introduced between the encoder and decoder to improve model learning efficiency and generalization ability. These residual connections can be achieved through the attention mechanism's ability to handle multiple inputs. The transformer module can be represented as:
[0051] output = TB(query, key, value)
[0052] In this module, TB stands for Transformer. It takes the features extracted from the intermediate layer of the encoder as the key and value and inputs them into the Transformer module of the decoder. Then, it uses the output of the previous layer in the decoder as the query, which can realize residual connections between features of different dimensions, accelerate the learning speed of the model and improve the generalization ability of the model.
[0053] Step 4): Using the reconstruction error as the loss function, apply gradient descent to the model parameters using the Adam optimizer. The reconstructed variable output by the model is... The loss function can be expressed as:
[0054]
[0055] By using the Adam optimizer, which has an adaptive learning rate adjustment function, to adjust the model's parameters using gradient descent, the training speed of the model can be accelerated.
[0056] Step 3: Construct fault detection thresholds using the trained model and training data.
[0057] The fault detection metric is the squared prediction error. The specific construction process is as follows: First, calculate the squared prediction error of all offline training data after model inference; then, obtain the distribution of the squared prediction error using kernel density estimation; finally, use the 99% confidence interval as the fault detection threshold.
[0058] Step 4: Achieve real-time online monitoring through real-time inference from the model and existing thresholds.
[0059] The currently monitored process signal is input into the trained model for inference. The reconstruction error of the inference result is compared with the threshold obtained during training to determine whether a fault has occurred, thus realizing online fault detection.
[0060] The specific judgment principle is as follows: if the current squared prediction error exceeds the threshold obtained during the training process, it is judged that the current process has failed; otherwise, if the current squared prediction error is within the threshold obtained during the training process, the current process has not failed.
[0061] The function and effect of the industrial process fault detection method based on RATransformer of this invention are as follows:
[0062] (1) Using transformer as the backbone network, more abstract and effective process features were extracted, which improved the fault detection effect.
[0063] (2) The use of an autoencoder structure improves the learning efficiency of the model and enhances its generalization ability.
[0064] (3) The residual connection method based on transformer structure is used to expand the scope of application of residual connection and strengthen the connection between the model decoding part and process features.
[0065] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A method for detecting industrial process faults based on RATransformer, characterized in that, Includes the following steps: S1: Collect industrial process data under normal operating conditions and preprocess the data; S2: Establish a deep learning network and train the model using fault-free data through unsupervised learning. S3: Construct fault detection thresholds using the trained model and training data; S4: Real-time online monitoring is achieved through real-time inference from the model and existing thresholds; In S2, the training process of the deep learning network specifically includes the following steps: S21: Using linear transformation, each input variable is converted into vector data suitable for the transformer model; S22: The extracted features are gradually abstracted through multiple transformer modules, and then dimensionality is reduced by using convolutional layers with transformer kernels and max pooling layers to achieve unsupervised learning based on the autoencoder structure. S23: Stack multiple transformer modules in each layer of the encoder to increase the feature dimension, thereby reconstructing the input variables. By introducing residual connections between the encoder and decoder, the efficiency of model learning is improved and the generalization ability of the model is enhanced. S24: Use the reconstruction error as the loss function and apply gradient descent to the model parameters using the Adam optimizer; In step S22, bias matrices of different sizes are added to different heads of the multi-head attention mechanism. After the change, the multi-head attention mechanism's first... The size is represented as: ； in, , and These are the current attention heads. , and The coefficient; Characteristic matrix The Each feature dimension; It is a bias added to the attention mechanism; The number of heads in this attention mechanism; For the first The offset coefficient of the head is used to control the magnitude of the effect of the offset on the head. For the first The feature matrix output by each head; within the same transformer module, since the bias matrix added to different heads is the same, the advantages of transformer and autoencoder are integrated by controlling the size of the bias coefficients on different heads; In step S23, the residual connection between the encoder and decoder is achieved through the attention mechanism, which has multiple inputs. The transformer module is represented as follows: ； in, For the transformer module, the features extracted from the intermediate layers of the decoder are used as... and The input is fed into the transformer module in the decoder, and then the output of the previous layer in the decoder is used as... This enables residual connections between features of different dimensions, thereby accelerating the model's learning speed and improving its generalization ability.
2. The industrial process fault detection method based on RATransformer according to claim 1, characterized in that, In step S1, the maximum-minimum normalization method is used to preprocess the process data collected under normal conditions.
3. The industrial process fault detection method based on RATransformer according to claim 2, characterized in that, In S1, the maximum-minimum normalization method is specifically used as follows: first, the training dataset is normalized, and then the extreme values of each variable are retained and the data is preprocessed using the online monitoring data.
4. The industrial process fault detection method based on RATransformer according to claim 1, characterized in that, In step S21, the input variable is set as follows: Then the input variable The linear transformation into a vector is expressed as: ； in, It is a linear transformation layer; For the first One input variable; For use in the first The coefficient vector of a linear transformation; This is the feature matrix output after the linear transformation; by making the linear transformations corresponding to different variables relatively independent, the corresponding process variable information is preserved.
5. The industrial process fault detection method based on RATransformer according to claim 1, characterized in that, In step S24, the reconstructed variable output by the model is set as follows: The loss function is then expressed as: ； The training speed of the model is accelerated by using the Adam optimizer, which has an adaptive learning rate adjustment function, to adjust the model parameters in a gradient descent manner.
6. The industrial process fault detection method based on RATransformer according to claim 1, characterized in that, In S3, the construction of the fault detection threshold is specifically as follows: first, the squared prediction error of all offline training data after model inference is calculated; then, the distribution of the squared prediction error is obtained through the kernel density estimation method; and finally, the 99% confidence interval is used as the fault detection threshold.
7. The industrial process fault detection method based on RATransformer according to claim 1, characterized in that, Specifically, S4 involves inputting the data obtained from online monitoring into the trained model, and the squared error between the predicted value and the input value is the squared prediction error; when the current squared prediction error exceeds the threshold obtained during the training process, it is determined that a fault has occurred in the current process.