Unsupervised model vibration anomaly detection method based on sparse self-attention and VAE

Abstract

The present application belongs to the technical field of intelligent detection, and discloses an unsupervised model vibration anomaly detection method based on sparse self-attention and VAE, comprising the following steps: S1, obtaining vibration data and performing pretreatment; S2, constructing a VAE-Informer model; S3, training the VAE-Informer model; S4, using the trained VAE-Informer model for fault anomaly detection of vibration data; the VAE-Informer model comprises an encoder and a decoder, the encoder maps input data to a latent variable in a latent space, and the decoding process maps the latent variable to the reconstruction of the original data; the VAE-Informer model uses an Informer unit instead of a general neural unit in VAE. The present application has stronger long-time dependence capturing capability in sequence data, and thus takes into account the distribution characteristics and timing characteristics in data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent detection, and particularly relates to an unsupervised model vibration anomaly detection method based on sparse self-attention and VAE. BACKGROUND

[0002] Common fault detection methods mainly include model-based, experience-based and data-based methods. The model-based method needs to calculate the model of the data generated by the equipment during operation, which depends on the mathematical model set by humans. However, the actual industrial equipment operation is very complex, and it is almost impossible to completely consider the related interference factors. Therefore, it is very difficult to establish an accurate mathematical model.

[0003] The experience-based method depends on the experience and knowledge of experts in the field to be detected, which requires high personnel requirements and may have subjective judgment problems. In addition, relying on manual work will lead to a lack of automation and intelligence, making it difficult to detect large-scale systems or real-time data.

[0004] Disadvantages of the prior art data-based method: such methods mine the implicit information in the historical operation data of the equipment, generally analyze the vibration signal to determine the state of the equipment operation. Such methods do not need to establish a mathematical model and do not depend on the experience of professionals, and can be applied to relatively complex systems. At present, they are widely used, especially various machine learning and deep learning methods. However, machine learning is difficult to fit complex models, and the fitting ability of deep learning is better. However, the prior art has insufficient ability to capture long-time dependence in sequence data, and cannot balance the distribution characteristics and time sequence characteristics in vibration data, resulting in the need for further improvement of the actual fault detection effect. SUMMARY

[0005] The application aims to provide an unsupervised model vibration anomaly detection method based on sparse self-attention and VAE, thereby solving the technical problem that the prior art has insufficient ability to capture long-time dependence in sequence data and cannot balance the distribution characteristics and time sequence characteristics in vibration data.

[0006] The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE comprises the following steps.

[0007] S1, obtaining vibration data and performing preprocessing.

[0008] S2, constructing a VAE-Informer model.

[0009] S3, training the VAE-Informer model.

[0010] S4. Use the trained VAE-Informer model for fault and anomaly detection in vibration data.

[0011] The VAE-Informer model includes an encoder and a decoder. The encoder maps the input data to latent variables in a latent space, while the decoding process maps the latent variables to the reconstruction of the original data. The VAE-Informer model uses Informer units instead of ordinary neural units in VAE. The Informer units use convolution and sparse self-attention mechanisms from the encoding part of the Informer model to process the input data. The sparse self-attention mechanism reduces the distance between two data points to a constant, capturing long-term dependencies in sequence data.

[0012] Preferably, step three includes...

[0013] S3.1 Obtain the features of the model input data.

[0014] S3.2 The encoder uses multi-head sparse self-attention to map the input to latent variables.

[0015] S3.3 The decoder uses multi-head sparse self-attention to map latent variables to reconstructed data.

[0016] Step S3 follows the previous process steps S3.1 to S3.3. The VAE-Informer model is trained using normal data. A set of data is input to obtain the reconstructed output. The difference between the reconstructed data and the original data is calculated on the test set to obtain the error range [error_min, error_max]. However, abnormal data will not be close to normal data after model reconstruction. Therefore, when the reconstruction error is greater than the maximum error error_max, it is considered that an anomaly has occurred.

[0017] Preferably, in step one, vibration data is collected over a period of time, and then the data is divided into m sequences of length n. After standardization, an input matrix of shape (m, n) is formed, and the input matrix is ​​used as the model input data.

[0018] Preferably, step S3.1 specifically includes:

[0019] S3.1.1. Use d one-dimensional convolutions with kernel size 3 and stride 1 to extract the feature information X from the model input data. c Feature information X c Represented as: X c =Conv(X).

[0020] S3.1.2. Perform position encoding on the model input data to obtain the data's position information X. p Location information Xp is represented as:

[0021]

[0022] where p ij The calculation method of p i is in the range of [0, m), j is in the range of [0, d), m is the number of sequences in the model input data, and d represents the dimension of the position information and is the same as the number of convolution kernels used.

[0023] S3.1.3, the position information X p and the feature information X c are added to obtain the complete data feature The dimension of the data feature is (m, n, d), and is represented as:

[0024] Preferably, in the calculation of each head in step S3.2, q i represents the i-th row vector of the query Q, and the corresponding sparsity measure is represented as:

[0025]

[0026] where the first term is the logarithmic sum exponential function LSE (Log-Sum-Exp) of the row vector q i on all keys K, and the second term is the arithmetic mean of the row vector q i , represents the transpose of the j-th row vector of the key K, L k indicates the number of rows of the key K.

[0027] Preferably, in the calculation of each head in step S3.2, the maximum mean measure is used to calculate the sparsity measure of the i-th row vector q i of the query Q, which is represented as:

[0028]

[0029] represents the transpose of the j-th row vector of the key K, L k indicates the number of rows of the key K.

[0030] Preferably, the calculation formula of the output A is:

[0031]

[0032] where only contains the u largest values of the sparsity measure M(q, K) of the query, K T represents the transpose of the key K.

[0033] Preferably, in the step S3.2.4, the output A is transformed into an output data with a dimension of (m, n, d) after d-dimensional transformation, the convolution is a convolution with a step size of 1, a kernel size of 3, padding of 1, and an activation function of ELU, the maximum pooling processing is performed with a pooling kernel of 3, a step size of 2, and padding of 1; in the step S3.2.5, after repeating the steps S3.2.2 to S3.2.4 once, the dimension of the output data is (m, n / 4, d).

[0034] Preferably, in the step S3.3, in the decoding process, one-dimensional convolution is used to extract data information with a kernel size of 3, a step size of 1, and padding of 1, then multi-head sparse self-attention is calculated, the multi-head sparse self-attention is re-transformed into d dimensions through a linear layer and an ELU activation function, then a convolution with a step size of 1, a kernel size of 3, and an activation function of ELU is used, and then a maximum pooling processing with a pooling kernel of 3, a step size of 2, and padding of 1 is performed, the above decoding method is repeated once again, and finally the reconstruction data X' of the input data is obtained through a linear layer, and the dimension of the reconstruction data X' is (m, n).

[0035] The present application has the following advantages: on the one hand, the present application is based on the idea of the self-encoder and the variational inference in the VAE model, the input data is mapped to a latent variable in a latent space through an encoder, and the decoding process maps the latent variable to the reconstruction of the original data. In the training, the distribution of the normal data is learned, the difference of the abnormal data is captured in the process of reconstructing the abnormal data, and whether the data is abnormal is judged by the reconstruction error or the degree of abnormality in the latent space. On the other hand, the present application uses an Informer unit instead of a normal neural unit in the VAE, uses a sparse self-attention mechanism to reduce the distance between two position data to a constant, and has a stronger ability to capture long-time dependence in sequence data, and thus takes into account the distribution characteristics and time sequence characteristics of the data. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the unsupervised model vibration anomaly detection method based on sparse self-attention and VAE of the present application. DETAILED DESCRIPTION

[0037] The specific embodiments of the present application will be further described below with reference to the drawings, and the description of the embodiments will help those skilled in the art to have a more complete, accurate and comprehensive understanding of the inventive concept and technical solutions of the present application.

[0038] As Figure 1As shown, the present application provides an unsupervised model vibration anomaly detection method based on sparse self-attention and VAE, including the following steps.

[0039] S1, acquiring vibration data and preprocessing.

[0040] This step collects vibration data within a period of time, then divides the data into m sequences with a length of n, standardizes the sequences to form an input matrix with a shape of (m, n), and takes the input matrix as model input data.

[0041] S2, constructing a VAE-Informer model.

[0042] The VAE (Variational Autoencoder, Variational Autoencoder) model is a generative model that uses the ideas of autoencoder and variational inference. The entire model consists of two parts: an encoder and a decoder. The encoder maps the input data to a latent variable in a latent space (i.e., a latent variable), and the decoding process maps the latent variable to a reconstruction of the original data. The VAE model performs fault anomaly detection based on variational inference and reconstruction of data. It can learn the distribution of normal data, capture the differences of anomalies in the process of reconstructing abnormal data, and judge whether the data is abnormal by the reconstruction error or the degree of abnormality in the latent space.

[0043] The Informer model can be used for long time series processing. It can capture long-term dependencies in sequence data, and unlike RNN-related algorithms, it uses a sparse self-attention (ProSparse self-attention) mechanism to reduce the distance between two position data to a constant, and has stronger ability to capture long-term dependencies in sequence data. In this scheme, the VAE-Informer model uses an Informer unit instead of a normal neural unit in VAE, thereby combining the advantages of VAE and Informer models, and taking into account the distribution characteristics and time sequence characteristics of the data. The Informer unit uses the convolution and sparse self-attention mechanism in the encoding part of the Informer model to process the input data. The structure of the VAE-Informer model is as follows Figure 1 .

[0044] S3, training the VAE-Informer model. The specific process is as follows.

[0045] S3.1, obtaining the features of the model input data.

[0046] S3.1.1, using a 1D convolution with d convolution kernels of size 3 and step 1 to extract feature information X c from the model input data, where X c is represented as: Xc = Conv (X).

[0047] S3.1.2, position encoding is performed on the model input data to obtain position information X of the data p , the position information X p is expressed as:

[0048]

[0049] where p ij is calculated as i is in the range of [0, m), j is in the range of [0, d), m is the number of sequences in the model input data, d represents the dimension of the position information, and is a specified value, which is the same as the number of convolution kernels used.

[0050] S3.1.3, the position information X p and the feature information X c are added to obtain complete data features The dimension of the data features is (m, n, d), and is expressed as:

[0051] S3.2, the encoder uses multi-head sparse self-attention to map the input to a latent variable.

[0052] The calculation method of the multi-head sparse self-attention is as follows.

[0053] S3.2.1, obtain query Q, key K and value V, in terms of data, query Q, key K and value V are linear transformations of data features , and data features are the input of multi-head sparse self-attention, and the corresponding query Q, key K and value V are expressed as

[0054] S3.2.2, divide the obtained query Q, key K and value V into h groups, discard the redundant data, and then perform h-head multi-head self-attention calculation. In each head calculation, q i represents the i-th row of query Q, i.e. the i-th row vector of query Q, and the corresponding sparsity measure is expressed as:

[0055]

[0056] where the first term is the logarithmic sum exponential function LSE (Log-Sum-Exp) of the row vector q i on all keys K, and the second term is the arithmetic mean of the row vector q i , and represents the transpose of the j-th row vector of key K, and L kthe number of rows of the key K (i.e., the number of row vectors k j If the sparsity measure M on the i-th query Q is larger, its attention probability p will be more diversified and more likely to contain more important dot product pairs in the long-tail self-attention distribution.

[0057] In actual use, to reduce the calculation, the method uses the maximum mean measure As an alternative to M(q i , K), the sparsity measure is calculated as follows:

[0058]

[0059] q i represents the i-th row of the query Q, i.e., the i-th row vector of the query Q, represents the transpose of the j-th row vector of the key K, L k the number of rows of the key K (i.e., the number of row vectors k j ).

[0060] S3.2.3, each key only focuses on u main queries Q to obtain sparse self-attention, and the output A is obtained by the following calculation:

[0061]

[0062] where only contains the u queries with the largest value of the sparsity measure M(q, K), K T represents the transpose of the key K.

[0063] After that, the mapping to the latent variable is completed based on the calculation results of the multi-head sparse self-attention, including the following steps.

[0064] S3.2.4, the obtained output A is transformed into d dimensions through a linear layer and an ELU activation function, at this time the output data dimension is (m, n, d), then a convolution is used with a step of 1, a convolution kernel size of 3, a padding of 1, and an ELU activation function, and then a maximum pooling with a pooling kernel of 3, a step of 2, and a padding of 1 is used.

[0065] S3.2.5, repeat steps S3.2.2 to S3.2.4 once, at this time the output data dimension is (m, n / 4, d), then the latent variable Z of the VAE model is obtained by changing through a linear layer and adding random noise of the standard normal distribution, and the data dimension is (m, k), at this time the encoding process of the VAE model is completed.

[0066] S3.3, the decoder uses multi-head sparse self-attention to map the latent variable to reconstructed data.

[0067] In the decoding process, the data obtained by the encoding process is first improved in dimension by a linear layer to (m, n). The decoding method comprises: using a one-dimensional convolution with a convolution kernel size of 3, a step of 1, and padding of 1 to extract data information, then calculating multi-head sparse self-attention, and then transforming the multi-head sparse self-attention into d dimensions through a linear layer and an ELU activation function. Then, a convolution with a step of 1, a convolution kernel size of 3, and an ELU activation function is used. Then, a maximum pooling with a pooling kernel of 3, a step of 2, and padding of 1 is used. The above decoding method is repeated once again. Finally, the reconstructed data X' of the input data is obtained through a linear layer, and the dimension is (m, n).

[0068] Step S3 uses normal data to train the VAE-Informer model according to the previous process steps S3.1 to S3.3. A set of data is input to obtain the reconstructed output. Since the data used is normal data, the normal data reconstructed by the model will be close to the normal data. The difference between the reconstructed data and the original data is calculated on the test set to obtain the error range [error_min, error_max]. When the data set is large enough, the reconstruction error should be within this range when normal data is input. Abnormal data reconstructed by the model will not be close to normal data, so when the reconstruction error is greater than the maximum error error_max, it is considered that an anomaly has occurred.

[0069] S4, using the trained VAE-Informer model for fault anomaly detection of vibration data.

[0070] In actual application, the collected vibration data is preprocessed according to step one and input into the trained VAE-Informer model to output the fault anomaly detection result.

[0071] The above describes the present application in conjunction with the drawings, and it is obvious that the specific implementation of the present application is not limited by the above method. Any non-essential improvement or direct application of the inventive concept and technical solution to other occasions is within the scope of protection of the present application.

Claims

1. An unsupervised model vibration anomaly detection method based on sparse self-attention and VAE, characterized in that: The method comprises the following steps: S1, acquiring vibration data and preprocessing; S2, constructing a VAE-Informer model; S3, training the VAE-Informer model; S4, using the trained VAE-Informer model for fault anomaly detection of vibration data; The VAE-Informer model comprises an encoder and a decoder, the encoder maps input data to a latent variable in a latent space, and the decoding process maps the latent variable to the reconstruction of the original data; the VAE-Informer model uses an Informer unit instead of a normal neural unit in VAE, the Informer unit uses convolution and sparse self-attention mechanism in the encoding part of the Informer model to process the input data, and uses the sparse self-attention mechanism to reduce the distance between two position data to a constant, and capture the long-time dependence in the sequence data.

2. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 1, characterized in that: The step S3 comprises: S3.1, obtaining the features of the model input data; S3.2, the encoder uses multi-head sparse self-attention to map the input to a latent variable; S3.3, the decoder uses multi-head sparse self-attention to map the latent variable to reconstructed data; Step S3 uses normal data to train the VAE-Informer model according to the previous process steps S3.1 to S3.3, inputs a group of data to obtain the reconstructed output, calculates the difference between the reconstructed data and the original data on the test set, and obtains the error range [error_min, error_max], and the abnormal data cannot be close to the normal data after being reconstructed by the model, so when the reconstruction error is greater than the maximum error error_max, it is considered that an abnormality has occurred.

3. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 2, characterized in that: The step S1 collects vibration data within a period of time, then divides the data into m sequences with a length of n, and forms an input matrix with a shape of (m, n) after standardization, and takes the input matrix as the model input data.

4. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 3, characterized in that: The step S3.1 specifically comprises: S3.1.1, using d convolution kernels of 1-dimensional convolution with size 3 and step 1 to extract feature information X of the model input data c , feature information X c is expressed as: X c = Conv(X); S3.1.2, position encoding is performed on the model input data to obtain position information X of the data p , the position information X p is expressed as: wherein p ij The calculation method is i is in the range of [0, m), j is in the range of [0, d), m is the number of sequences in the model input data, and d represents the dimension of the position information and is the same as the number of convolution kernels used. S3.1.3, add position information X p and feature information X c to obtain complete data features The dimension of data features is (m, n, d), and is expressed as:

5. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 4, characterized in that: In the step S3.2, the calculation method of multi-head sparse self-attention comprises: S3.2.1, obtaining a query Q, a key K and a value V; S3.2.2, dividing the obtained query Q, key K and value V into h groups, discarding the remaining data, and then performing h-head multi-head self-attention calculation, and calculating the i-th query sparse metric in each head; S3.2.3, each key only focuses on u main queries to obtain sparse self-attention, and calculates the output A; S3.2.4, the obtained output A is transformed into d dimensions again through a linear layer and an ELU activation function, and then convolution and maximum pooling processing are performed again; S3.2.5, repeating steps S3.2.2 to S3.2.4 once, and then changing through a linear layer and adding random noise of standard normal distribution to obtain the latent variable Z of the VAE model.

6. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 5, characterized in that: In the calculation of each head in the step S3.2, q i The i-th row vector of the query Q is represented, and the corresponding sparsity measure is represented as: where the first term is the row vector q i the second term is the arithmetic mean of the row vectors q i , denotes the transpose of the j-th row vector of the key K, L k denotes the number of rows of the key K.

7. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 5, characterized in that: In the calculation of each head, in the step S3.2, the maximum mean measure is used The sparsity measure of the i-th row vector q of the query Q is calculated, denoted as: i The sparsity measure of the i-th row vector q of the query Q is calculated, denoted as: denotes the transpose of the j-th column vector of the key K, L k denotes the number of rows of the key K.

8. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 6 or 7, characterized in that: The calculation formula of the output A is: wherein Only include the u queries with the largest values of sparsity metric M(q, K), K T denotes the transpose of key K.

9. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 8, characterized in that: In the step S3.2.4, the output data dimension of the output A after the d-dimensional conversion is (m, n, d), the convolution is a convolution with a step size of 1, a kernel size of 3, padding of 1, and an ELU activation function, and the maximum pooling processing is performed with a pooling kernel of 3, a step size of 2, and padding of 1; in the step S3.2.5, after the steps S3.2.2 to S3.2.4 are repeated once, the output data dimension is (m, n / 4, d).

10. The unsupervised model vibration anomaly detection method based on sparse self-attention and VAE according to claim 5, characterized in that: In the step S3.3, in the decoding process, one-dimensional convolution is used to extract data information with d convolution kernels with a kernel size of 3, a step size of 1, and padding of 1, then multi-head sparse self-attention is calculated, the multi-head sparse self-attention is re-converted to d dimensions through a linear layer and an ELU activation function, then a convolution with a step size of 1, a kernel size of 3, and an ELU activation function is used, and then the maximum pooling processing is performed with a pooling kernel of 3, a step size of 2, and padding of 1, the above decoding method is repeated once, and finally the reconstruction data X' of the input data is obtained through a linear layer, and the reconstruction data X' dimension is (m, n).

Images