Industrial equipment abnormality detection method and related device

Through the two-stage Transformer model and the causal multi-head self-attention mechanism, the industrial equipment data is preprocessed and abnormal scores and thresholds are calculated, which solves the problem of low accuracy of traditional detection technology, and realizes accurate detection and confrontation samples of industrial equipment anomalies the robustness.

CN116757533BActive Publication Date: 2024-11-15WUXI INNOVATION CENT CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310698676.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-13
Publication Date
2024-11-15
Estimated Expiration
2043-06-13

AI Technical Summary

Technical Problem

Traditional industrial equipment abnormality detection technology has low detection accuracy, making it difficult to effectively monitor the operating status of the equipment throughout its life cycle, especially when obtaining deterministic abnormal data.

Method used

The input matrix is ​​preprocessed by a two-stage Transformer model, and the reconstruction output is obtained through the encoder and decoder processing, and the abnormal score is calculated based on the reconstruction error, and the abnormal threshold is determined to determine whether the device is abnormal, combined with the causal multi-head self-attention mechanism and position coding to improve the robustness of the counter samples.

Benefits of technology

Accurate detection of industrial equipment abnormalities is achieved, robust expression ability of countermeasures is improved, and equipment abnormalities can be effectively identified and detection efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116757533B_ABST
    Figure CN116757533B_ABST
Patent Text Reader

Abstract

The present application discloses an industrial equipment anomaly detection method and related devices, which relate to the field of industrial equipment monitoring technology, including extracting questionable data and constructing an input matrix based on the questionable data; preprocessing the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix and the attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; the matrix obtained after preprocessing is processed by the encoder and decoder of the Transformer model to obtain the reconstruction output of the first stage and the second stage; the reconstruction error is calculated based on the reconstruction output of the first stage and the second stage, and the anomaly score is determined based on the reconstruction error; the anomaly threshold is determined, and the detection result is obtained based on the anomaly score and the anomaly threshold. The method can accurately detect whether the industrial equipment is abnormal, and has a strong robust expression ability for adversarial samples.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of industrial equipment monitoring, and in particular to an industrial equipment anomaly detection method; and also to an industrial equipment anomaly detection device, equipment and computer-readable storage medium. Background Art

[0002] Industrial equipment has the characteristics of long operation time, high operation and maintenance cost, and low operation and maintenance efficiency. It is particularly important to effectively monitor the operation status of industrial equipment throughout its life cycle and detect potential anomalies in a timely manner. Since it is difficult to obtain industrial data with deterministic anomalies in the industrial production process, anomaly detection technology based on unsupervised learning is becoming increasingly popular. The detection accuracy of traditional anomaly detection technology is not high. Therefore, providing a technical solution that can accurately detect anomalies in industrial equipment has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0003] The purpose of this application is to provide an industrial equipment anomaly detection method that can accurately detect whether the industrial equipment is abnormal and has strong robust expression capabilities for adversarial samples. Another purpose of this application is to provide an industrial equipment anomaly detection device, equipment, and computer-readable storage medium, all of which have the above technical effects.

[0004] To solve the above technical problems, the present application provides an industrial equipment anomaly detection method, comprising:

[0005] Extracting questionable data and constructing an input matrix based on the questionable data;

[0006] Preprocessing the input matrix through a two-stage Transformer model; wherein preprocessing the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage;

[0007] Processing the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage;

[0008] A reconstruction error is calculated based on the reconstruction outputs of the first stage and the second stage, and an anomaly score is determined based on the reconstruction error;

[0009] An abnormality threshold is determined, and a detection result is obtained according to the abnormality score and the abnormality threshold.

[0010] Optionally, the preprocessing the input matrix by using a two-stage Transformer model further includes:

[0011] Performing sliding window processing on the time series to obtain multiple window matrices; the time series includes multiple input matrices;

[0012] After merging the window matrix and the attention matrix, position encoding is performed on the merged matrix to obtain a position matrix, so as to process the position matrix through the Transformer model.

[0013] Optionally, the calculating the reconstruction error according to the reconstruction outputs of the first stage and the second stage includes:

[0014] According to the reconstruction output of the first stage and the reconstruction output of the second stage, the reconstruction error of the multivariate feature and the reconstruction error of the univariate feature are calculated respectively.

[0015] Optionally, determining an anomaly score according to the reconstruction error includes:

[0016] determining an anomaly score of the multivariate feature based on a reconstruction error of the multivariate feature;

[0017] An anomaly score of the unary feature is determined based on a reconstruction error of the unary feature.

[0018] Optionally, determining the abnormal threshold includes:

[0019] According to the abnormal score and extreme value index of the univariate feature, the abnormal threshold of the univariate feature is estimated by the POT model;

[0020] The abnormal threshold of the multi-element feature is calculated based on the abnormal threshold of each of the uni-element features.

[0021] Optionally, obtaining a detection result according to the anomaly score and the anomaly threshold includes:

[0022] If the anomaly score of the multi-feature exceeds the anomaly threshold of the multi-feature, the device is determined to be abnormal.

[0023] Optionally, the Transformer model adopts a causal multi-head self-attention mechanism, and the calculation context matrix includes:

[0024] The odd query vectors of the query matrix are sequentially multiplied by the odd-numbered vectors of the transposed key matrix to obtain the first type of data of the odd-numbered rows of the attention score matrix, and the even query vectors of the query matrix are sequentially multiplied by the even-numbered items of the transposed key matrix to obtain the second type of data of the even-numbered rows of the attention score matrix, and the second type of data in each row of the attention score matrix is ​​set to negative infinity until the query vectors of the query matrix are traversed to obtain the attention score matrix;

[0025] Normalizing the non-negative infinite items in the attention score matrix and setting the negative infinite items to zero to obtain a weight matrix;

[0026] The weight matrix is ​​multiplied by the value matrix to obtain the context matrix.

[0027] In order to solve the above technical problems, the present application also provides an industrial equipment abnormality detection device, comprising:

[0028] An extraction module, used to extract questionable data and construct an input matrix based on the questionable data;

[0029] A preprocessing module, configured to preprocess the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; and the attention matrix of the first stage is different from the attention matrix of the second stage;

[0030] A reconstruction module, used to process the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage;

[0031] A calculation module, configured to calculate a reconstruction error according to the reconstruction outputs of the first stage and the second stage, and determine an anomaly score according to the reconstruction error;

[0032] The determination module is used to determine an abnormality threshold and obtain a detection result according to the abnormality score and the abnormality threshold.

[0033] In order to solve the above technical problems, the present application also provides an industrial equipment abnormality detection device, including:

[0034] Memory for storing computer programs;

[0035] A processor is used to implement the steps of the above-mentioned industrial equipment abnormality detection method when executing the computer program.

[0036] In order to solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the industrial equipment anomaly detection method as described above are implemented.

[0037] The industrial equipment anomaly detection method provided in the present application includes: extracting questionable data and constructing an input matrix based on the questionable data; preprocessing the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; processing the matrix obtained after preprocessing through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage; calculating the reconstruction error based on the reconstructed output of the first stage and the second stage, and determining the anomaly score based on the reconstruction error; determining the anomaly threshold, and obtaining the detection result based on the anomaly score and the anomaly threshold.

[0038] It can be seen that the industrial equipment anomaly detection method provided by this application reconstructs the suspicious data through a two-stage Transformer model, and determines whether the equipment is abnormal based on the reconstruction error, so that it can accurately detect whether the equipment is abnormal. In addition, this application adopts a two-stage adversarial training, which can effectively improve the model's robust expression ability for adversarial samples.

[0039] The industrial equipment anomaly detection device, equipment, and computer-readable storage medium provided in this application all have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the prior art and the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of an industrial equipment anomaly detection method provided in an embodiment of the present application;

[0042] Figure 2 A block diagram of an industrial equipment anomaly detection implementation provided by an embodiment of the present application;

[0043] Figure 3 A schematic diagram of multi-head attention provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a fast approximate multi-head attention calculation provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of an industrial equipment abnormality detection device provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of an industrial equipment anomaly detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The core of this application is to provide an industrial equipment anomaly detection method that can accurately detect whether the industrial equipment is abnormal and has strong robust expression capabilities for adversarial samples. Another core of this application is to provide an industrial equipment anomaly detection device, equipment and computer-readable storage medium, all of which have the above technical effects.

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] Please refer to Figure 1 , Figure 1 A flowchart of an industrial equipment abnormality detection method provided in an embodiment of the present application is provided. Figure 1 As shown, the method includes:

[0050] S101: extracting questionable data, and constructing an input matrix according to the questionable data;

[0051] In order to reduce unnecessary redundant calculations and improve detection efficiency, the data that may be abnormal in the collected industrial time series data can be identified and extracted based on the prior knowledge base. The data that may be abnormal is the suspicious data. Furthermore, the suspicious data can also be normalized. The input matrix is ​​constructed based on the normalized suspicious data. The input matrix is ​​expressed as follows:

[0052] The input matrix includes M features or variables.

[0053] S102: preprocessing the input matrix through a two-stage Transformer model; wherein preprocessing the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage;

[0054] The traditional Encoder-Decoder model has difficulty in obtaining short-term trends and will miss some anomalies with smaller deviations. In addition, during the adversarial training process, the samples will be mixed with some small disturbances in order to allow the neural network to adapt to such disturbances, thereby being robust to adversarial samples. For this reason, this embodiment introduces adversarial training to create a two-stage L-depth Transformer model, which is used as a reconstruction model to reconstruct the time series input. The Transformer model can be reconstructed and trained in advance using normal data to learn the long- and short-term dependencies and multi-feature correlations between normal data, so that the Transformer model can be used to find abnormal events that deviate greatly from normal data and cannot be effectively reconstructed.

[0055] In the adversarial training process, in order to enable the Transformer model to quickly achieve self-adjustment, this embodiment defines an attention matrix and uses the attention matrix for preprocessing. The attention matrix is ​​expressed as follows:

[0056]

[0057] Where F is the K×M-dimensional attention matrix, abs(·) represents the absolute value of the elements in the matrix, and O 1,t , O 2,t They represent the reconstructed outputs of decoder 1 and decoder 2 of stage I at the tth time step respectively.

[0058] Combination Figure 2 As shown, in some embodiments, the preprocessing of the input matrix by the two-stage Transformer model further includes:

[0059] Performing sliding window processing on the time series to obtain multiple window matrices; the time series includes multiple input matrices;

[0060] After merging the window matrix and the attention matrix, position encoding is performed on the merged matrix to obtain a position matrix, so as to process the position matrix through the Transformer model.

[0061] As the name implies, the dual-stage includes two stages, namely the first stage and the second stage. The difference between the first stage and the second stage is only that the focus matrix is ​​different during preprocessing. Therefore, the first stage (stage I) is used as an example for explanation below.

[0062] In order to improve the robustness of the reconstruction model, the time series is processed by sliding window to obtain T window matrices. The time series is represented as: X = {x1, x2, ..., x t ,...,x T} T, where T is the length of the time series, that is, the number of input matrices contained. The window matrix is ​​expressed as: Where W t ={x t-K+1 ,x t-K ,...,x t} T , t∈[1,T], where W t is a K×M dimensional window matrix, where K is an adjustable window length. In particular, for W t For example, when t<K, use x t To fill.

[0063] Sliding window processing consumes a lot of storage overhead, and the shorter the window length, the more memory it consumes. For real-time stream processing scenarios, sliding window processing will affect the online real-time detection effect to a certain extent, and may even lag behind. In addition, the abnormal state of the device has a certain persistence, so the sliding window length K can be dynamically adjusted according to the detection results, so as to better balance the real-time performance of anomaly detection and the robustness of the reconstruction model.

[0064] Since the Transformer model does not have a loop structure similar to RNN, it cannot capture the position information of the input sequence. Therefore, this embodiment uses PE (Positional Embedding) encoding to positionally encode the input sequence so that the Transformer model can extract the temporal position information. The PE encoding format is:

[0065]

[0066] Where pos is the temporal position, pos∈[1,K], is the number of features after dimensionality increase, and Denotes a round-down operation. After PE encoding, the position matrix P = {P pos,i} K×2M .

[0067] S103: Processing the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage;

[0068] The encoder consists of L layers of cascaded encoder units. The structure of each layer of encoder units is the same and the parameters are not shared. The output of the previous layer of encoder units is used as the input of the current encoder unit. The processing of the encoder unit is as follows:

[0069]

[0070]

[0071] In the formula, LayerNorm(·) represents the residual connection and layer normalization operation, CausalMultiHeadAttention(·) represents the causal multi-head self-attention, FFN(·) represents the feedforward neural network, is the intermediate output of the encoder unit, I1 is the input matrix of the encoder unit of K×2M dimensions, and is also the intermediate output of the encoder unit of the previous layer. In particular, for the first layer of encoder units, in, represents the merging of two matrices in the feature dimension, It means adding the elements at the same position of two matrices.

[0072] The residual connection is to add the input and output of the network, that is, the output of the network is F(x)+x. When the network structure is deep, the gradient backpropagation to update the parameters is prone to vanishing gradients. If x is added to the output of each layer, it is equivalent to adding a constant term "1" when deriving each layer, which can effectively solve the gradient vanishing problem.

[0073] Normalization is performed layer by layer to make the gradient conduction more stable.

[0074] The single-head self-attention mechanism obtains the attention weights of each position of the input sequence during the encoding process through dot product operations. However, when encoding the information of the current position, it will over-focus on its own position. In view of this, this embodiment adopts a multi-head self-attention mechanism to obtain contextual features from multiple angles, which is conducive to balancing the deviations that may be generated by the same attention mechanism.

[0075] refer to Figure 3 As shown, the multi-head self-attention mechanism analyzes the current input information from different angles. In order to ensure the "order" relationship between the system input and output during dot product calculation, this embodiment adopts a causal multi-head self-attention mechanism, which is mathematically described as:

[0076]

[0077] It can be understood that the h-head attention uses linear transformation to reduce the K×2M dimensions Q, K, and V to K×2M / h dimensions, and then concatenates them into K×2M dimensions after dot product calculation.

[0078] In the above formula, Q = IW q is the query matrix, K = IW k is the bond matrix, V = IW v is the value matrix, I is the input matrix, is the i-th context matrix,

[0079]

[0080] Among them, Softmax(·) represents the normalization operation, is the causal matrix, whose dimension is size(Q i K i T ,2)×size(V i ,1),Q i =QW i q , K i =KW i k , V i =VW i v , W q , W k , W v , W i q , W i k , W i v are all learned linear transformation coefficient matrices.

[0081] In order to improve computational efficiency, in some embodiments, the Transformer model adopts a causal multi-head self-attention mechanism, and the computation context matrix includes:

[0082] The odd query vectors of the query matrix are sequentially multiplied by the odd-numbered vectors of the transposed key matrix to obtain the first type of data of the odd-numbered rows of the attention score matrix, and the even query vectors of the query matrix are sequentially multiplied by the even-numbered items of the transposed key matrix to obtain the second type of data of the even-numbered rows of the attention score matrix, and the second type of data in each row of the attention score matrix is ​​set to negative infinity until the query vectors of the query matrix are traversed to obtain the attention score matrix;

[0083] Normalizing the non-negative infinite items in the attention score matrix and setting the negative infinite items to zero to obtain a weight matrix;

[0084] The weight matrix is ​​multiplied by the value matrix to obtain the context matrix.

[0085] Specifically, the time and space complexity of the traditional multi-head self-attention mechanism is very large. In the attention calculation process, due to the dot product calculation Q i K i TAfter that, the value after multiplying the irrelevant feature vectors will be very small, that is, the time dependency between the current time step and these data can be completely ignored. In addition, considering that the industrial equipment status data has high repeatability and continuity, this embodiment proposes a fast approximate multi-head attention calculation to improve the calculation efficiency.

[0086] Combination Figure 4 As shown, the steps of calculating the i-th context matrix include:

[0087] Step 1: First, transform the matrix Q i The first query vector is sequentially compared with the matrix K i T The odd-numbered vectors of are multiplied to obtain the first type of data in the first row, such as Figure 4 As shown in the gray square in the first row, the second type of data in the first row is Figure 4 All white squares in the first row shown in the figure are set to -∞;

[0088] Step 2: Transform the matrix Q i The second query vector is sequentially compared with the matrix K i T The even-numbered vectors of are multiplied to obtain the first type of data in the second row, such as Figure 4 As shown in the gray box, the second type of data in the second row is Figure 4 All white squares in the second row shown in the figure are set to -∞;

[0089] Step 3: According to Step 1 to Step 2, the remaining query vectors and matrix K i T Multiply the vectors in until the matrix Q is traversed. i All query vectors in , get the i-th attention score matrix;

[0090] Step 4: Perform Softmax(·) normalization on the non-∞ items in the i-th attention score matrix, and directly set the -∞ items to 0 to obtain the i-th normalized attention score matrix, i.e., the weight matrix;

[0091] Step 5: Finally, combine the i-th weight matrix with V i Multiply and output the i-th context matrix.

[0092] The decoder consists of a pair of L-layer cascaded decoder units. The structure of each layer of decoder units is the same, and the parameters are not shared. The output of the previous layer of decoder units is used as the input of the current decoder unit. The processing process of the decoder unit is as follows:

[0093]

[0094]

[0095]

[0096] Where d1 and d2 represent the intermediate outputs of decoder 1 and decoder 2 in stage I, respectively. I2 is the input matrix of the K×2M-dimensional decoder unit, which is also the intermediate output of the decoder unit in the previous layer. In particular, for the first layer decoder unit, Among them, O t,ref is the reference matrix. Considering that it is adversarial training, the window matrix can be selected as the reference matrix, that is, O t,ref =W t .

[0097] After being processed by the decoder, it is sent to the fully connected layer and the Softmax layer to obtain the reconstruction matrix of stage I:

[0098] O i,t =Softmax(Linear(d i )),i∈[1,2];

[0099] In the formula, Linear(·) represents full connection processing, and the one with the highest probability will be output after Softmax(·) normalization processing. are the reconstructed outputs of decoder 1 and decoder 2 at the tth time step of stage I, respectively, where

[0100] Similarly, the reconstruction matrix of stage II is obtained:

[0101]

[0102] In the formula, is the intermediate output of decoder 2 in stage II at the tth time step, is the reconstructed output of decoder 2 at the tth time step in stage II, where

[0103] S104: Calculating a reconstruction error according to the reconstruction outputs of the first stage and the second stage, and determining an abnormality score according to the reconstruction error;

[0104] The reconstruction error is used to quantify the difference between the original input and the reconstructed output. The anomaly score reflects the probability of an abnormality in the device.

[0105] In some embodiments, calculating the reconstruction error according to the reconstruction outputs of the first stage and the second stage includes:

[0106] According to the reconstruction output of the first stage and the reconstruction output of the second stage, the reconstruction error of the multivariate feature and the reconstruction error of the univariate feature are calculated respectively.

[0107] Specifically, the reconstruction error of multivariate features is calculated as follows:

[0108]

[0109] In the formula, e t is the reconstruction error of the multivariate features at the t-th time step, Σ i,t , are all covariance matrices and satisfy:

[0110]

[0111]

[0112] The reconstruction error of a unary feature is calculated as follows:

[0113]

[0114] In the formula, is the reconstruction error of the i-th feature at the t-th time step, M is the number of features, and |·| represents the absolute value.

[0115] In some embodiments, determining an anomaly score based on the reconstruction error comprises:

[0116] determining an anomaly score of the multivariate feature based on a reconstruction error of the multivariate feature;

[0117] An anomaly score of the unary feature is determined based on a reconstruction error of the unary feature.

[0118] Specifically, the anomaly score of a multivariate feature is calculated as follows:

[0119]

[0120] In the formula, β t,score is the anomaly score of the multivariate feature, Φ(·) is the cumulative distribution function, μ i , σ i are the empirical mean and empirical standard deviation of the ith feature respectively.

[0121] The anomaly score of a unigram feature is calculated as follows:

[0122]

[0123] In the formula, is the anomaly score of the ith feature.

[0124] S105: Determine an abnormality threshold, and obtain a detection result according to the abnormality score and the abnormality threshold.

[0125] In some embodiments, determining the abnormal threshold comprises:

[0126] According to the abnormal score and extreme value index of the univariate feature, the abnormal threshold of the univariate feature is estimated by a POT (Peak Over Threshold) model;

[0127] The abnormal threshold of the multi-element feature is calculated based on the abnormal threshold of each of the uni-element features.

[0128] Specifically, this embodiment estimates the abnormal threshold of the unary feature based on the POT model:

[0129]

[0130] In the formula, is the abnormal threshold of the i-th feature after the extreme value indicator strategy is introduced, θ i is the extreme value indicator of the i-th feature, and POT(·) represents the dynamic anomaly threshold estimation function.

[0131] Assume that the reconstruction error sequence {e1,...,e t ,...,e T} have the same distribution function F(e t ), and {e1,...,e t ,...,e T} are independent of each other. Let μ be a sufficiently large threshold. If the random variable satisfies e t >μ, then it is called e t is the over-limit value, y=e t -μ is the excess. According to the PBdH theorem in EVT extreme value theory, when the threshold μ is sufficiently large and e t >μ, then the conditional over-limit distribution F μ (y) obeys the generalized Pareto distribution (GPD) and satisfies:

[0132]

[0133] Where σ>0 is the scale parameter, ξ is the shape parameter, is the tail index.

[0134] Select the extreme value threshold μ:

[0135] When using GPD to fit the distribution of excess quantities, the extreme threshold μ is crucial for accurately estimating the POT model parameters σ and ξ. When μ is too large, the number of excess quantities screened out is small, and the variance of the parameter estimate will be too large; when μ is too small, the conditional excess distribution F cannot be guaranteed. μ (y), that is, it cannot satisfy the conditions for using the PBdH theorem.

[0136] Therefore, the sample average excess function method can be used to select the extreme value threshold μ. t ,...,e T}, define the average excess function:

[0137]

[0138] Where N μ is the number of data in the observation sequence that exceeds the extreme value threshold μ, and k is the starting position of a certain abnormal sample. According to the average excess function curve, the tail characteristics of the observation data can be better estimated. t When the δ(μ) function curve is upward sloping when e≥μ, it means that the shape parameter ξ>0 and the data tail is thick. t When the δ(μ) function curve is downward sloping when e≥μ, it means that the shape parameter ξ<0 and the data tail is thin. t When the δ(μ) function curve is a relatively horizontal straight line when ≥μ, it means that the shape parameter ξ=0 and the data obeys the exponential distribution.

[0139] The selected extreme value threshold μ should satisfy: when e t When ≥μ, the δ(μ) function curve is approximately a straight line.

[0140] Estimate the parameters of the POT model:

[0141] The maximum likelihood method can be used to estimate the parameters σ and ξ and establish the log-likelihood function:

[0142]

[0143] In the formula, f μ (y) = F μ ′(y) is the probability density function.

[0144] By taking partial derivatives of the above log-likelihood function with respect to the parameters σ and ξ, and setting the partial derivatives to zero, we can obtain the estimated value σ * , * .

[0145] Estimated anomaly threshold:

[0146] In the process of equipment anomaly detection, reasonable and reliable anomaly judgment should meet the following requirements:

[0147] Given a confidence level q, there is a judgment threshold z q , so that P(e t >z q )≤q holds.

[0148] Using empirical distribution to describe F(μ), then According to the conditional probability formula After sorting, we get F(e t ) distribution:

[0149]

[0150] When e t >μ and the threshold μ is sufficiently large, then F(e t ) also approximately obeys the GPD distribution.

[0151] According to the definition P(e t >z q )≤q, then 1-F(e t )≤q, after simplification, we can get:

[0152]

[0153] In the formula, z q is the abnormal threshold, and q is the confidence level.

[0154] The above process can be encapsulated into a mathematical model:

[0155] z q =POT(e1,...,e t ,...,e T );

[0156] z q It is not static, but updated online, that is, dynamic.

[0157] Since the time series of industrial equipment status often does not meet the independent and identically distributed conditions, extreme value indicators can be introduced to deal with the correlation problem of stationary time series. Among them, the block method can be used to estimate the extreme value indicators of stationary time series. Specifically, the time series is divided into T / k groups with a sample size of k in each group, and g = [T / k], then the estimator satisfies:

[0158]

[0159] In the formula, θ is the extreme value index, and 0<θ<1, G μ is the number of groups whose maximum value is greater than the extreme value threshold μ.

[0160] After the extreme value index is introduced, According to the above abnormal threshold estimation method, we can get:

[0161]

[0162] In the formula, is the abnormal threshold after the extreme value indicator is introduced.

[0163] The above mathematical model can be updated to obtain the final POT model used to estimate the abnormal threshold:

[0164]

[0165] Determining the anomaly threshold based on extreme value indicators can effectively enhance the adaptability of anomaly detection to complex industrial field environments.

[0166] Furthermore, based on the abnormal threshold of each feature, the abnormal threshold of the multivariate feature is obtained:

[0167]

[0168] In the formula, w i is the design parameter, is the abnormal threshold of the multivariate feature.

[0169] Based on the above embodiment, as a specific implementation manner, obtaining the detection result according to the anomaly score and the anomaly threshold includes:

[0170] If the anomaly score of the multi-feature exceeds the anomaly threshold of the multi-feature, the device is determined to be abnormal.

[0171] Specifically, if If the current state is abnormal, it is considered normal.

[0172] Furthermore, we can also organize and record abnormal events, abnormal thresholds, abnormal frequencies, start and end times, severity, etc. during the abnormality detection process to build a shared knowledge base of abnormal cases. On the one hand, it can be used to accurately identify suspicious data during data preprocessing, and on the other hand, it can provide valuable case references for abnormality detection of industrial equipment.

[0173] In summary, the industrial equipment anomaly detection method provided by this application reconstructs the suspicious data through a two-stage Transformer model, and determines whether the equipment is abnormal based on the reconstruction error, so that it can accurately detect whether the equipment is abnormal. In addition, this application adopts a two-stage adversarial training, which can effectively improve the model's robust expression ability for adversarial samples.

[0174] The present application also provides an industrial equipment abnormality detection device, and the device described below can be referred to in correspondence with the method described above. Figure 5 , Figure 5 A schematic diagram of an industrial equipment abnormality detection device provided in an embodiment of the present application, combined with Figure 5 As shown, the device comprises:

[0175] An extraction module 10, for extracting questionable data and constructing an input matrix according to the questionable data;

[0176] A preprocessing module 20, configured to preprocess the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; and the attention matrix of the first stage is different from the attention matrix of the second stage;

[0177] A reconstruction module 30, used to process the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage;

[0178] A calculation module 40, configured to calculate a reconstruction error according to the reconstruction outputs of the first stage and the second stage, and determine an anomaly score according to the reconstruction error;

[0179] The determination module 50 is used to determine an abnormality threshold and obtain a detection result according to the abnormality score and the abnormality threshold.

[0180] Based on the above embodiment, as a specific implementation, the preprocessing module 20 is further used for:

[0181] Performing sliding window processing on the time series to obtain multiple window matrices; the time series includes multiple input matrices;

[0182] After merging the window matrix and the attention matrix, position encoding is performed on the merged matrix to obtain a position matrix, so as to process the position matrix through the Transformer model.

[0183] Based on the above embodiment, as a specific implementation, the calculation module 40 is specifically used for:

[0184] According to the reconstruction output of the first stage and the reconstruction output of the second stage, the reconstruction error of the multivariate feature and the reconstruction error of the univariate feature are calculated respectively.

[0185] Based on the above embodiment, as a specific implementation, the calculation module 40 includes:

[0186] A first determining unit, configured to determine an abnormality score of the multivariate feature according to a reconstruction error of the multivariate feature;

[0187] The second determining unit is used to determine an abnormality score of the unary feature according to the reconstruction error of the unary feature.

[0188] Based on the above embodiment, as a specific implementation, the determination module 50 includes:

[0189] A first abnormality threshold determination unit, configured to obtain an abnormality threshold of a unary feature through POT model estimation according to the abnormality score and extreme value index of the unary feature;

[0190] The second abnormality threshold determination unit is used to calculate the abnormality threshold of the multivariate feature according to the abnormality threshold of each of the univariate features.

[0191] Based on the above embodiment, as a specific implementation manner, the determination module 20 is specifically used for:

[0192] If the anomaly score of the multi-feature exceeds the anomaly threshold of the multi-feature, the device is determined to be abnormal.

[0193] Based on the above embodiment, as a specific implementation method, the Transformer model adopts a causal multi-head self-attention mechanism, and the calculation context matrix includes:

[0194] The odd query vectors of the query matrix are sequentially multiplied by the odd-numbered vectors of the transposed key matrix to obtain the first type of data of the odd-numbered rows of the attention score matrix, and the even query vectors of the query matrix are sequentially multiplied by the even-numbered items of the transposed key matrix to obtain the second type of data of the even-numbered rows of the attention score matrix, and the second type of data in each row of the attention score matrix is ​​set to negative infinity until the query vectors of the query matrix are traversed to obtain the attention score matrix;

[0195] Normalizing the non-negative infinite items in the attention score matrix and setting the negative infinite items to zero to obtain a weight matrix;

[0196] The weight matrix is ​​multiplied by the value matrix to obtain the context matrix.

[0197] This application also provides an industrial equipment anomaly detection device, referring to Figure 6 As shown, the device includes a memory 1 and a processor 2 .

[0198] Memory 1, used for storing computer programs;

[0199] Processor 2 is used to execute the computer program to implement the following steps:

[0200] Extract questionable data and construct an input matrix based on the questionable data; preprocess the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; process the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage; calculate the reconstruction error based on the reconstructed output of the first stage and the second stage, and determine the anomaly score based on the reconstruction error; determine the anomaly threshold, and obtain the detection result based on the anomaly score and the anomaly threshold.

[0201] For an introduction to the equipment provided in this application, please refer to the above method embodiments, and this application will not go into details here.

[0202] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0203] Extract questionable data and construct an input matrix based on the questionable data; preprocess the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; process the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage; calculate the reconstruction error based on the reconstructed output of the first stage and the second stage, and determine the anomaly score based on the reconstruction error; determine the anomaly threshold, and obtain the detection result based on the anomaly score and the anomaly threshold.

[0204] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0205] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0206] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the devices, equipment, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0207] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0208] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0209] The above is a detailed introduction to the industrial equipment anomaly detection method, device, equipment and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for detecting abnormality of industrial equipment, characterized in that: include: Based on the prior knowledge base, the possibly abnormal data in the collected industrial time series data is identified and extracted to obtain the doubtful data, and an input matrix is ​​constructed according to the doubtful data; The input matrix is ​​preprocessed by a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; the attention matrix ; for dimensional attention matrix, K is the adjustable window length, M is the number of features or variables, It means taking the absolute value of the elements in the matrix. , They represent the reconstructed outputs of the two decoders in stage I at the tth time step; for -dimensional window matrix; Processing the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage; A reconstruction error is calculated based on the reconstruction outputs of the first stage and the second stage, and an anomaly score is determined based on the reconstruction error; Determine an abnormality threshold, and obtain a detection result according to the abnormality score and the abnormality threshold; The Transformer model adopts a causal multi-head self-attention mechanism, and the calculation context matrix includes: The odd query vectors of the query matrix are sequentially multiplied by the odd-numbered vectors of the transposed key matrix to obtain the first-category data of the odd-numbered rows of the attention score matrix, and the even query vectors of the query matrix are sequentially multiplied by the even-numbered items of the transposed key matrix to obtain the first-category data of the even-numbered rows of the attention score matrix, and the second-category data in each row of the attention score matrix is ​​set to negative infinity until the query vectors of the query matrix are traversed to obtain the attention score matrix; Normalizing the non-negative infinite items in the attention score matrix and setting the negative infinite items to zero to obtain a weight matrix; The weight matrix is ​​multiplied by the value matrix to obtain the context matrix.

2. The method for detecting abnormality of industrial equipment according to claim 1, characterized in that: The preprocessing of the input matrix by the two-stage Transformer model further includes: Performing sliding window processing on the time series to obtain multiple window matrices; the time series includes multiple input matrices; After merging the window matrix and the attention matrix, position encoding is performed on the merged matrix to obtain a position matrix, so as to process the position matrix through the Transformer model.

3. The method for detecting abnormality of industrial equipment according to claim 1, characterized in that: The calculation of the reconstruction error according to the reconstruction output of the first stage and the second stage includes: According to the reconstruction output of the first stage and the reconstruction output of the second stage, the reconstruction error of the multivariate feature and the reconstruction error of the univariate feature are calculated respectively.

4. The method for detecting abnormality of industrial equipment according to claim 3, characterized in that: Determining an anomaly score according to the reconstruction error comprises: determining an anomaly score of the multivariate feature based on a reconstruction error of the multivariate feature; An anomaly score of the unary feature is determined based on a reconstruction error of the unary feature.

5. The method for detecting abnormality of industrial equipment according to claim 4, characterized in that: Determining the abnormal threshold comprises: According to the abnormal score and extreme value index of the univariate feature, the abnormal threshold of the univariate feature is estimated by the POT model; The abnormal threshold of the multi-element feature is calculated based on the abnormal threshold of each of the uni-element features.

6. The method for detecting abnormality of industrial equipment according to claim 5, characterized in that: The obtaining of the detection result according to the anomaly score and the anomaly threshold comprises: If the anomaly score of the multi-feature exceeds the anomaly threshold of the multi-feature, the device is determined to be abnormal.

7. An industrial equipment abnormality detection device, characterized in that: include: An extraction module, used to identify and extract possibly abnormal data from the collected industrial time series data based on a priori knowledge base to obtain questionable data, and to construct an input matrix based on the questionable data; A preprocessing module, configured to preprocess the input matrix through a two-stage Transformer model; wherein the preprocessing of the input matrix includes merging the input matrix with an attention matrix; the attention matrix of the first stage is different from the attention matrix of the second stage; the attention matrix ; for dimensional attention matrix, K is the adjustable window length, M is the number of features or variables, It means taking the absolute value of the elements in the matrix. , They represent the reconstructed outputs of the two decoders in stage I at the tth time step; for -dimensional window matrix; A reconstruction module, used to process the preprocessed matrix through the encoder and decoder of the Transformer model to obtain the reconstructed output of the first stage and the second stage; A calculation module, configured to calculate a reconstruction error according to the reconstruction outputs of the first stage and the second stage, and determine an anomaly score according to the reconstruction error; A determination module, used to determine an abnormality threshold, and obtain a detection result according to the abnormality score and the abnormality threshold; The Transformer model adopts a causal multi-head self-attention mechanism, and the calculation context matrix includes: The odd query vectors of the query matrix are sequentially multiplied by the odd-numbered vectors of the transposed key matrix to obtain the first-category data of the odd-numbered rows of the attention score matrix, and the even query vectors of the query matrix are sequentially multiplied by the even-numbered items of the transposed key matrix to obtain the first-category data of the even-numbered rows of the attention score matrix, and the second-category data in each row of the attention score matrix is ​​set to negative infinity until the query vectors of the query matrix are traversed to obtain the attention score matrix; Normalizing the non-negative infinite items in the attention score matrix and setting the negative infinite items to zero to obtain a weight matrix; The weight matrix is ​​multiplied by the value matrix to obtain the context matrix.

8. An industrial equipment anomaly detection device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the industrial equipment anomaly detection method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the industrial equipment abnormality detection method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Time series data anomaly detection method and device and medium

    CN113110972A

  • Power machine room anomaly detection method, device and equipment and readable storage medium

    CN115078894A

  • Time sequence anomaly prediction method and device, equipment and storage medium

    CN115829160A