Aviation time series data anomaly detection method based on Transform network

Abstract

The invention discloses an aviation time series data anomaly detection method based on a Transform network, belongs to the technical field of aviation data processing, and focuses on the context dependency relationship of time series data. Enhancing time sequence relation mining of the anomaly detection model based on the Transform network on the aviation time sequence data through a differential positive anomaly sample representation learning mode; in order to realize differentiation, firstly, a training set is expanded in an abnormal injection mode, then a differentiation objective function is designed, and an alternate training mode is adopted to promote an abnormal detection model based on a Transform network to identify the difference between normal and abnormal samples; the problem that an existing anomaly detection model is not obvious in positive anomaly sample distinguishing degree is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of aviation data processing, and in particular relates to an aviation time series data anomaly detection method based on a Transformer network. Background Art

[0002] Time series data is ordered data recorded in chronological order and is used to describe changes in the system. In the field of civil aviation, flight data such as operation parameters, status parameters, warning parameters and environmental parameters recorded by the quick access recorder are typical time series data, which is called aviation time series data in this invention.

[0003] Aviation time series data prediction is based on artificial intelligence and big data technology. It predicts future status by mining the characteristic laws contained in historical data. It is mainly used for health status monitoring, fault diagnosis, and life prediction of key aircraft components. It is a necessary condition for realizing fault prediction and diagnosis technology.

[0004] Aviation time series data anomaly detection aims to detect data points or fragments in the time series that are inconsistent with normal data. It has applications in many fields, such as industry, aerospace, and the Internet. Accurately identifying and locating anomalies in time series data is crucial to ensuring system security and improving operational efficiency. However, the time series data generated in practical applications usually have the characteristics of non-stationarity, high dimensionality, and abnormal scarcity, making it very difficult to detect anomalies. At present, most time series anomaly detection algorithms are unsupervised algorithms. According to the anomaly judgment criteria, existing methods can be divided into methods based on density estimation, clustering, prediction, and reconstruction. Recently, there are also studies based on contrastive learning to achieve anomaly detection.

[0005] Density estimation-based methods identify abnormal samples by estimating the density distribution of data. Representative methods include: LOF is a non-parametric method that determines outliers by considering the relative density of data points and their neighboring points. LOF focuses on the degree of outliers in local areas and detects samples with local abnormal behaviors; the parametric methods DAGMM and MPPCACD reduce the dimensionality of data through autoencoders and probabilistic principal component analysis, respectively, and then use a mixed Gaussian model to fit the data distribution, thereby calculating the probability density of the sample.

[0006] Clustering-based methods mainly judge abnormal samples by calculating the similarity or distance between data points. Representative methods include iForest, which builds a random partitioning tree to gradually partition the data, and measures the isolation of the sample based on its depth in the tree to identify anomalies. In addition, there are machine learning-based methods such as OC-SVM and Deep-SVDD.

[0007] As the mainstream research direction in recent years, prediction-based methods build models to learn the normal pattern of time series, and then use the model to predict future data points. VAR associates the current value of each time series variable with the value of the past time step and the values ​​of other related variables, and performs linear combination to build a multivariate autoregressive model for prediction. However, this method assumes that the relationship between variables is linear and does not take into account nonlinear relationships. LSTM and GRU are commonly used recurrent neural network variants that can learn and model nonlinear relationships of time series changes. They also have the ability to capture long-term dependencies in sequence data and have been widely used in time series anomaly detection methods based on prediction and reconstruction. In addition, due to its powerful time series modeling capabilities, the Transformer model is widely used in natural language processing, time series data prediction, time series data anomaly detection and other tasks.

[0008] Aviation time data usually exhibits contextual dependencies in the time dimension, that is, there is a correlation between past and present observations and future observations, and this relationship is latent and dynamic. Fully modeling this contextual dependency is crucial to achieve accurate anomaly detection. Traditional methods usually only consider data at a single moment and ignore contextual information, and perform poorly on time series anomaly detection tasks. Many deep methods are based on recurrent neural networks that iteratively input data. Although these methods incorporate contextual information from all past moments, they lack moment-level dependency representation. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the aviation time series data anomaly detection method based on Transformer network provided by the present invention solves the problem that the existing methods for aviation time series data anomaly detection tasks only consider single moment data and ignore contextual information, thereby affecting the accuracy of anomaly detection.

[0010] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for detecting anomalies in aviation time series data based on a Transformer network, comprising the following steps:

[0011] S1. Collect aviation time series data and inject abnormal data to form a training data set;

[0012] S2. Build an anomaly detection model based on the Transformer network, use it to extract the sequence features of the training data set, and extract the weight matrix in the encoder in the anomaly detection model as a differentiation credential;

[0013] S3, performing differentiated enhancement according to differentiated credentials, thereby forming a differentiated objective function;

[0014] S4. Alternately train the anomaly detection model using the differential objective function;

[0015] S5. Use the trained anomaly detection model to detect anomalies in aviation time series data.

[0016] Further, the specific steps of step S1 are as follows:

[0017] S11. Collect an aviation time series data set X = [x1,…,x T ∈ R T×D ; where D is the data dimension, and each moment of data has a corresponding label, denoted as Y = [y1,...,y T , the label 0 indicates normal, and 1 indicates abnormal;

[0018] S12. Use a sliding window to divide the aviation time series data to form a window data set where N is the sliding window length, S is the sliding step, X t ,Y t are the data and labels of each sliding window respectively;

[0019] S13. Divide the window data set D into a training set D train and a test set D test ;

[0020] S14. Inject abnormal data into the training set D train to form a training data set.

[0021] Further, the specific steps of step S14 are as follows:

[0022] S14-1. Copy X t as the template for the abnormal window;

[0023] S14-2. Select a subsequence of length p that satisfies p < N from the abnormal window;

[0024] S14-3. Inject abnormal data into the subsequence;

[0025] S14-4. Mark each moment in the abnormal window as abnormal

[0026] S14-5. Add the subsequence (X Ano ,Y Ano ) injected with abnormal data to the training set to form a training data set; where X Ano is the normal data before abnormal data injection.

[0027] Further, in step S14-3, the abnormal data injection methods include migration injection, Gaussian injection, and constant injection.

[0028] Furthermore, in step S2, the anomaly detection model based on the Transformer network includes an encoder and a decoder; wherein the encoder includes an embedding layer and a plurality of stacked encoding layers.

[0029] Furthermore, the working process of the encoder is:

[0030] A1. X t As the input data of the anomaly detection model, it is encoded through the embedding layer, feature information is extracted, and position encoding is added. The calculation process expression is:

[0031] X emb =PosEmb(X)+InputEmb(X)

[0032] Where PosEmb(·) and InputEmb(·) represent position embedding and input feature embedding respectively. Represents X t The representation vector obtained by the embedding layer is used as the input of the first encoding layer, where d model represents the size of the hidden layer;

[0033] A2. Input the feature information with position coding into the coding layer and process it layer by layer. The processing expression is:

[0034] Z l =Layer-Norm(X l-1 +Attention(X l-1 ))

[0035] X l =Layer-Norm(Z l +MLP(Z l ))

[0036] In the formula, represents the output representation of the lth encoding layer, represents the intermediate representation of the lth coding layer, MLP(·) and Layer-Norm(·) represent the multi-layer perceptron and layer normalization calculation respectively, and Attention(·) represents the self-attention operation in the time dimension through the attention module.

[0037] Furthermore, in step S2, the method for extracting the weight matrix in the encoder is:

[0038] Through the Attention(·) operation, a context dependency is established between any two windows at any time, which is expressed as the attention weight, and then the weight matrix is ​​constructed. The calculation process is expressed as:

[0039]

[0040]

[0041]

[0042] In the formula, for the attention module of the lth encoding layer, represents the query vector at the i-th moment, is the key value vector at the i-th moment, Represents the dependency weight of the i-th moment on each moment in the entire window, that is, the attention weight, and obtains the weight matrix at the i-th moment Wherein, i=1, 2, ..., N, L represents the total number of coding layers, and N represents the total number of time moments.

[0043] Furthermore, the step S3 is specifically as follows:

[0044] S31. Calculate the differentiated distance according to the differentiated voucher;

[0045] S32, calculating the differentiation loss according to the differentiation distance;

[0046] S33. Calculate a differentiated objective function according to the differentiated loss.

[0047] Furthermore, in step S31, the differential distance L dis The calculation formula is:

[0048]

[0049] In the formula, represents the center of the hypersphere corresponding to time i at the lth coding layer, L represents the total number of coding layers, KL(·) represents the KL divergence function, and the index i measures the relative position of different time in the window;

[0050] In step S32, the difference loss L diff The calculation formula is:

[0051]

[0052] In the formula, represents the difference distance of normal samples, represents the differential distance of abnormal samples, λ∈[0,1] represents the loss weight of positive and abnormal moments respectively, γ represents a hyperparameter, min(·) is the minimization function, and N represents the window length, i.e. the total number of moments;

[0053] in, and Respectively expressed as:

[0054]

[0055]

[0056] In the formula, y i represents the abnormal label at the i-th moment;

[0057] In step S33, the differentiation objective function is:

[0058]

[0059]

[0060] In the formula, represents the reconstructed output result of the anomaly detection model for the input window X, Indicates that in calculating L diff When the parameters of the hypersphere center C are frozen, Indicates that in calculation Freeze the parameters of the differentiated voucher.

[0061] Furthermore, in step S4, the differentiated objective functions L1 and L2 are used alternately to train the anomaly detection model based on the Transformer network to implement parameter updating and obtain a trained anomaly detection model based on the Transformer network.

[0062] The beneficial effects of the present invention are:

[0063] (1) The present invention focuses on the contextual dependency of time series data and enhances the time series relationship mining of aviation time series data by using the Transformer network-based anomaly detection model by learning the representation of differentiated positive and abnormal samples.

[0064] (2) In order to achieve differentiation, the present invention first expands the training set by anomaly injection, then designs a differentiation objective function, and uses alternating training to enable the Transformer network-based anomaly detection model to recognize the differences between normal and abnormal samples, thereby solving the problem that the existing anomaly detection model has unclear ability to distinguish between positive and abnormal samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1A flow chart of the aviation time series data anomaly detection method based on Transformer network provided by the present invention. DETAILED DESCRIPTION

[0066] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0067] The embodiment of the present invention provides a method for detecting anomalies in aviation time series data based on a Transformer network. Figure 1 As shown, the following steps are included:

[0068] S1. Collect aviation time series data and inject abnormal data to form a training data set;

[0069] S2. Build an anomaly detection model based on the Transformer network, use it to extract the sequence features of the training data set, and extract the weight matrix in the encoder in the anomaly detection model as a differentiation credential;

[0070] S3, performing differentiated enhancement according to differentiated credentials, thereby forming a differentiated objective function;

[0071] S4, alternately train the anomaly detection model using a differentiated objective function;

[0072] S5. Use the trained anomaly detection model to perform anomaly detection on aviation time series data.

[0073] Step S1 of the embodiment of the present invention is specifically as follows:

[0074] S11, collect the aviation time series data set X=[x1,…,x T ]∈R T×D ; Where D is the data dimension, and the data at each moment has a corresponding label, expressed as Y = [y1,...,y T ], the label is 0 for normal and 1 for abnormal;

[0075] S12. Use sliding windows to divide aviation time series data to form a window data set Among them, N is the sliding window length, S is the sliding step size, and X t ,Y t The data and labels of each sliding window respectively;

[0076] S13, divide the window data set D into training set Dtrain and the test set D test ;

[0077] S14. Inject abnormal data into the training set D train to form a training data set.

[0078] The specific steps of step S14 in this embodiment are as follows:

[0079] S14-1. Copy X t as the template of the abnormal window, that is, X Ano = Copy(X t );

[0080] S14-2. Select a subsequence of length p that satisfies p < N from the abnormal window;

[0081] S14-3. Inject abnormal data into the subsequence;

[0082] S14-4. Mark each moment within the abnormal window as abnormal

[0083] S14-5. Add the subsequence (X Ano , Y Ano ) injected with abnormal data to the training set to form a training data set, that is, D train ← D train ∪{(X Ano , Y Ano )}; where X Ano is the normal data before abnormal data injection.

[0084] In step S14-3 of this embodiment, the abnormal data injection methods include migration injection, Gaussian injection, and constant injection; specifically, migration injection means replacing the segments of X k ∈ D train with segments of the same length within another window X Ano ; Gaussian injection means replacing the segments of X Ano with a Gaussian noise sequence, and each value of the noise follows a Gaussian distribution; constant injection means sampling a constant from a uniform distribution and replacing the value of each moment of the subsequence with this constant.

[0085] In step S2 of the embodiment of the present invention, the anomaly detection model based on the Transformer network includes an encoder and a decoder; among them, the encoder includes an embedding layer and multiple stacked encoding layers.

[0086] The working process of the encoder of the anomaly detection model based on the Transformer network in this embodiment is as follows:

[0087] A1. Take X tAs the input data of the anomaly detection model, it is encoded through the embedding layer, feature information is extracted, and position encoding is added. The calculation process expression is:

[0088] X emb =PosEmb(X)+InputEmb(X)

[0089] Where PosEmb(·) and InputEmb(·) represent position embedding and input feature embedding respectively. Represents X t The representation vector obtained by the embedding layer is used as the input of the first encoding layer, where d model represents the size of the hidden layer;

[0090] A2. Input the feature information with position coding into the coding layer and process it layer by layer. The processing expression is:

[0091] Z l =Layer-Norm(X l-1 +Attention(X l-1 ))

[0092] X l =Layer-Norm(Z l +MLP(Z l ))

[0093] In the formula, represents the output representation of the lth encoding layer, represents the intermediate representation of the lth coding layer, MLP(·) and Layer-Norm(·) represent the multi-layer perceptron and layer normalization calculation respectively, and Attention(·) represents the self-attention operation in the time dimension through the attention module.

[0094] In step S2 of the embodiment of the present invention, the method for extracting the weight matrix in the encoder in the Transformer-based anomaly detection model is:

[0095] Through the Attention(·) operation, a context dependency is established between any two windows at any time, which is expressed as the attention weight, and then the weight matrix is ​​constructed. The calculation process is expressed as:

[0096]

[0097] In the formula, for the attention module of the lth encoding layer, represents the query vector at the i-th moment, is the key value vector at the i-th moment, Represents the dependency weight of the i-th moment on each moment in the entire window, that is, the attention weight, and obtains the weight matrix at the i-th moment Wherein, i=1, 2, ..., N, L represents the total number of coding layers, and N represents the total number of time moments.

[0098] Step S3 of the embodiment of the present invention is specifically:

[0099] S31. Calculate the differentiated distance according to the differentiated voucher;

[0100] S32, calculating the differentiation loss according to the differentiation distance;

[0101] S33. Calculate a differentiated objective function according to the differentiated loss.

[0102] Specifically, in step S31 of this embodiment, the differential distance L dis The calculation formula is:

[0103]

[0104] In the formula, represents the center of the hypersphere corresponding to the time i of the lth coding layer, L represents the total number of coding layers, KL(·) represents the KL divergence function, and the index i measures the relative position of different time in the window; in step S32 of this embodiment, the differential loss L diff The calculation formula is:

[0105]

[0106] In the formula, represents the difference distance of normal samples, represents the differential distance of abnormal samples, λ∈[0,1] represents the loss weight of positive and abnormal moments respectively, γ represents a hyperparameter, min(·) is the minimization function, and N represents the window length, i.e. the total number of moments;

[0107] in, and Respectively expressed as:

[0108]

[0109]

[0110] In the formula, y i represents the abnormal label at the i-th moment;

[0111] In step S33 of this embodiment, the differentiation objective function is:

[0112]

[0113]

[0114] In the formula, represents the reconstructed output result of the anomaly detection model for the input window X, Indicates that in calculating L diff When the parameters of the hypersphere center C are frozen, Indicates that in calculation Freeze the parameters of the differentiated voucher.

[0115] In step S4 of the embodiment of the present invention, based on the constructed differentiation objective function, the differentiation objective functions L1 and L2 are used alternately to train the anomaly detection model based on the Transformer network, implement parameter update, and obtain a trained anomaly detection model based on the Transformer network. Among them, L1 is responsible for the optimization of the model for temporal relationship learning, and L2 is responsible for adjusting the parameters of the hypersphere center.

[0116] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0117] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for detecting anomalies in aviation time series data based on Transformer network, characterized in that: It includes the following steps: S1. Collect aviation time-series data, inject abnormal data, and form a training dataset; S2. Build an anomaly detection model based on the Transformer network, use it to extract the sequence features of the training dataset, and extract the weight matrix in the encoder of the anomaly detection model as a differentiation voucher; S3. Perform differential enhancement according to the differentiation voucher, and then form a differential objective function; S4. Use the differential objective function to alternately train the anomaly detection model; S5. Use the trained anomaly detection model to detect anomalies in aviation time-series data.

2. According to claim 1, the method for detecting anomalies in aviation time series data based on Transformer network is characterized in that: The specific content of step S1 is: S11, collect the aviation time series data set X=[x1,…,x T ]∈R T×D ; Where D is the data dimension, and the data at each moment has a corresponding label, expressed as Y = [y1,...,y T ], the label is 0 for normal and 1 for abnormal; S12. Use sliding windows to divide aviation time series data to form a window data set Among them, N is the sliding window length, S is the sliding step size, and X t ,Y t The data and labels of each sliding window respectively; S13, divide the window data set D into training set D train and the test set D test ; S14, to the training set D train Abnormal data is injected into the training dataset to form a training data set.

3. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 2 is characterized in that: The specific content of step S14 is: S14-1, X t Copy the template as the exception window; S14-2. Select a subsequence with a length of p and satisfying p < N from the anomaly window; S14-3. Inject abnormal data into the subsequence; S14-4. Mark each moment in the abnormal window as abnormal S14-5, inject the subsequence of abnormal data (X Ano ,Y Ano ) is added to the training set to form a training data set; where X Ano Normal data before abnormal data injection.

4. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 3 is characterized in that: In step S14-3, the abnormal data injection methods include migration injection, Gaussian injection, and constant injection.

5. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 3 is characterized in that: In step S2, the anomaly detection model based on the Transformer network includes an encoder and a decoder; among them, the encoder includes an embedding layer and multiple stacked encoding layers.

6. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 5 is characterized in that: The working process of the encoder is: A1. X t As the input data of the anomaly detection model, it is encoded through the embedding layer, feature information is extracted, and position encoding is added. The calculation process expression is: X emb =PosEmb(X)+InputEmb(X) Where PosEmb(·) and InputEmb(·) represent position embedding and input feature embedding respectively. Represents X t The representation vector obtained by the embedding layer is used as the input of the first encoding layer, where d model represents the size of the hidden layer; A2. Input the feature information added with positional encoding into the encoding layer and process it layer by layer. The expression of its processing process is: Z l =Layer-Norm(X l-1 +Attention(X l-1 )) X l =Layer-Norm(Z l +MLP(Z l )) In the formula, represents the output representation of the lth encoding layer, represents the intermediate representation of the lth coding layer, MLP(·) and Layer-Norm(·) represent the multi-layer perceptron and layer normalization calculation respectively, and Attention(·) represents the self-attention operation in the time dimension through the attention module.

7. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 6 is characterized in that: In step S2, the method for extracting the weight matrix in the encoder is: Through the Attention(·) operation, establish an association between any two moments among the windows to form a context dependence relationship, represent it as attention weights, and then construct a weight matrix. Its calculation process is expressed as: In the formula, for the attention module of the lth encoding layer, represents the query vector at the i-th moment, is the key value vector at the i-th moment, Represents the dependency weight of the i-th moment on each moment in the entire window, that is, the attention weight, and obtains the weight matrix at the i-th moment Wherein, i=1, 2, ..., N, L represents the total number of coding layers, and N represents the total number of time moments.

8. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 7 is characterized in that: The specific content of step S3 is: S31. Calculate the differential distance according to the differentiation voucher; S32. Calculate the differential loss according to the differential distance; S33. Calculate the differential objective function according to the differential loss.

9. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 8, characterized in that: In step S31, the differential distance L dis The calculation formula is: In the formula, represents the center of the hypersphere corresponding to time i at the lth coding layer, L represents the total number of coding layers, KL(·) represents the KL divergence function, and the index i measures the relative position of different time in the window; In step S32, the difference loss L diff The calculation formula is: In the formula, represents the difference distance of normal samples, represents the differential distance of abnormal samples, λ∈[0,1] represents the loss weight of positive and abnormal moments respectively, γ represents a hyperparameter, min(·) is the minimization function, and N represents the window length, i.e. the total number of moments; in, and Respectively expressed as: In the formula, y i represents the abnormal label at the i-th moment; In step S33, the differential objective function is: In the formula, represents the reconstructed output result of the anomaly detection model for the input window X, Indicates that when calculating L diff When the parameters of the hypersphere center C are frozen, Indicates that in calculation Freeze the parameters of the differentiated voucher.

10. The method for detecting anomalies in aviation time series data based on Transformer network according to claim 9, characterized in that: In step S4, alternately use the differential objective functions L1 and L2 to train the anomaly detection model based on the Transformer network, realize parameter update, and obtain the trained anomaly detection model based on the Transformer network.

Images