A flight data anomaly detection method and system based on a dual-view enhanced Transformer

Abstract

The application relates to a flight data abnormality detection method and system based on a dual-view enhanced Transformer, and belongs to the technical field of aviation safety. The application extends the flight safety abnormality detection task to beyond the traditional overrun event, and aims at the subtle and often undetected abnormality in the flight process. According to the fact that QAR data is a typical form of time series data, the flight safety problem is re-expressed as an abnormality detection task in time series analysis. In addition, in order to solve the challenge of excessive dependence on expert labeled data in flight safety research, an unsupervised abnormality detection model specially designed for QAR data is designed. Finally, according to the sparse characteristics of QAR data, a time series abnormality detection model architecture based on reconstruction is designed. The application can timely identify the condition deviating from the normal flight mode, provide early warning information for airlines, and effectively assist experts in analyzing abnormal flight behavior, and has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of aviation safety technology and relates to a method and system for detecting flight data anomalies based on dual-view enhanced Transformer. Background Technology

[0002] Aviation safety has always been a core issue for the civil aviation industry. According to Airbus's statistical analysis of commercial aviation accidents, the number of flight accidents in 2022 was significantly lower than a decade ago. However, considering the economic losses and severity of accidents, aviation safety remains a top priority for airlines and has become an important direction for the continuous improvement and optimization of the civil aviation industry. Quick Access Recorders (QARs), as flight data recording devices, are now widely installed on various types of commercial passenger aircraft in China. They record multiple parameters in real time, including aircraft status, pilot operations, and environmental factors, throughout the entire flight phase. Analyzing deviations from reference data to determine the presence of malfunctions is crucial for ensuring flight safety. Currently, anomaly data analysis for QARs is still very limited. Airlines mostly use exceedance detection (ED) methods to analyze QAR data, but this method is limited to predefined anomalies and cannot analyze the large number of unknown anomalies in QAR data, thus still posing certain safety risks. Furthermore, the scale of modern aviation operations and the complexity of QAR data make threshold-based methods unusable because they are not scalable. Therefore, how to utilize advanced big data and artificial intelligence technologies to improve the analysis and processing capabilities of flight data, thereby enhancing the level of civil aviation safety management, is a key focus of the civil aviation industry.

[0003] In recent years, scholars have conducted extensive research on flight safety. This research can be broadly categorized into two types: flight safety studies based on expert knowledge analysis and data-driven flight safety incident studies. Expert knowledge-based flight safety studies rely heavily on flight experts' profound insights into civil aviation operations, identifying risk factors for unsafe events from the perspectives of risk causes, crew psychology, and flight management, and proposing preventative measures. Chen Zhihua et al. analyzed hard landing events from a physical perspective, further discovering a close relationship between crew psychological factors (fear of hard landings and pursuit of standard landing procedures) and hard landings, tail strikes, and runway overruns, and providing suggested elements and specific measures for preventing hard landings. Zhou Jin analyzed the patterns of hard landing events in recent years, obtaining the probability and impact of each risk factor based on expert scoring, and evaluating and ranking the risk causes using entropy-weighted TOPSIS, identifying the main causes of hard landing events. Based on these causes, the author proposed operational and psychological suggestions. Data-driven flight safety incident studies are typically abstracted as anomaly detection based on flight data. These studies are mainly divided into sequential anomalies and point anomalies, referred to as flight-level anomalies and instantaneous anomalies, respectively. Flight-level anomalies refer to sequences or subsequences in flight data that form unusual shapes. For example, Li Xu et al. used curve clustering to identify hard landing types. First, they summarized the causes of hard landing events through group and individual comparison analysis, then established a secondary classification directory tree, and extracted curve features for hard landing pattern identification. Transient anomalies refer to outliers in flight data that deviate from expectations. They are usually detected by exceeding limits to determine whether a flight safety event has occurred. For example, Wang Lei et al. extended the hard landing risk assessment model to a landing risk assessment model, considering three types of exceeding limits: tail strike, hard landing, and runway overrun. Chen et al. explored the parameters affecting the maximum vertical acceleration during the landing phase using quantile regression. In addition, some machine learning-based methods such as support vector machines and random forests have also been used for hard landing event prediction and early warning. Kang Zongwei et al. used a multi-layer spatiotemporal encoder and TG-Attention to predict aircraft landing speed to directly warn of the risk of runway overrun. Tong Chao and others used LSTM models to predict the vertical load and landing speed of aircraft during landing, thereby providing early warning of hard landing and runway overrun events.

[0004] Furthermore, since QAR data is typically high-dimensional time-series data, the problem of anomaly detection in QAR data is also a typical time-series anomaly detection problem. As a crucial real-world issue, it has been widely discussed by scholars in academia. Based on whether the training data contains anomaly labels, existing research is mainly divided into supervised and unsupervised time-series anomaly detection. Supervised methods perform better when anomaly labels are available or accessible; these methods can be traced back to AutoEncoder, LSTM-VAE, etc. On the other hand, when anomaly labels are difficult to obtain, unsupervised methods often yield models with stronger generalization capabilities, including OmniAnomaly, GDN, and DAGMM. Due to the massive, high-dimensional, irregular, and difficult-to-obtain anomaly labels characteristics of real-world time-series data, unsupervised time-series anomaly detection research is more extensive. Depending on the anomaly identification criteria, these studies broadly include density estimation-based, clustering-based, reconstruction-based, and autoregressive-based methods. Density estimation-based methods identify outliers in low-density regions, assuming they reside in dense neighborhoods. The density of a sample point is compared to the density of its neighbors. Classical methods like LOF and COF calculate the local density and connectivity of outliers for anomaly identification. Clustering-based methods typically formalize anomaly scores as distances from data points to cluster centers. SVDD and Deep SVDD cluster normal data points into compact clusters. THOC fuses multi-scale temporal features from intermediate layers through hierarchical clustering and detects anomalies using multi-layer distances. Reconstruction-based methods attempt to detect anomalies through reconstruction errors. Park et al. proposed the LSTM-VAE model, using LSTM for temporal modeling and a Variational Autoencoder (VAE) for reconstruction. OmniAnomaly further extended the LSTM-VAE model by adding a normalized flow and using reconstruction probabilities for detection. Autoregressive models detect anomalies through prediction errors at the next time step; similarly, LSTM can be used as an alternative to autoregressive models.

[0005] In addition, many novel methods have been proposed in recent years to improve the accuracy of anomaly detection. Examples include MTAD-GAT and GDN, which combine graph neural networks to model features and temporal correlations; MAD-GAN and USAD, which use adversarial training for modeling; MSCRED and CAE-M, which use convolutional neural networks and LSTM as encoder-decoder structures; TranAD and Anomaly Transformer, which use Transformers to further improve temporal representation capabilities; and DCdetector and CARLA, which use contrastive learning to more clearly distinguish between normal and anomalous data point representations.

[0006] Currently, most existing aviation safety incident studies using QAR data generally suffer from the following problems:

[0007] Limited to specific safety events: Existing QAR data anomaly detection mostly targets a single out-of-limit event. For example, hard landing, runway overrun, tail strike, etc. Although these analyses of flight out-of-limit events can prevent aircraft safety accidents, the anomaly detection efficiency of these studies is low and could lead pilots to become QAR pilots, that is, only ensuring that out-of-limit events do not occur during flight while ignoring other potential flight safety issues.

[0008] Reliance on manual annotation: Anomaly detection in QAR data relies heavily on extensive expert experience and a large amount of manually annotated data. Current research largely focuses on supervised learning analysis of flight safety incidents, but this requires a large amount of high-quality annotated data to train models to improve their generalization and accuracy. Collecting large-scale annotated data implies a significant investment of domain expertise and human resources.

[0009] Feature sparsity: Current research primarily focuses on features relevant to specific events, which can lead to feature sparsity. In multidimensional datasets, feature sparsity occurs when only a small subset of features significantly contributes to the task, while the majority remain irrelevant or inactive, increasing the difficulty of effective feature selection and extraction. Since QAR data contains thousands of flight parameters, the choice of QAR parameters varies depending on the specific research task, resulting in different feature densities that significantly impact model performance. Summary of the Invention

[0010] In view of this, the purpose of this invention is to provide a method and system for flight data anomaly detection based on dual-view augmented Transformer. This invention extends the task of flight safety anomaly detection beyond traditional over-limit events, targeting subtle and often undetected anomalies during flight. Given that QAR data is a typical form of time series data, the flight safety problem is reformulated as an anomaly detection task within time series analysis. Furthermore, to address the challenge of over-reliance on expert-labeled data in flight safety research, an unsupervised anomaly detection model specifically designed for QAR data is presented. Finally, considering the sparse characteristics of QAR data, a reconstruction-based time series anomaly detection model architecture is designed.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A method for detecting flight data anomalies based on dual-view enhanced Transformer, the method specifically includes the following steps:

[0013] S1: Collect flight data from the QuickAccess Recorder (QAR);

[0014] S2: Input the collected time-series data into the constructed reconstruction-based time-series anomaly detection model for detection;

[0015] S3: Determine whether the flight data is abnormal based on the model detection output results.

[0016] Furthermore, in step S2, the constructed reconstruction-based temporal anomaly detection model employs a Dual-View Enhanced Transformer (DUVET) to achieve high-precision temporal anomaly detection; specifically, it includes:

[0017] 1) DUVET is based on the Transformer, which has stronger temporal modeling capabilities, and uses the Transformer's encoder as the base encoder of the model.

[0018] 2) DUVET treats data within a time window as a time patch and uses it as the basic unit for model input, extracting local semantic information from the time series and improving the model's time series modeling ability.

[0019] 3) To further address the issue of poor temporal reconstruction capability, DUVET employs a self-attention mechanism to model the global correlation between data points and other data points, and a bell-shaped attention mechanism based on a bell function to model the local correlation between data points and neighboring data points. It also combines the representations of the two views to enhance the temporal reconstruction capability of Transformer for anomaly detection.

[0020] Furthermore, the DUVET comprises four components from bottom to top: (1) Instance Normalization: used to smooth the distribution of input time series data and improve training stability; (2) Channel Independence & Patching: channel independence is used to reduce the number of model parameters and alleviate overfitting problems, while patching is used to enhance local semantics while reducing the computational complexity of the attention mechanism; (3) Dual-View Transformer Encoder: used to capture global and local dependencies between patches and learn the periodicity and trend of time series data; (4) Reconstructed Linear Layer: used to map the representation that combines global and local correlations to the same dimension as the input time series, thereby reconstructing the input.

[0021] Furthermore, the instance normalization is used to adjust the distribution of network layer outputs, specifically including:

[0022]

[0023] Where m is the number of variables in the multivariate time series, i = 1,...,m, and T is the series length; u i and σ i Let be the mean and standard deviation of the time series of the i-th variable, γ and β be the learnable parameters used for scaling and offsetting, respectively, and ε be a small constant used to avoid zero errors caused by division by the denominator. It is worth noting that after obtaining the reconstructed sequence, the output needs to be restored to the original data range through inverse normalization, i.e., the mean and standard deviation are added back into the output prediction. The specific process is as follows:

[0024]

[0025] Inverse normalization is a necessary operation when reconstructing the input time series, making the reconstructed data easier to interpret, visualize, or use for post-processing.

[0026] Furthermore, the channel independently separates the multivariate time series into multiple univariate sequences. One univariate sequence is input into the Transformer backbone network at a time, and each sequence is then reconstructed. The reconstructed sequences are then concatenated as the final output of the model. Specifically, this includes: for the output X∈R after instance normalization... m×T X is divided into m univariate sequences X i ∈R 1×T Let i = 1, ..., m, where each univariate sequence X i Each data point is independently fed into the same Transformer backbone network according to the rules, and the network finally outputs a reconstructed univariate time series.

[0027] The specific process of patching includes: for each univariate time series X i ∈R 1×T Each patch is divided into N time blocks based on its size P and stride length S (the non-overlapping region between two consecutive patches), as calculated below:

[0028]

[0029] Where N is the number of patches, This is a floor operation; a patch sequence is generated from the original univariate sequence using patching. It is worth noting that the last value X of the original sequence needs to be removed before patching.i,end Repeat ∈R S times, and then pad it to the end of the original sequence to ensure the integrity of the patching.

[0030] Furthermore, the dual-view Transformer encoder integrates a dual-view transformer encoder, combining a self-attention mechanism of global-local bell attention to capture global and local dependencies in patches. This dual-view integration ensures the model can distinguish between normal and anomalous data, thus preventing over-reconstruction. Even with feature sparsity and significant error range in the dataset, the model enhances robustness and stability. The task of the reconstructed linear layer is to map the representation, encapsulating the global and local correlations between the two, returning to the same dimension of the input time series, thereby facilitating the reconstruction of the input data. By effectively reconstructing the input model, the information loss caused by feature sparsity is mitigated, thus improving its reconstruction capability and accuracy in anomaly detection.

[0031] Furthermore, the dual-view Transformer Encoder mainly consists of two parts: (1) a Transformer encoder based on self-attention mechanism is used to extract global temporal dependencies between different patches, and learn the periodicity of temporal data in the global view based on the global association between time blocks, thereby capturing the global temporal representation; (2) a Transformer encoder based on bell attention mechanism is used to extract local dependencies between patches and adjacent patches, and learn the trend of temporal data in the local view based on the local association of time blocks, thereby capturing the local temporal representation.

[0032] The self-attention mechanism employs a multi-head attention mechanism, enabling the encoder to simultaneously focus on information in different subspaces and encode the patch from multiple perspectives. Specifically, it includes: first, initializing the query vector (query), key vector (key), and value vector (value) required for the l-th layer self-attention.

[0033]

[0034] in These represent the query, key, and value in self-attention, respectively. Representing the l-th layer Q i ,K i V i The linear projection matrix is ​​calculated, where H represents the number of attention heads. Then, the global attention weights between patches are calculated, and the semantic information in the patch sequence is weighted and combined according to the attention weights to obtain a richer and more accurate representation. The specific implementation is as follows:

[0035]

[0036] in This represents the attention weights learned by the i-th attention head in layer l. The Softmax() function is used to normalize the attention weights. Since the global correlation between patches is considered, therefore... The patch semantic information is more comprehensive. Finally, the outputs of each attention head are concatenated to obtain the final representation:

[0037]

[0038] in It is a learnable parameter matrix;

[0039] Through self-attention, DUVET can capture the global dependencies between patches and further characterize the patches by combining these global associations. For time series data, these global associations often represent periodicity in the time series, which helps DUVET reconstruct the input time series by combining the time series periodicity.

[0040] The bell-shaped attention mechanism first initializes the a, b, and value required by the l-th layer BellAttention:

[0041]

[0042] in Let a and b represent the mapping matrices obtained by the i-th attention head in the l-th layer, respectively. Then, calculate the local attention weights between patches and obtain the patch representation of the fusion temporal trend based on the weights. The specific implementation is as follows:

[0043] a i ,b i =σ(a i )+ε,σ(b i )+ε

[0044] a i ,b i = repeat(a i ), repeat(b i )

[0045]

[0046] Where σ(·) is the sigmoid function used to scale a and b to the (0,1) interval, ε is a small constant to prevent the attention weights from approaching 0; repeat(·) represents repeated sampling, after which a i ,bi ∈R N×N , This represents the local attention weights learned by the i-th attention head in the l-th layer; finally, the outputs of each attention head are concatenated to obtain the final representation:

[0047]

[0048] in It is a learnable weight matrix.

[0049] Furthermore, in this method, the mean squared error loss function (MSE loss) is employed. MSE loss aims to measure the difference between the input sequence X and the reconstructed sequence. The difference in data points at each time step, specifically, it collects the loss of each channel across m time series and averages them to obtain the overall target loss:

[0050]

[0051] Where ||||2 represents the l2 norm, X i,1:T Let represent the value of the i-th univariate sequence from the beginning to the end. The input sequence and the reconstructed output sequence are made as similar as possible by minimizing the mean squared error loss function.

[0052] For normal data points, the reconstruction model is trained on large-scale time-series data, and the model learns normal time-series patterns, making reconstruction easier. However, due to the rarity of outliers, it is difficult for outliers to establish strong correlations with global and local time-series patterns, making reconstruction challenging. Based on this idea, this method uses reconstruction error as the definition criterion for outliers, where X∈R. m×T The abnormal score can be defined as follows:

[0053]

[0054] Where AnomalyScore(X)∈R T It is a time-series anomaly score, where the anomaly score of anomalous data points is typically higher than that of normal data points. Based on the time-series anomaly score, a hyperparameter threshold δ is used to determine whether a data point is anomaly. If the score exceeds the threshold, the output is anomaly.

[0055] Y t =I(AnomalyScore(X) t )≥δ) t=1,...,T

[0056] Where I(·) is an indicator function, which outputs 1 when the condition is met and 0 otherwise.

[0057] The present invention also provides a flight data anomaly detection system based on dual-view enhanced Transformer.

[0058] The beneficial effects of this invention are as follows:

[0059] 1) The method provided by this invention can identify deviations from normal flight patterns in a timely manner by analyzing a large amount of multivariate time series data generated during real flight operations, providing early warning information for airlines and effectively assisting experts in analyzing abnormal flight behavior.

[0060] 2) This invention performs exceptionally well in processing multivariate and univariate time series data, especially for complex aerospace flight data, and can maintain stable performance in high-dimensional sparse data scenarios.

[0061] 3) This invention transforms anomaly detection in flight data into unsupervised time-series anomaly detection, providing new application scenarios and research challenges for the field of time-series data anomaly detection.

[0062] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0063] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0064] Figure 1 The graph is of the sine function;

[0065] Figure 2 Overall architecture diagram of the dual-view enhanced Transformer (DUVET);

[0066] Figure 3 This is a schematic diagram of an independent channel;

[0067] Figure 4 This is a diagram illustrating the patching process.

[0068] Figure 5 This is an architecture diagram of a dual-view Transformer encoder.

[0069] Figure 6 A bell-shaped curve;

[0070] Figure 7 A schematic diagram illustrating the parameter sensitivity of the main hyperparameters in DEVT;

[0071] Figure 8This is a schematic diagram illustrating the impact of patch size (P) on training time.

[0072] Figure 9 A visualization of DUVET anomaly detection;

[0073] Figure 10 This is an example of anomaly detection results for a flight. Detailed Implementation

[0074] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0075] Existing reconstruction-based anomaly detection methods mainly suffer from poor temporal reconstruction capabilities. This is primarily reflected in two aspects: (1) the base encoders used have weak temporal capture capabilities; for example, CNNs, RNNs, and their variants perform poorly on long-sequence data due to their weak ability to capture long-term temporal dependencies; (2) existing methods use data from a single time step as the basic unit, acquiring information from a single time step while ignoring information from adjacent time steps, resulting in insufficient semantic information extracted by the encoder from temporal data. Both of these factors reduce the model's ability to reconstruct temporal data, further affecting the accuracy of anomaly detection. Meanwhile, inspired by real-world observations, when reconstructing a data point, information beneficial to its reconstruction can be obtained from two perspectives. For example... Figure 1 As shown, for the data points to be reconstructed (red data points) of a sine function, reconstruction can be performed using either the global data points marked with a green asterisk or the data points within the local purple dashed box. Therefore, when reconstructing time series data, one can reconstruct it by capturing the periodicity of the time series data globally or by capturing the trend of the time series data locally.

[0076] This invention proposes a reconstruction-based temporal anomaly detection model—Dual-View Enhanced Transformer (DUVET)—to achieve high-precision temporal anomaly detection. Specifically, DUVET leverages the Transformer's stronger temporal modeling capabilities, employing the Transformer's encoder as the model's base encoder. Simultaneously, DUVET treats data within a time window as a time patch, using it as the basic unit of model input to extract local semantic information from the time series, thereby improving the model's temporal modeling ability. To further address the issue of poor temporal reconstruction capability, DUVET models the global correlation between data points and other data points through a self-attention mechanism, and employs a novel bell-shaped attention mechanism based on a bell function to model the local correlation between data points and neighboring data points. It combines the representations from both views to enhance the Transformer's temporal reconstruction capability for anomaly detection.

[0077] The specific architecture of the DUVET model of this invention is as follows: Figure 2 As shown. DUVET consists of four components from bottom to top: (1) Instance Normalization: used to smooth the distribution of input time series data and improve training stability; (2) Channel Independence & Patching: channel independence is used to reduce the number of model parameters and alleviate overfitting problems, while patching is used to enhance local semantics while reducing the computational complexity of the attention mechanism; (3) Dual-View Transformer Encoder: used to capture the global and local dependencies between patches and learn the periodicity and trend of time series data; (4) Reconstructed Linear Layer: used to map the representation that combines global and local correlations to the same dimension as the input time series, thereby reconstructing the input. Specifically, it includes:

[0078] 1) Instance normalization:

[0079] Instance normalization is used to adjust the distribution of network layer outputs. Its main functions include reducing internal covariate shift and improving the model's generalization ability. Most importantly, in training on large-scale data, it stabilizes the gradient flow of the network, ensuring a stable gradient flow during backpropagation, helping to avoid gradient vanishing or exploding problems, and thus improving the model's training stability. This includes:

[0080]

[0081] Where m is the number of variables in the multivariate time series, i = 1,...,m, and T is the series length. i and σ i Let be the mean and standard deviation of the time series of the i-th variable, γ and β be the learnable parameters used for scaling and offsetting, respectively, and ε be a small constant to avoid zero errors caused by division by the denominator. It is worth noting that after obtaining the reconstructed sequence, the output needs to be restored to the original data range through inverse normalization, i.e., the mean and standard deviation are added back into the output prediction. The specific process is as follows:

[0082]

[0083] Inverse normalization is a necessary operation when reconstructing the input time series, making the reconstructed data easier to interpret, visualize, or use for post-processing.

[0084] 2) Independent Channels & Patching:

[0085] Channel independence: Channel independence is a generalization strategy that separates a multivariate time series into multiple univariate sequences. One univariate sequence is input into the Transformer backbone network at a time, each sequence is reconstructed, and the reconstructed sequences are concatenated as the model's final output. For example... Figure 3 As shown, Figure 3 This demonstrates the specific idea of ​​channel independence. Specifically, for the output X∈R after instance normalization, m×T X is divided into m univariate sequences X i ∈R 1×T Let i = 1, ..., m, where each univariate sequence X i Each data point is independently fed into the same Transformer backbone network according to the rules. Finally, the network outputs the reconstructed univariate time series. Most importantly, it is through this parameter-sharing mechanism that the dependencies between different variable sequences are captured. Therefore, channel independence, to a certain extent, enables the model to generalize to the input time series, that is, it allows the model to input time series data with a different number of variables during inference than during training, which greatly improves the model's generalization ability.

[0086] Patching: Initially proposed in computer vision, patching is primarily used to divide images into small patches and then use these patches, rather than individual pixels, as the basic unit. This addresses resource constraints without sacrificing pixel information, thereby improving model performance. Time series tasks typically face the challenges of enormous computational resource costs and limited model performance due to long sequences. In recent years, patching has gained significant traction in the time series field, mainly due to its simplification of the computational complexity of the self-attention mechanism in Transformers. Figure 4 The diagram illustrates the specific process of patching in time series data. Each univariate time series X... i ∈R 1×T Each patch is divided into N time blocks based on its size P and stride length S (the non-overlapping region between two consecutive patches), as calculated below:

[0087]

[0088] Where N is the number of patches, This is a floor operation. A patch sequence is generated from the original univariate sequence using the patching function. It is worth noting that the last value X of the original sequence needs to be removed before patching. i,endRepeat ∈R S times, and then pad it to the end of the original sequence to ensure the integrity of the patching.

[0089] Through patching, the original time series is no longer input as a single time step but rather as a combination of contextual time blocks as tokens into the Transformer. This means that when the model understands the semantics of a given time unit, it no longer extracts information from a single time step but also considers data information from surrounding times. This enhances the Transformer's ability to extract local semantic information from the time series, thereby improving model performance. More importantly, patching significantly reduces the number of tokens input into the Transformer, from the original T to approximately T / s. Therefore, the time and space complexity of the self-attention mechanism in the Transformer are both reduced from O(T^T) to O(T^T). 2 Reduced to O((T / S)) 2 This effect of reducing memory consumption and computational complexity is even more significant in long sequence scenarios. Therefore, patching allows Transformer to process long sequences more quickly and lightly while improving Transformer's ability to perform temporal modeling.

[0090] 3) Dual-view Transformer encoder:

[0091] This component integrates a dual-view transformer encoder that combines a self-attention mechanism of global-local bell attention to capture global and local dependencies in patches. This dual-view integration ensures the model can distinguish between normal and anomalous data, thus preventing over-reconstruction. Even with feature sparsity in the dataset, the model exhibits a significant error range, enhancing robustness and stability.

[0092] 4) Reconstruct the linear layer:

[0093] The task of this layer is to map the representation, encapsulating the global and local relationships between the two, returning to the same dimension of the input time series, thus facilitating the reconstruction of the input data. By effectively reconstructing the input model, the information loss caused by feature sparsity is mitigated, thereby improving its reconstruction capability and accuracy in anomaly detection.

[0094] The architecture of the Dual-View Transformer Encoder is as follows: Figure 5As shown, it mainly consists of two parts: (1) A Transformer encoder based on self-attention mechanism is used to extract the global temporal dependencies between different patches, and learn the periodicity of temporal data in the global view based on the global association between time blocks, thereby capturing the global temporal representation; (2) A Transformer encoder based on bell attention mechanism is used to extract the local dependencies between patches and adjacent patches, and learn the trend of temporal data in the local view based on the local association of time blocks, thereby capturing the local temporal representation.

[0095] 1) Embedding layer:

[0096] The embedding layer is primarily used for data vectorization and dimensionality adjustment; its essence is a linear mapping transformation. After patching, the original univariate time series X... i Segmented into time block sequences Then, a patch is treated as the basic unit of model input, and an embedding operation is performed along the patch size P dimension to map it to a high-dimensional vector to represent the semantic information of the patch. The embedding operation is as follows:

[0097] X i e =W emb T X i p

[0098] in These are the embedding layer weight parameters, d model For the embedded dimension, This represents the dense representation of each patch.

[0099] 2) Encoder overall architecture:

[0100] The dual-view Transformer encoder is characterized by alternating stacks of attention modules and feedforward neural network layers. This stacked structure helps learn the latent relationships between patches from deep, multi-level features. Assume the dual-view Transformer encoder contains L layers, with each layer taking a sequence of patches of length N as input. Then the overall equation for the l-th layer can be expressed as:

[0101] Z l =LayerNorm(Attention(χ l-1 )+χ l-1 )

[0102] χ l=LayerNorm(Feed-Forward(Z l )+Z l )

[0103] in χ represents the output of the l-th layer encoder and the input of the 0th layer encoder. 0 That is, X i e . This represents the latent vector representation of layer l. Attention(·) is used to extract global or local associations between patches. Feed-Forward(·) is used to... l The nonlinear transformation is then performed to extract deeper features. The specific process is as follows:

[0104] Feed-Forward(Z l )=max(0,ZlW1+b1)W2+b2

[0105] in These are the weights and bias parameters of the feedforward neural network layer, respectively.

[0106] 3) Self-attention mechanism:

[0107] This invention uses the self-attention mechanism built into the original Transformer encoder to extract global dependencies between patches, such as... Figure 5 The left-hand architecture is shown. Unlike traditional recurrent neural networks or convolutional neural networks, which are limited by high computational cost and local receptive fields, the self-attention mechanism can simultaneously capture the correlation between each position and other positions, thus possessing global semantic capture capabilities. To enhance the encoder's representation ability of patches, this invention introduces a multi-head attention mechanism, enabling the encoder to simultaneously focus on information in different subspaces and encode patches from multiple perspectives. Specifically, the query vector, key vector, and value vector required for the l-th layer self-attention are first initialized:

[0108]

[0109] in These represent the query, key, and value in self-attention, respectively. Representing the l-th layer Q i ,K i V iThe linear projection matrix is ​​used, where H represents the number of attention heads. Then, the global attention weights between patches are calculated, and the semantic information in the patch sequence is weighted and combined according to these attention weights to obtain a richer and more accurate representation. The specific implementation is as follows:

[0110]

[0111] in This represents the attention weights learned by the i-th attention head in layer l. The Softmax() function is used to normalize the attention weights. Since global correlations between patches are considered, therefore... The patch semantic information is more comprehensive. Finally, the outputs of each attention head are concatenated to obtain the final representation:

[0112]

[0113] in It is a learnable parameter matrix.

[0114] Through self-attention, DUVET can capture the global dependencies between patches and further characterize the patches by combining these global associations. For time series data, these global associations often represent periodicity in the time series, which helps DUVET reconstruct the input time series by combining the time series periodicity.

[0115] 4) Bell-shaped attention mechanism:

[0116] While self-attention excels at modeling global relationships in time series and helps capture features such as periodicity, it still falls short in reconstructing data points at a specific moment or time period. Therefore, to capture the trends in time series, this invention aims to ensure that the model focuses on the influence of surrounding patches when learning dependencies between patches; that is, the current patch establishes strong correlations with surrounding patches, while establishing weak correlations with distant patches. This is reflected in the function graph as a high-in-the-middle, low-at-the-sides characteristic, which is precisely what the bell curve represents, as shown in the example. Figure 6 As shown. Its mathematical representation is:

[0117]

[0118] Where 'a' controls the height of the wave crest and 'b' controls the width of the wave crest.

[0119] This invention proposes a novel attention mechanism—BellAttention—for extracting local dependencies between patches. Similarly, a multi-head attention mechanism is used to enhance patch representation capabilities. First, the a, b, and value required for the l-th layer of BellAttention are initialized:

[0120]

[0121] Where a i ,b i ∈R N , Let a and b represent the mapping matrices obtained by the i-th attention head in layer l, respectively. Then, the local attention weights between patches are calculated, and the patch representation of the fused temporal trend is obtained based on these weights. The specific implementation is as follows:

[0122] a i ,b i =σ(a i )+ε,σ(b i )+ε

[0123] a i ,b i = repeat(a i ), repeat(b i )

[0124]

[0125] Where σ(·) is the sigmoid function used to scale a and b to the (0,1) interval, and ε is a small constant to prevent the attention weights from approaching 0. repeat(·) represents repeated sampling, after which a i ,b i ∈R N×N , This represents the local attention weights learned by the i-th attention head in layer l. Finally, the outputs of each attention head are concatenated to obtain the final representation:

[0126]

[0127] in It is a learnable weight matrix.

[0128] In BellAttention, 'a' primarily controls the maximum attention weight, while 'b' controls the breadth and depth of association between the current patch and its neighboring patches. Through a multi-head attention mechanism, each head learns different 'a' and 'b' values, representing the encoder's focus on different breadths and intensities of temporal trends when representing patches. This local association helps the model represent the semantics of patches from a temporal trend perspective.

[0129] 5) Reconstruct the linear layer:

[0130] After the above dual-view Transformer encoder, for the time series of the i-th variable, the output of the L-th layer encoder using self-attention as the base attention mechanism is denoted as... Similarly, using BellAttention as the output of the last encoder layer of the basic attention mechanism, let be... The average fusion is performed to obtain the final input representation vector Z to the reconstructed linear layer. i The specific calculations are as follows:

[0131]

[0132] in It is a two-dimensional vector, which is converted into a one-dimensional vector by flattening. Finally, it is mapped to the same dimension as the input sequence through a linear layer, as follows:

[0133]

[0134] in b linear ∈R T These are the weights and biases for reconstructing the linear layer, respectively. Output The time series of the i-th variable reconstructed from the model.

[0135] In this invention, the mean squared error loss (MSE loss) is a commonly used loss function in reconstruction-based anomaly detection methods. MSE loss aims to measure the difference between the input sequence X and the reconstructed sequence. The difference in data points at each time step, specifically, it collects the loss of each channel across m time series and averages them to obtain the overall target loss:

[0136]

[0137] Where ||||2 represents the l2 norm, X i,1:T Let represent the value of the i-th univariate sequence from the beginning to the end. The goal is to minimize the MSE loss so that the input sequence and the reconstructed output sequence are as similar as possible.

[0138] For normal data points, the reconstruction model is trained on large-scale time-series data, and the model learns normal time-series patterns, making reconstruction easier. However, due to the rarity of outliers, it is difficult for outliers to establish strong correlations with global and local time-series patterns, making reconstruction challenging. Based on this idea, this invention uses reconstruction error as the definition criterion for outliers, where X∈R m×T The abnormal score can be defined as follows:

[0139]

[0140] Where AnomalyScore(X)∈R T This is a time-series anomaly score, where anomalous data points typically have higher anomaly scores than normal data points. Based on the time-series anomaly score, a hyperparameter threshold δ is used to determine whether a data point is an anomaly. If the score exceeds the threshold, the output is an anomaly, meaning:

[0141] Y t =I(AnomalyScore(X) t )≥δ) t=1,...,T

[0142] Where I(·) is an indicator function, which outputs 1 when the condition is met and 0 otherwise.

[0143] In this embodiment, the proposed DUVET model is evaluated through four experiments. The first type of experiment is a comparative experiment, showcasing the performance improvement of DUVET compared to existing mainstream state-of-the-art anomaly detection methods, and presenting the results through visualization. The second type of experiment is an ablation experiment, used to verify the effectiveness of Patching, channel independence, and BellAttention in DUVET. The third type is a hyperparameter sensitivity analysis experiment, used to verify the impact of hyperparameters on the experimental results. The fourth type of experiment demonstrates the application of DUVET on the QAR dataset to detect and analyze anomalous events in flight parameters.

[0144] Table 1 Dataset Statistics

[0145]

[0146] Six publicly available datasets (NAB is univariate, the others are multivariate) and the QAR professional dataset were used in the experiments. Table 1 summarizes the statistical characteristics of each dataset, where the values ​​in parentheses are the number of sequences in the dataset repository. Experiments were conducted on all sequences in the dataset, and the average scores were recorded. Detailed descriptions of each dataset are as follows:

[0147] (1) NAB is a dataset that tracks multiple real-world data, including temperature sensor readings, CPU utilization of cloud machines, service request latency, and taxi demand in New York City. However, the sequences in this dataset are known to have incorrect outlier labels and were excluded in the experiments.

[0148] (2) The MBA is a collection of electrocardiogram recordings from four patients, containing multiple instances of two different types of abnormalities (supraventricular contractions or premature beats). This is a large-scale dataset that is popular in the data management community.

[0149] (3) SMAP is a dataset of soil samples and telemetry information used by NASA's Mars rovers.

[0150] (4) MSL is a dataset similar to SMAP, but it corresponds to the sensor and actuator data of the Mars rover itself.

[0151] (5)SMD is a five-week dataset containing stack traces of resource utilization of 28 machines in a computing cluster.

[0152] (6) SWaT Dataset: This dataset was collected from a real-world water treatment plant that operated normally for 7 days and experienced abnormal operation for 4 days. The dataset includes sensor values ​​(water level, flow rate, etc.) and actuator operations.

[0153] (7) The QAR dataset records multiple parameters, including aircraft status, pilot operations, and environmental information, throughout the flight process at different sampling frequencies. The entire experiment was implemented using the PyTorch deep learning framework. For all datasets, the time window size input to the DUVET model was set to 100 (T = 100). DUVET contains three encoder layers (L = 3), with a hidden state vector dimension d. model =256, attention head H=8, and the patch size P will be evaluated in the hyperparameter sensitivity analysis experiment. Regarding the model training process, all methods are trained using the Adam optimizer, with an initial learning rate set to 0.001 and a batch size of 128. This invention divides the training time series in all datasets into 70% training data, 10% validation data, and 20% test data, and uses an early stopping strategy based on the model's performance on the validation set to prevent overfitting. If the outlier score at a certain time point is greater than a certain threshold δ, it is marked as an outlier. δ is determined by searching for the optimal threshold based on the model's performance on the validation set. Finally, the test set is evaluated using multiple metrics based on δ and the trained model, including precision, recall, and F1 score.

[0154] Benchmark model and comparison results:

[0155] The comparative experiments compared DUVET with existing unsupervised multivariate time series anomaly detection models. These included 11 baseline models for comparison: OmniAnomaly, CAE-M, SimAD, and RestAD (based on reconstruction); MTAD-GAT and GDN (based on LSTM-VAE using graph neural networks); MAD-GAN (combining adversarial generative methods); DCdetector (based on contrastive learning); and TranAD and Anomaly Transformer (based on Transformers). To ensure a fair and transparent comparison, the experimental settings adhered to the hyperparameters described in the original works of each baseline model.

[0156] Table 2 presents the experimental results of DUVET and other methods on all datasets across three evaluation metrics. It is observed that DUVET consistently outperforms baseline methods in most cases. Specifically, DUVET achieves the best performance on all evaluation metrics across the three datasets (MSL, MBA, and SWAT). Regarding the overall evaluation metric, F1-score, DUVET achieves the best results on 6 out of 7 datasets and a second-best result on the SMAP dataset. The results demonstrate that DUVET can effectively capture complex temporal dependencies and subtle patterns in time series data, exhibiting significant performance across different datasets, particularly on datasets with high feature sparsity and complex temporal dependencies, such as QAR, MSL, MBA, and SWAT. The model's dual-view encoder cleverly balances local and global dependencies, while the Bell attention mechanism effectively captures short-term temporal features by optimizing multi-head attention. Notably, on the QAR dataset, which exhibits significant feature sparsity and subtle patterns, DUVET achieves an impressive F1 score of 94.26, significantly outperforming all baseline models. Furthermore, DUVET demonstrates highly competitive performance on the SMAP, NAB, and SMD datasets, matching or slightly exceeding the performance of leading methods. Transformer-based models, including AnomalyTrans, TranAD, RestAD, and the proposed DUVET, consistently outperform other methods such as SMD, SMAP, and NAB on strongly periodic datasets. Built on a dual-view augmented Transformer architecture, DUVET exhibits superior performance in both univariate and multivariate time series tasks, highlighting the exceptional modeling capabilities of Transformer-based methods for time series data. In contrast, graph neural network-based methods, such as MTAD-GAT and GDN, show relatively weaker performance on these datasets due to their reliance on graph-based representations. While these methods aim to capture complex spatiotemporal dependencies, they often struggle to effectively model long-term temporal relationships and the subtle patterns inherent in periodic datasets. CAE-M struggles on NAB, SWAT, and QAR due to its limited ability to model long-term temporal dependencies, sensitivity to feature sparsity, and inadequate handling of multivariate interactions, which are crucial for these datasets. LSTM-VAE performs poorly on the high-period SMAP dataset because RNN-based temporal encoding models often struggle to effectively capture long-term periodic dependencies in extended sequences, resulting in suboptimal performance. DCdetector demonstrates strong performance on most datasets, leveraging contrastive learning to distinguish between normal and anomalous patterns, enhancing its anomaly detection capabilities. However, despite its dual-branch structure that captures both global and local information, DCdetector still falls short of the performance of DUVET.This difference may stem from DCdetector's over-reliance on the contrastive loss function, which minimizes the impact of outliers but lacks the adaptability to dynamically adjust attention based on different data features. Therefore, this limitation restricts DCdetector's ability to capture subtle and complex patterns in more complex datasets, while DUVET's dual-view attention mechanism and adaptability have proven to be more effective.

[0157] The experimental results not only validate the effectiveness and superiority of DUVET in time series anomaly detection tasks, but also highlight its potential as an effective tool in the field of flight safety. Its ability to generalize on datasets with varying feature densities, from industrial control systems like SWaT to complex flight safety data in QAR datasets, distinguishes it from existing time series anomaly detection models.

[0158] Table 2 shows the experimental results of DUVET and baseline on all datasets, where P, R, and F1 represent precision, recall, and F1 score (%), respectively. The best results are indicated in bold, and the second-best results are indicated by horizontal lines.

[0159]

[0160] Ablation experiment:

[0161] To explore the importance of each component in the DUVET model, an ablation experiment was conducted to eliminate one major component at a time and observe how it affected the F1 score for each dataset. Specifically, firstly, instance normalization in DUVET was removed, and the input sequences were fed directly into the Transformer encoder without normalization. Secondly, instead of using the channel-independent approach, a conventional channel-mixing approach was adopted, where all variable sequences were subjected to uniform patching before being input into the Transformer backbone network, and the reconstructed sequences of all variables were output. Thirdly, patching was eliminated, treating the observation value at each time step as a basic unit (i.e., a token input into the Transformer). Finally, the bell-shaped attention mechanism for learning temporal trends under a local view was removed, retaining only the self-attention mechanism for learning global temporal correlations.

[0162] Table 3 shows the ablation results of DUVET, where F1 represents the F1 score (%), and the best results are indicated in bold.

[0163]

[0164] The final experimental results are shown in Table 3. From the table, we can observe that:

[0165] (1) DUVET's performance slightly decreased after removing instance normalization. The main reason is that the input distribution of each layer changes drastically during the training of deep neural networks, which leads to model training instability. Instance normalization helps to reduce the drastic changes, improve training stability, and further improve model performance.

[0166] (2) Channel independence helps improve model performance, mainly because models based on the channel independence concept are more adaptable. Specifically, each time series is processed separately by the Transformer, which means that different variable time series can learn different attention patterns and then be reconstructed according to their respective attention maps. In contrast, in the channel fusion method, all time series share the same attention pattern, which can greatly reduce the model's performance if different variable time series have different temporal patterns.

[0167] (3) When the input sequence is directly input into the Transformer without patching, DUVET's F1 score decreases slightly across all datasets. This further confirms that patching can increase the receptive field of the Transformer, thereby better capturing the local semantics of time-series data.

[0168] (4) The performance of DUVET after removing Bell Attention drops significantly on the four datasets. This is mainly because the input sequence is reconstructed only through temporal periodicity in the global view. When the periodicity of the time series is not obvious, its performance is unsatisfactory. With Bell Attention, the reconstruction of the sequence can also be combined with the trend reconstruction of the time series in the local view, thereby enhancing the reconstruction capability of the time series.

[0169] In short, each of the major components in DUVET plays a crucial role and has a positive impact on DUVET's performance.

[0170] Hyperparameter influence experiment:

[0171] Figure 7 and Figure 8 This demonstrates how different hyperparameters affect the F1 score and training time of the DUVET model on six datasets. From Figure 8 It can be seen that the F1 score fluctuates with changes in parameter P, and there is an optimal P value for each dataset. This indicates that the patch size directly affects model performance. Furthermore, as... Figure 9As shown, the training time per epoch decreases significantly with increasing P, primarily due to Patching simplifying the computational complexity of the attention mechanism in the Transformer. Overall, the choice of P affects the overall performance of DUVET, but the model still maintains superior performance compared to other baseline methods at different P values. Moreover, Patching significantly reduces the training time and memory consumption of the DUVET model, thereby improving training efficiency. Figure 8 This demonstrates the F1 score of the DUVET model on various datasets as a function of the hidden vector dimension (d). model The graph shows the variation in the number of attention heads and the F1 score, indicating that the DUVET model has strong robustness. Despite the optimal d... model It may vary depending on the dataset, but it usually performs well in the range of {128,512}.

[0172] Visual analysis:

[0173] To more intuitively demonstrate DUVET's anomaly detection capabilities on time series data, a series of visualization experiments showcase DUVET's performance on the univariate time series dataset NAB and the multivariate time series datasets SMD and MB, such as... Figure 9 As shown in the figure, for multivariate time series datasets, a summary sequence is generated by averaging all variable dimensions to provide a more concise representation. Furthermore, if any dimension is anomalous at a given time point, that time point is also marked as anomalous in the overall sequence. In the figure, blue rectangles represent actual anomalous data points, and red areas represent predicted anomalous points. The DUVET model determines whether a data point is anomalous based on its anomalous score (reconstruction error) and a corresponding threshold. The visualization clearly shows that the DUVET model can effectively identify anomalous regions in time series data. Moreover, subsequences in the original sequence that deviate from the expected time pattern have higher anomalous scores at these time points, indicating that the reconstruction-based DUVET model is more sensitive to data points that deviate from the expected normal pattern of the overall sample, thus making it easier to detect anomalous points. In summary, the visualization of the detection results intuitively verifies the state-of-the-art capabilities of the DUVET model.

[0174] Application of DUVET in flight safety analysis:

[0175] QAR data, as a typical representative of high-dimensional time series datasets, is characterized by complexity, specialization, and a wide range of parameters. The diversity of QAR parameters leads to equally diverse anomaly detection applications, such as engine fault detection, over-limit event detection, and pilot operation analysis. Therefore, from a technical perspective, time series anomaly detection based on QAR data often presents unique challenges, such as varying feature density levels. DUVET effectively addresses these complexities, demonstrating stable and generalizable performance on multiple publicly available datasets and QAR datasets, making it particularly suitable for the diverse technical requirements of QAR anomaly detection. From an application perspective, QAR-based anomaly detection research meets different research needs, each emphasizing different combinations of QAR parameters. To rigorously evaluate the performance of DUVET, this invention uses QAR to conduct experiments with various parameter combinations, thereby achieving a comprehensive evaluation of different configurations.

[0176] Table 4. Description of QAR parameters used for anomaly detection.

[0177]

[0178] To detect and further analyze anomalous data in the flight parameter sequence, this experiment applied the DUVET model to the QAR dataset. Specifically, the dataset used in this experiment consisted of 25,914 samples from Airbus A321 flights at Beijing Airport in China. The training, validation, and test sets were divided in a 7:1:2 ratio. The experiment focused on the critical approach and leveling phases during aircraft landing, specifically the 20 seconds before touchdown. This was chosen because these 20 seconds represent the period from 200 feet to landing. During this phase, the pilot needs to ensure the aircraft gradually approaches the runway at an appropriate rate of descent and speed, maintaining a stable attitude for landing. At this time, the pilot needs to accurately assess and adjust wind speed, altitude, speed, and landing point to ensure a safe and successful landing. The flight parameters selected in the experiment primarily reflect the crew's operations and the aircraft's own state, as detailed in Table 4.

[0179] Experimental results for univariate and multivariate time series anomaly detection are shown in Tables 5 and 6, respectively. In the univariate scenario, DUVET achieved excellent F1 scores across all QAR parameters, with the highest PITCH score reaching 98.25%, demonstrating its effectiveness in capturing single time series anomalies. For the multivariate experiments, different combinations of QAR parameters were considered by selecting multiple subsets of parameters from the 10 parameters in Table 5. As shown in Table 6, the model consistently provided stable F1 scores, achieving the best performance (F1 = 96.08%) when using all parameters. Experimental results indicate that DUVET has strong generalization ability, effectively handling different feature combinations and maintaining high accuracy in scenarios with varying feature complexity.

[0180] Table 5. Anomaly detection experimental results for QAR data.

[0181]

[0182] Table 6. Experimental results of multivariate anomaly detection using QAR data.

[0183]

[0184] To further illustrate DUEVT's ability to detect flight safety risks, this embodiment visualizes the anomaly detection results for 10 flight parameters in an example flight, such as... Figure 10 As shown, the parameter curves for this flight are compared with the group pattern, and detected anomalies are contained within the red dashed boxes. Note that the curve length (number of data points) for each QAR parameter varies due to different sampling frequencies. The lower half of each subplot shows the outlier curve and the corresponding threshold (red horizontal line), with outlier data points marked in red. It can be observed that for some parameters, including GS, IVV_CA, and N11, the results show no significant anomalies. Conversely, for pilot operating parameters, including pitch control (PITCH_CPT) and roll control (ROLL_CPT), they exhibit significant fluctuations throughout the final approach phase, deviating markedly from their normal range. The rudder control parameter (RUD_PEDA_POS) deviates from the normal range a few seconds before landing. Figure 10 (e), (f), (g), and (h) show four key parameters of the flight status: roll, rudder position, pitch, and vertical load (VRTG). From this, we can observe that in the initial phase of the 20-second descent, the aircraft exhibits an abnormally low roll, followed by an unstable rudder position. Figure 10As shown in (g), the aircraft's pitch angle remained within the normal range during the mid-phase, but exhibited significant fluctuations deviating from its overall safe trend, followed by an anomalous pattern in the final moments before landing. Furthermore, as... Figure 10 As shown in (h), the vertical load (VRTG) exhibits significant fluctuations, deviating from the normal range before landing, thus indicating an increased risk of a hard landing.

[0185] The visualization results demonstrate that DUVET can identify transient and sudden risks from a local perspective, as well as safety hazards throughout the landing phase. The results lead to the conclusion that correct pilot operation and stable flight conditions are crucial for a smooth landing, thereby reducing the likelihood of flight safety incidents such as hard landings and tail strikes. These findings contribute to improving flight safety. On one hand, by identifying abnormal flight patterns in real time, DUVET enables crew members to take timely corrective actions, thereby reducing potential risks and ensuring the safety of passengers and crew. On the other hand, airlines can utilize DUVET to develop standardized safety protocols, guide pilot operations, and proactively respond to abnormal behaviors, ultimately improving pilot performance and operational safety.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting flight data anomalies based on dual-view enhanced Transformer, characterized in that: The method specifically includes the following steps: S1: Acquire flight data from the Quick Access Recorder (QAR); S2: Input the collected time-series data into the constructed reconstruction-based time-series anomaly detection model for detection; S3: Determine whether the flight data is abnormal based on the model detection output results; In step S2, the constructed reconstruction-based temporal anomaly detection model employs dual-view augmented TransformerDUVET to achieve high-precision temporal anomaly detection; specifically, it includes: DUVET is based on the Transformer, which has stronger temporal modeling capabilities, and uses the Transformer's encoder as the base encoder of the model. DUVET treats data within a time window as a time patch, using it as the basic unit for model input, extracting local semantic information from the time series, and improving the model's time series modeling ability. DUVET models the global correlation between data points and other data points through a self-attention mechanism, and models the local correlation between data points and neighboring data points through a bell-shaped attention mechanism based on a bell function. It also combines the representations of the two views to enhance the Transformer's temporal reconstruction capability for anomaly detection. The DUVET consists of four components from bottom to top: (1) Instance normalization: used to smooth the distribution of input time series data and improve training stability; (2) Channel independence & Patching: channel independence is used to reduce the number of model parameters and alleviate overfitting problems, while patching is used to enhance local semantics while reducing the computational complexity of the attention mechanism; (3) Dual-view Transformer encoder: used to capture global and local dependencies between patches and learn the periodicity and trend of time series data; (4) Reconstructed linear layer: used to map the representation that combines global and local correlations to the same dimension as the input time series, thereby reconstructing the input. The dual-view Transformer encoder mainly consists of two parts: (1) a Transformer encoder based on self-attention mechanism is used to extract the global temporal dependencies between different patches, and learn the periodicity of temporal data in the global view based on the global association between time blocks, thereby capturing the global temporal representation; (2) a Transformer encoder based on bell attention mechanism is used to extract the local dependencies between patches and adjacent patches, and learn the trend of temporal data in the local view based on the local association of time blocks, thereby capturing the local temporal representation. The self-attention mechanism employs a multi-head attention mechanism, enabling the encoder to simultaneously focus on information in different subspaces and encode the patch from multiple perspectives. Specifically, it includes: first, initializing the first... The query vector, key vector, and value vector required for the self-attention layer are: in These represent the query, key, and value in self-attention, respectively. They represent the first layer The linear projection matrix, This represents the number of attention heads; then, the global attention weights between patches are calculated, and the semantic information in the patch sequence is weighted and combined according to the attention weights to obtain a richer and more accurate representation. The specific implementation is as follows: in Indicates the first Layer The attention weights learned by each attention head. The function is used to normalize the attention weights; finally, the outputs of each attention head are concatenated to obtain the final representation: in It is a learnable parameter matrix; The bell-shaped attention mechanism first initializes the... The Bell Attention layer requires , And value: in , They represent the first Layer Attention Received The mapping matrix is ​​then used; subsequently, the local attention weights between patches are calculated, and the patch representation of the fused temporal trend is obtained based on the weights. The specific implementation is as follows: in It is used to and The sigmoid function scaled to the (0,1) interval. It is a very small constant to prevent the attention weight value from approaching 0; This represents repeated sampling; after repeated sampling , Indicates the first Layer The local attention weights learned by each attention head are analyzed; finally, the outputs of each attention head are concatenated to obtain the final representation. in It is a learnable weight matrix.

2. The flight data anomaly detection method based on dual-view enhanced Transformer according to claim 1, characterized in that: The instance normalization is used to adjust the distribution of network layer outputs, specifically including: in It is the number of variables in a multivariate time series. , The sequence length; and It is the first Mean and standard deviation of time series of each variable. and These are the learnable parameters used for scaling and offset, respectively. This is a very small constant used to avoid zero errors caused by division by the denominator; after obtaining the reconstructed sequence, the output needs to be restored to the original data range through an inverse normalization operation, that is, the mean and standard deviation are added back to the output prediction. The specific process is as follows: Inverse normalization is a necessary operation when reconstructing the input time series, making the reconstructed data easier to interpret, visualize, or use for post-processing.

3. The flight data anomaly detection method based on dual-view enhanced Transformer according to claim 2, characterized in that: The channel independently separates the multivariate time series into multiple univariate sequences. One univariate sequence is input into the Transformer backbone network at a time, and each sequence is then reconstructed. The reconstructed sequences are then concatenated as the final output of the model. Specifically, this includes the output after instance normalization. , Classified as A single variable sequence Each univariate sequence Each data point is independently fed into the same Transformer backbone network according to the rules, and the network finally outputs a reconstructed univariate time series. ; The specific process of patching includes: each univariate time series It will depend on the size of the patch. and stride length Divided into Each time block is calculated as follows: in The number of patches. This is a floor operation; a patch sequence is generated from the original univariate sequence using patching. Before patching, the last value at the end of the original sequence needs to be removed. Repeat S times, then pad the result to the end of the original sequence to ensure the integrity of the patching.

4. The flight data anomaly detection method based on dual-view enhanced Transformer according to claim 3, characterized in that: The dual-view Transformer encoder integrates a dual-view transformer encoder, combining a global self-attention mechanism and a local bell-shaped attention mechanism to capture global and local dependencies. The task of the reconstructed linear layer is to map the representation, which encapsulates the global and local correlations between the two, returning to the same dimension of the input time series, thereby facilitating the reconstruction of the input data and mitigating the information loss caused by feature sparsity by effectively reconstructing the input model.

5. The flight data anomaly detection method based on dual-view enhanced Transformer according to claim 4, characterized in that: In this method, the mean squared error loss function is used to measure the input sequence. and reconstructed sequence The difference in data points at each time step, specifically, it The loss of each channel is collected over each time series and averaged to obtain the overall target loss: in express Norm, Indicates the first The values ​​of a single variable sequence from the beginning to the end are obtained by minimizing the mean squared error loss function to make the input sequence and the reconstructed output sequence as similar as possible. This method uses reconstruction error as the criterion for defining anomalies. The abnormal score can be defined as follows: in It is a time-series anomaly score, where the anomaly score of outlier data points is typically higher than that of normal data points; based on the time-series anomaly score, a hyperparameter threshold is used. This determines whether a data point is outlier. If the score exceeds a threshold, the output is "outlier," meaning: in This is an indicator function that outputs 1 when the condition is met and 0 otherwise.

6. A flight data anomaly detection system based on dual-view enhanced Transformer, characterized in that: The system employs the method described in any one of claims 1 to 5.

Images