Anomaly detection method for high-dimensional dynamic data stream based on stacked habituation autoencoders

Abstract

The present invention relates to a high-dimensional dynamic data stream anomaly detection method based on a stacked habituation autoencoder. The method comprises an initialization phase and an online anomaly detection phase. In the initialization phase, a static anomaly detection algorithm is used to perform anomaly detection on historical data to generate anomaly candidate sets and normal data. The stacked habituation autoencoder is then trained using micro-clusters generated from the normal data. In the online anomaly detection phase, a sliding window is used to separate the dynamic data stream. A stacked habituation autoencoder is used to perform data dimensionality reduction on the current window data and the anomaly candidate set. A joint probability density estimator is then used to perform online anomaly detection. The anomaly detection generates true outliers, anomaly candidate sets, and normal data. The normal data is used to continuously update the stacked habituation autoencoder, and the anomaly candidate sets are used for anomaly detection in the next window. The method is advantageous in improving the anomaly detection effect of high-dimensional dynamic data streams and reducing anomaly detection time.

Description

Technical Field

[0001] The present invention belongs to the technical field of data stream anomaly detection, and specifically relates to a high-dimensional dynamic data stream anomaly detection method based on a stacked habituation autoencoder. Background Art

[0002] With the rapid development of current communication technologies, the application of complex, heterogeneous networks of various sensors is becoming increasingly widespread. These large-scale applications generate massive amounts of high-dimensional, dynamic data streams. These high-dimensional, dynamic data streams exist as continuous, unbounded data sequences. Implicit or explicit timestamps keep them in an orderly fashion, resulting in data points in the data stream being available only at certain points in time, making random access to the entire data stream impossible. This makes anomaly detection in these high-dimensional, dynamic data streams even more challenging. Developing an effective anomaly detection framework for high-dimensional, dynamic data streams to improve anomaly detection accuracy is a pressing need in many application areas, such as the Industrial Internet of Things. Anomaly detection involves identifying data points or patterns that do not conform to expected behavior. In fields such as industrial production, healthcare, finance, and social networking, anomaly detection technology can be applied to a range of critical system and application security issues, addressing practical problems such as intrusion detection, fault prevention, and fraud detection. Consequently, detecting anomalies in these constantly changing data streams is gaining increasing attention from both industry and academia.

[0003] However, online anomaly detection in data streams faces multiple challenges, both from the challenges of anomaly detection and from the difficulties of stream learning. In static data anomaly detection algorithms, the model learns the behavior of "normal data" and then detects anomalies in that data. However, in data streaming scenarios, there may be difficulties such as concept drift and new class detection, making it impossible to store the entire data stream and requiring the model to be continuously updated over time.

[0004] Currently, there are many micro-clustering-based data stream anomaly detection algorithms. For example, DenStream is a classic micro-clustering-based data stream anomaly detection algorithm, but it struggles to adapt to high-dimensional and distributionally changing data streams. MCOD uses a nearest neighbor search to determine whether a data point is an outlier and uses micro-clustering to accelerate the nearest neighbor search. However, when a single data point forms a micro-cluster, MCOD's nearest neighbor search becomes a linear search. M_MCOD improves on MCOD by incorporating the minimum detection principle, leveraging the continuity of the data stream to find nearest neighbors close to the arrival time of the data point. However, the algorithm's drawback is that the nearest neighbor search becomes inefficient when a large number of data points are stored in the candidate list. While these methods are effective for anomaly detection in data streams, they still face several challenges in the high-dimensional, dynamic data streams of the Internet of Things: 1) There is a lack of efficient anomaly detection frameworks for high-dimensional, dynamic data streams; 2) Conventional micro-clustering models cannot detect anomalies in dynamic data streams based on the changing distribution of the data; and 3) Most current algorithms still fail to effectively mitigate the concept drift and evolution issues in data stream anomaly detection. Summary of the Invention

[0005] The purpose of the present invention is to provide a high-dimensional dynamic data stream anomaly detection method based on stacked habituation autoencoders, which is beneficial to improving the anomaly detection effect of high-dimensional dynamic data streams and reducing the anomaly detection time.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a high-dimensional dynamic data stream anomaly detection method based on stacked habituation autoencoders, comprising an initialization phase and an online anomaly detection phase;

[0007] In the initialization phase, a static anomaly detection algorithm is used to detect anomalies in historical data, generate anomaly candidate sets and normal data, and use the micro-clusters generated from the normal data to train the stacked habituation autoencoder.

[0008] In the online anomaly detection stage, the dynamic data stream is separated by a sliding window, and the stacked habituation autoencoder is used to reduce the data dimension of the current window data and the anomaly candidate set. Then, a joint probability density estimator is used for online anomaly detection. Anomaly detection generates real outliers, anomaly candidate sets and normal data. The normal data is used to continuously update the stacked habituation autoencoder, and the anomaly candidate set is used for anomaly detection in the next window.

[0009] Furthermore, the stacked habituation autoencoder is constructed by adding a habituation autoencoder on the basis of the stacked autoencoder. For a stacked habituation autoencoder, it is necessary to construct it level by level. The first level is the habituation autoencoder, which includes a three-layer structure:

[0010]

[0011]

[0012]

[0013] Where x is the input data, is the projection data, LN is the habituation activation function, g1 is the encoding layer activation function, W1 and b1 are the encoding layer network parameters, h is the hidden layer data, g2 is the decoding layer activation function, W2 and b2 are the decoding layer network parameters, is the output data;

[0014] The input data x is first processed by habituation to increase the generalization ability of the model and improve the accuracy of outlier detection. At the same time, ReLU is used as the activation function in each layer, and L2 normalization is added in the training process of the hidden layer h. The decoding layer is used to reconstruct the input data x. During the training phase, the mean square error n is the total number of data points;

[0015] Each level of the stacked habituation autoencoder is trained step by step, and the hidden layer of the previous level of autoencoder is used as the input layer of the next level of autoencoder. The stacked habituation autoencoder is constructed by stacking step by step.

[0016] Furthermore, the historical data is set as window 0. In the initialization phase, the data in window 0 is trained, and the stacked habituation autoencoder, micro-cluster set, and anomaly candidate set are returned. The specific method is: cluster the data in window 0 to obtain normal data and abnormal data, construct a micro-cluster set for the normal data, train the stacked habituation autoencoder at the same time, and put the abnormal data into the anomaly candidate set.

[0017] Furthermore, in the online anomaly detection stage, the data stream is separated into window data through a sliding window, the n-1th window data is trained to obtain basic components, and the basic components are used to perform online anomaly detection on the data of the n-1th window; after the end of the n-1 window, a network model is obtained, namely the stacked habituation autoencoder and two sets, namely the micro-cluster set and the anomaly candidate set; due to the concept drift phenomenon in the data stream, anomaly detection is performed on the anomaly candidate set at the same time when processing the n-th window data. The specific method is: first, the stacked habituation autoencoder is used to encode and reduce the dimension of the data, and then the anomaly detection factor of each data point is calculated, and the data is evaluated according to the anomaly detection factor threshold.

[0018] Furthermore, for abnormal data points, they are not processed immediately, but an abnormal review strategy is used. Specifically, when the data anomaly detection factor in the current window is lower than the specified anomaly detection factor threshold, the anomaly is not reported and confirmed immediately, but is regarded as a candidate outlier and re-evaluated in future windows. At the same time, anomaly weights are used to track each candidate anomaly data. The increase or decrease of the anomaly weight of the data point depends on the relationship between the data point and the specified anomaly detection factor threshold in the current window. If the anomaly weight of the data point is equal to the custom weight W, the data point is immediately returned as an outlier. If the anomaly weight of the data point is less than zero, the similarity score of the data point is calculated, and the best micro-cluster is selected for addition.

[0019] Furthermore, the Mahalanobis distance is used as the normal data similarity score evaluation, and the Mahalanobis distance is defined as:

[0020]

[0021] Where m is the mean of the micro-cluster and Λ is the covariance matrix of the micro-cluster.

[0022] Furthermore, after the window ends, the network model and the set are dynamically updated to detect anomalies in the next window. In the current window, the similarity of normal data is evaluated. Due to the dynamic nature of the data stream, the micro-cluster set is updated. The update formula is as follows:

[0023]

[0024]

[0025]

[0026] in, represents the number of historical data in the i-th micro-cluster, Indicates the number of new micro-clusters added to the i-th micro-cluster in the current window; represents the mean of the i-th micro-cluster, Represents the mean of the data newly added to the i-th micro-cluster in the current window; represents the covariance matrix of the i-th micro-cluster, Represents the covariance matrix of the data newly added to the i-th micro-cluster in the current window;

[0027] In addition to updating individual micro-clusters, the micro-cluster set is also updated. The micro-cluster set is a LRU queue. The update method is: first, a merge operation is performed to merge micro-clusters with higher similarity and higher activity. Subsequently, an LRU operation is performed to keep the overall number of micro-clusters in the set within the set range. Due to the dynamic evolution of the data stream, new categories will be generated over time, so new categories are searched in the anomaly candidate set. At the same time, the stacked habituation autoencoder is trained using micro-clusters to decode and reduce the dimensionality of the data in the post-order window.

[0028] Compared with existing technologies, this invention has the following advantages: It addresses the problem of anomaly detection in large-scale, high-dimensional data streams and proposes a high-dimensional dynamic data stream anomaly detection method based on a stacked habituation autoencoder. This method uses a stacked habituation autoencoder for training on historical window data, effectively detecting similar anomalies. Furthermore, within the current window data, the stacked habituation autoencoder is used to reduce the data dimensionality, and anomaly detection is performed using a candidate set strategy combined with a joint kernel density estimator, reducing the time required for anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the change in similarity of data points before and after habituation processing in an embodiment of the present invention.

[0030] Figure 2 It is a block diagram of the implementation principle of the method of an embodiment of the present invention.

[0031] Figure 3 2 is a schematic diagram of the structure of a three-stage stacked habituation autoencoder in an embodiment of the present invention.

[0032] Figure 4 2 is a schematic diagram comparing the average accuracy of different algorithms on the synthetic dataset (a) and the real datasets CIFAR-10 (b), ALOI (c), and GSAD (d) in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the impact of different parameter sizes on experimental accuracy under two real data sets in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] This section introduces the basics of habituation and defines the habituation activation function. It explains its role in anomaly detection in high-dimensional dynamic data streams. It also describes the basic kernel density estimation and then defines the variant kernel density estimator required to calculate the anomaly detection factor.

[0038] 1. Habituation and its application

[0039] Habituation is a simple form of memory that suppresses neuronal responses to repeated neutral stimuli. This process is crucial for helping organisms focus on the most salient and novel features in their environment. Because of its properties, habituation is widely used in robotics and to enhance object recognition in deep neural networks. These studies reveal the benefits of using habituation in deep learning applications and demonstrate that models that include additional habituation features can lead to more powerful algorithms.

[0040] Definition 1 Habitual activation function Let the i-th input neuron be x I , the corresponding i-th projection neuron is s i Let the local habituation neuron be l, and its corresponding weight be w i Then x i The habituation activation function LN(.) is defined as:

[0041] s i =LN(x i )=max(x i -lw i ,0)

[0042] Without loss of generality, we set l to 1, then

[0043] s i =max(x i -w i ,0)

[0044] At time t, the habituation rate α∈[0,1], the recovery rate β∈[0,1], then w i The update rule is:

[0045]

[0046] Studies have shown that after becoming accustomed to the scent of a flower, even an untrained nose can distinguish the scents of two other flowers more accurately than an unaccustomed control group. Habituation allows the nose to more easily distinguish similar scents. If this feature is applied to streaming anomaly detection, the habituated model can better capture the characteristic relationships between data points. High-dimensional dynamic data streams are represented by S, where S = [S1, S2, ..., S k ,...], each data is processed by the habituation activation function LN(.), for similar data points S A , S B , S C , S D , where S A and S B Belong to the same category, S C and S D Belonging to different categories, after habituation, the similarity of data points of the same category remains unchanged, and data points of different categories will be easier to distinguish, such as Figure 1 shown. Figure 1 In, S A , S B , S C , S D Represents n-dimensional data. If the horizontal directions are all colored, it means that the data are similar in this dimension.

[0047] 2. Kernel Density Estimator and Anomaly Detection Factor

[0048] To address the concept drift and anomaly detection problems in data streams, we use the probability density function f(x) to perform dynamic probability density estimation on the current window data. Therefore, the probability density function f(x) always reflects the latest distribution of the current data stream. In addition, to address the uncertainty of the data stream, we use the kernel density estimation (KDE) as the density estimator. Kernel density estimation is a parameterless estimator that is used to estimate the probability density function of a random variable. The probability density function f(x) of the kernel density estimation is defined as:

[0049]

[0050] Where n is the total number of data points, x i is the i-th data point, x is a random variable, k(x, x i ) is the kernel function.

[0051] The kernel function reflects the data point x i The Gaussian kernel function, as the basic kernel function of kernel density estimation, shows a smoother estimate of the probability distribution around the Gaussian kernel function. The Gaussian kernel function is defined as follows:

[0052]

[0053] Where x is a random variable, x i is the i-th data point, d is the dimension of the data, H is the diagonal matrix representing the bandwidth of the kernel function, x j represents the value of random variable x in dimension j, x ij Represents the data point x i The value of h in dimension j j represents the bandwidth in dimension j.

[0054]

[0055] Bandwidth mainly controls x i The expansion of surrounding data points, for each dimension j, bandwidth h i The larger the x i The more points there are around, the more likely the probability distribution of the data point will be from +∞ to -∞

[29] , but for the Gaussian kernel function, the probability density of most data points will only be distributed in x i We use Scott's rule and take the following formula to calculate the bandwidth of each dimension:

[0056]

[0057] where n is the total number of data points, d is the dimension of the data, and σ i is the standard deviation on dimension j.

[0058] Definition 2 Window Probability Density Function Sliding windows are used to separate high-dimensional dynamic data streams. The Gaussian kernel function is applied to the data point x in the current sliding window to obtain the window probability density function, which is defined as follows:

[0059]

[0060] Where n is the number of data points in the current sliding window.

[0061] Definition 3 Micro-cluster A micro-cluster is a triplet that can preserve the clustering characteristics of normal data in the window. The micro-cluster is defined as follows:

[0062] MC=(C,M,Λ)

[0063] where C represents the number of data points in the microcluster, M represents the mean vector, and Λ represents the covariance matrix.

[0064] Whether a high-dimensional dynamic data stream is an outlier is determined jointly by the micro-clustering kernel density estimator and the window kernel density estimator. The micro-clustering kernel density estimator is mainly used to determine the historical anomaly score of the data point, and the window kernel density estimator is used to determine the current anomaly score of the data point.

[0065] Definition5 Microcluster probability density function Let the i-th microcluster MC i The mean vector is M i , the number of its data points is C i , therefore, the micro-cluster probability density function of data point x is defined as follows:

[0066]

[0067] Where C represents the total number of all micro-cluster data points, m represents the number of micro-clusters, and M ij Indicates MC i The mean vector M i The value in dimension j.

[0068] Definition 6 Joint probability density function Joint probability function f union (x) is determined by the micro-cluster probability density function f mc (x) and the window probability density function f window (x) is defined as follows:

[0069] f union (x) = f mc (x)+f window (x)

[0070] Concept drift often occurs in dynamic data streams. In order to determine whether a data point x is an anomaly, we calculate the anomaly detection factor of the point The anomaly detection factor can quickly and accurately determine whether the data point x is an outlier. In order to more accurately determine the outlier, the anomaly detection factor threshold θ is set. threshold .

[0071]

[0072] Where 0<ξ<1, p avg is the joint probability density mean of all data points.

[0073] like Figure 2 As shown, this embodiment provides a high-dimensional dynamic data stream anomaly detection method based on stacked habituation autoencoders, including an initialization stage and an online anomaly detection stage.

[0074] In the initialization stage, a static anomaly detection algorithm is used to detect anomalies on historical data to generate anomaly candidate sets and normal data, and the stacked habituation autoencoder is trained using micro-clusters generated from normal data.

[0075] In the online anomaly detection stage, the dynamic data stream is separated by a sliding window, and the stacked habituation autoencoder is used to reduce the data dimension of the current window data and the anomaly candidate set. Then, a joint probability density estimator is used for online anomaly detection. Anomaly detection generates real outliers, anomaly candidate sets and normal data. The normal data is used to continuously update the stacked habituation autoencoder, and the anomaly candidate set is used for anomaly detection in the next window.

[0076] The present invention adds a habituation autoencoder on the basis of the stacked autoencoder to construct a stacked habituation autoencoder. The three-level stacked habituation autoencoder constructed in this embodiment is as follows Figure 3 As shown in the figure, a stacked habituation autoencoder needs to be constructed step by step. The first level is the habituation autoencoder, which consists of three layers:

[0077]

[0078]

[0079]

[0080] Where x is the input data, is the projection data, LN is the habituation activation function, g1 is the encoding layer activation function, W1 and b1 are the encoding layer network parameters, h is the hidden layer data, g2 is the decoding layer activation function, W2 and b2 are the decoding layer network parameters, is the output data.

[0081] The input data x is first processed by habituation to increase the generalization ability of the model and improve the accuracy of outlier detection. At the same time, ReLU is used as the activation function inside each layer because it is simple and computationally inexpensive and does not suffer from the vanishing gradient problem. In addition, L2 normalization is added during the training of the hidden layer h because adding L2 normalization in the potential hidden feature space can improve the distinguishability of different categories in various deep autoencoder models. The decoding layer is used to reconstruct the input data x. During the training phase, the mean square error n is the total number of data points. The function is minimized to reduce reconstruction error and optimize the network model and parameters. Each level of the stacked habituation autoencoder is trained step by step, using the hidden layer of the previous level as the input layer of the next level. By stacking the layers and performing additional overall training, while fine-tuning the parameters, a stacked habituation autoencoder is constructed.

[0082] This method constructs a data stream anomaly detection framework, consisting of three components: creating an initialization component (Algorithm 1), online anomaly detection (Algorithm 2), and an update component (Algorithm 3). We set the historical data as window 0. During the initialization phase, we train the data in window 0, returning a stacked habituated autoencoder, a set of microclusters, and a set of anomaly candidates. Specifically, we use KNN-BLOCK-DBSCAN (Y. Chen, L. Zhou, S. Pei, Z. Yu, Y. Chen, X. Liu, J. Du, and N. Xiong, “Knn-block dbscan: Fast clustering for large-scale data,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2019) to cluster the data in window 0, obtaining normal and anomaly data. Microclusters are constructed from the normal data, while the stacked habituated autoencoder is trained. The anomaly data is then placed in the anomaly candidate set.

[0083]

[0084]

[0085] During the online anomaly detection phase, a sliding window is used to separate the data stream into windowed data. The basic components are trained on the n-1th window data, and then used to perform online anomaly detection on the nth window data. After the n-1 window is completed, a network model (a stacked habituated autoencoder) and two sets (a micro-cluster set and anomaly candidate set) are obtained. Due to the concept drift phenomenon in the data stream, anomaly detection is performed on the anomaly candidate set while processing the nth window data. The specific method is as follows: first, the stacked habituated autoencoder is used to encode and reduce the dimensionality of the data. Then, the anomaly detection factor of each data point is calculated and the data is evaluated based on the anomaly detection factor threshold. For abnormal data points, they are not processed immediately, but an abnormal review strategy is used. Specifically, when the data anomaly detection factor in the current window is lower than the specified anomaly detection factor threshold, the anomaly is not reported and confirmed immediately, but it is regarded as a candidate outlier and re-evaluated in the future window; at the same time, an anomaly weight is used to track each candidate anomaly data. The increase or decrease of the anomaly weight of the data point depends on the relationship between the data point and the specified anomaly detection factor threshold in the current round window. If the anomaly weight of the data point is equal to the custom weight W, the data point is immediately returned as an outlier. If the anomaly weight of the data point is less than zero, the similarity score of the data point is calculated, and the best micro-cluster is selected for addition. In this embodiment, the Mahalanobis distance is used as the normal data similarity score evaluation. The Mahalanobis distance is defined as:

[0086]

[0087] Where m is the mean of the micro-cluster and Λ is the covariance matrix of the micro-cluster.

[0088]

[0089]

[0090] After the window ends, the network model and set are dynamically updated to detect anomalies in the next window. In the current window, Algorithm 2 performs similarity evaluation on normal data. Due to the dynamic nature of data flow, the micro-cluster set is updated. The update formula is as follows:

[0091]

[0092]

[0093]

[0094] in, represents the number of historical data in the i-th micro-cluster, Indicates the number of new micro-clusters added to the i-th micro-cluster in the current window; represents the mean of the i-th micro-cluster, Represents the mean of the data newly added to the i-th micro-cluster in the current window; represents the covariance matrix of the i-th micro-cluster, Represents the covariance matrix of the data newly added to the i-th micro-cluster in the current window.

[0095] In addition to updating individual micro-clusters, the micro-cluster set is also updated. The micro-cluster set is a LRU queue. The update method is as follows: first, a merge operation is performed to merge micro-clusters with high similarity and high activity. Subsequently, an LRU operation is performed to keep the overall number of micro-clusters in the set within the set range. Due to the dynamic evolution of the data stream, new categories will be generated over time, so new categories are searched in the anomaly candidate set. At the same time, the stacked habituation autoencoder is trained using micro-clusters to decode and reduce the dimensionality of the data in the subsequent window.

[0096]

[0097] Experiment and result analysis

[0098] In order to verify that the method proposed in this paper has good generalization ability and robustness, four large-scale Internet of Things datasets are used to conduct comparative experiments with current mainstream algorithms and perform parameter sensitivity analysis.

[0099] 1. Dataset

[0100] The following four real IoT high-dimensional datasets are used for simulation experiments.

[0101] (1) Synthetic Datasets: Synthetic datasets are generated from a Gaussian mixture distribution with outliers generated from a uniform distribution within a given range. We choose the Gaussian mixture model because its distribution changes over time, and this change in distribution simulates the concept drift of the data stream. To simulate this change, we use Gaussian distributions with different means but the same variance to generate 10,000 data samples of dimension 50.

[0102] (2) CIFAR-10: This dataset contains 60,000 color images, divided into 10 categories, with 6,000 images in each category.

[0103] (3) ALOI: This dataset is a set of images provided by Geusebroek et al. for outlier detection in different representations. The data is represented by 27 numerical attributes (HSB histogram) and consists of 50,000 instances, which are divided into 1,508 outliers (3.04%) and 48,492 normal values ​​(96.98%).

[0104] (4) Gas Sensor Array Drift (GSAD): This dataset has a dimension of 128 and contains 13,910 measurements from 16 chemical sensors exposed to 6 different gas concentration levels.

[0105] 2. Evaluation indicators

[0106] Anomaly detection can be considered a special binary classification problem because each data point needs to be classified as a normal value or an anomaly. In the complex environment of the Internet of Things, high-dimensional dynamic data streams are often unbalanced. In order to accurately measure the model, we use precision, recall, and F-score to evaluate the anomaly detection model. These three indicators are standard metrics for quantifying anomaly detection models and are widely used in anomaly detection model evaluation. The following are common category symbols: true positives (tp), true negatives (tn), false positives (fp), and false negatives (fn). Precision, recall, and F-score are defined as follows:

[0107]

[0108] Recall, which evaluates a model's quality in identifying normal samples, and precision, which quantifies how many samples identified as anomalous by the model are actually anomalous. Recall is high when an anomaly detection method is able to detect all anomalies, while precision is used to assess the reliability of an anomaly detection method. The F-score is a summary score that represents the trade-off between recall and precision and is calculated as the harmonic mean of recall and precision.

[0109] In addition, we also use the ROC curve to evaluate the model. The definitions of TRP and FPR are as follows:

[0110]

[0111] Here, P is the number of samples labeled as normal. We define a series of thresholds δ∈R. Each δ is associated with a different pair of TRP and FPR. These values ​​form a list of two-dimensional coordinates, which are plotted as a curve. The starting and ending points of this curve are (0,0) and (1,1), respectively. The area under the curve is defined as AUC. A larger AUC value indicates that an anomaly point in the data stream will receive a higher anomaly score than normal points, making AUC suitable for comparing different data stream anomaly detection algorithms.

[0112] 3. Comparison of algorithms

[0113] To verify the superiority of this method, we selected several novel and efficient algorithms in the field of data stream anomaly detection and conducted comparative experiments. The experimental results show that this method is feasible and superior.

[0114] (1) STORM is one of the more classic data stream anomaly detection algorithms. It uses a sliding window to process the data stream and uses an index buffer stream to save the neighbor point list of the data point in the previous window and the neighbor point instance of the data point in the current window.

[0115] (2) MCOD uses micro-clustering for clustering features and uses triangle inequality to judge data points.

[0116] (3) M_MCOD improves MCOD by relaxing the constraints of micro-clustering and using the minimum detection principle.

[0117] (4) NETS uses a grid-based model that can assign cells to data points and use cells to quickly detect anomalies.

[0118] (5) DeepStream uses traditional stacked autoencoders to train data in the offline stage, uses the offline model to reduce the dimensionality of the data in the online stage, and uses the data structure in the PcStream algorithm to process the data points.

[0119] (6) ADSS is a service selection framework that dynamically selects and configures anomaly detection services. A fast classification model based on XGBoost is trained to identify patterns in various streaming data, so that appropriate services can be selected and configured based on the patterns of the streaming data.

[0120] 4. Comparative experiment

[0121] Experiment 1: To demonstrate the ability of the habituated stacked autoencoder to extract nonlinear relationships, we designed this experiment to compare our method with the six algorithms mentioned above on three different metrics: precision, recall, and F-score. To better compare the results, we set ξ to 0.5, the microclustering cohort size m to 60, and the sliding window size to 1000 on both synthetic and real datasets. We conducted 30 experiments on each dataset. We evaluated the experimental results using precision, recall, and F-score.

[0122] The experimental results above show that this method achieves significant advantages in all indicators compared to traditional data anomaly detection algorithms such as STORM, NETS, MCOD, and M_MCOD, as well as deep learning-based algorithms such as DeepStream and ADSS. Furthermore, we can see that deep learning algorithms perform in-depth learning of data features in the offline phase, demonstrating strong anomaly detection capabilities in the online phase. Compared to DeepStream, which also uses deep learning algorithms, this method uses stacked habituation autoencoders in the offline phase, resulting in a stronger ability to distinguish similar anomalies and exhibiting a more significant advantage on the high-dimensional real-world CIFAR-10 data.

[0123] Experiment 2: To verify the superiority of using joint kernel density estimation for detecting data stream anomalies, we used the AUC, a commonly used evaluation metric in anomaly detection applications, on four different datasets to further test the robustness of our method and perform performance tests on different algorithms. The experimental results are shown in Table 1.

[0124] Table 1: Comparison of AUC values ​​of different algorithms on different datasets

[0125]

[0126] Note: Bold font indicates the maximum value of the evaluation index in the current dataset

[0127] Experimental results demonstrate superior performance on both synthetic and real-world datasets. For online anomaly detection, we leverage historical data combined with current micro-clustering to construct a joint kernel density estimator. Furthermore, this method does not immediately return outliers in the current window, but instead places them into an outlier candidate set. This effectively requires at least two evaluations to determine whether a value is an outlier. This effectively addresses concept drift in data streams and reduces the rate of false positives.

[0128] 5. Parameter sensitivity analysis

[0129] Experiment 3: Different parameter sizes significantly impact the performance of this method. To find the optimal parameter values, we conducted sensitivity analyses on two key parameters of this method on two real-world datasets, ALOI and CIFAR-10. We focused on the sliding window size and the micro-cluster queue size.

[0130] Figure 5The figure shows the effect of different parameter sizes on the experimental accuracy under two real data sets in this embodiment. (a) shows the effect of different window sizes on the accuracy under the same conditions; (b) shows the effect of different micro-cluster queue sizes on the accuracy under the same conditions. Figure 5 (a) It can be seen that as the window size increases, the accuracy of each data set shows a trend of first increasing and then decreasing. When the window size is set to 1000, the accuracy reaches the maximum value. This is because when the number of data points in a single window is small, the window kernel density estimate is low and the outliers cannot be accurately identified. However, when the number of data points in a single window is large, a large number of data points will enter the candidate outlier set, and the model will be overfitted. Figure 5 (b) It can be seen that the size of the micro-cluster cohort is also a significant factor affecting accuracy. As the cohort size increases, the accuracy on each dataset also shows a trend of first increasing and then decreasing. When the cohort size is set to 60, the accuracy is optimal. This is because if the number of micro-cluster cohorts is too large, outliers in the model will be classified as new categories, while if the number is too small, most normal points will not be correctly classified.

[0131] Experimental results show that this method has obvious advantages in solving the problem of anomaly detection in high-dimensional dynamic data streams.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A high-dimensional dynamic data stream anomaly detection method based on stacked habituation autoencoders, characterized in that: It includes the initialization phase and the online anomaly detection phase; In the initialization phase, a static anomaly detection algorithm is used to detect anomalies in historical data, generate anomaly candidate sets and normal data, and use the micro-clusters generated from the normal data to train the stacked habituation autoencoder. In the online anomaly detection phase, the dynamic data stream is separated by a sliding window. A stacked habituation autoencoder is used to reduce the dimensionality of the current window data and the anomaly candidate set. Then, a joint probability density estimator is used for online anomaly detection. Anomaly detection generates true outliers, anomaly candidate sets, and normal data. The normal data is used to continuously update the stacked habituation autoencoder, and the anomaly candidate set is used for anomaly detection in the next window. The stacked habituation autoencoder is constructed by adding a habituation autoencoder to the stacked autoencoder. A stacked habituation autoencoder needs to be constructed level by level. The first level is the habituation autoencoder, which contains three layers: Where x is the input data, is the projection data, LN is the habituation activation function, g1 is the encoding layer activation function, W1 and b1 are the encoding layer network parameters, h is the hidden layer data, g2 is the decoding layer activation function, W2 and b2 are the decoding layer network parameters, is the output data; After the window ends, the network model and the set are dynamically updated to detect anomalies in the next window. In the current window, the similarity of normal data is evaluated and the micro-cluster set is updated. The update formula is as follows: in, represents the number of historical data in the i-th micro-cluster, Indicates the number of new micro-clusters added to the i-th micro-cluster in the current window; represents the mean of the i-th micro-cluster, Represents the mean of the data newly added to the i-th micro-cluster in the current window; represents the covariance matrix of the i-th micro-cluster, Represents the covariance matrix of the data newly added to the i-th micro-cluster in the current window.

2. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 1 is characterized in that: The input data x is first processed by habituation. At the same time, ReLU is used as the activation function in each layer, and L2 normalization is added in the training process of the hidden layer h. The decoding layer is used to reconstruct the input data x. During the training phase, the mean square error n is the total number of data points; Each level of the stacked habituation autoencoder is trained step by step, and the hidden layer of the previous level of autoencoder is used as the input layer of the next level of autoencoder. The stacked habituation autoencoder is constructed by stacking step by step.

3. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 1 is characterized in that: Set the historical data as window 0. In the initialization phase, train the data in window 0 and return the stacked habituation autoencoder, micro-cluster set, and anomaly candidate set. The specific method is: cluster the data in window 0 to obtain normal data and abnormal data, construct a micro-cluster set for the normal data, train the stacked habituation autoencoder at the same time, and put the abnormal data into the anomaly candidate set.

4. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 3 is characterized in that: In the online anomaly detection stage, the data stream is separated into window data through a sliding window, and the n-1th window data is trained to obtain basic components, which are then used to perform online anomaly detection on the data of the n-1th window. After the end of the n-1th window, a network model is obtained, namely the stacked habituated autoencoder and two sets, namely the micro-cluster set and the anomaly candidate set. When processing the n-th window data, anomaly detection is also performed on the anomaly candidate set. The specific method is as follows: first, the stacked habituated autoencoder is used to encode and reduce the dimensionality of the data, then the anomaly detection factor of each data point is calculated, and the data is evaluated according to the anomaly detection factor threshold.

5. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 4 is characterized in that: Abnormal data points are not processed immediately, but an abnormal review strategy is used. Specifically, when the data anomaly detection factor in the current window is lower than the specified anomaly detection factor threshold, the anomaly is not reported and confirmed immediately, but is regarded as a candidate outlier and re-evaluated in future windows. At the same time, anomaly weights are used to track each candidate anomaly data. The increase or decrease of the anomaly weight of the data point depends on the relationship between the data point and the specified anomaly detection factor threshold in the current window. If the anomaly weight of the data point is equal to the custom weight W, the data point is immediately returned as an outlier. If the anomaly weight of the data point is less than zero, the similarity score of the data point is calculated, and the best micro-cluster is selected for addition.

6. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 5 is characterized in that: The Mahalanobis distance is used as the normal data similarity score evaluation, and the Mahalanobis distance is defined as: Where m is the mean of the micro-cluster and Λ is the covariance matrix of the micro-cluster.

7. The high-dimensional dynamic data flow anomaly detection method based on stacked habituation autoencoders according to claim 4 is characterized in that: In addition to updating individual microclusters, the microcluster set is also updated. The microcluster set is a LRU queue. The update method is: first, a merge operation is performed to merge microclusters with high similarity and high activity. Then, an LRU operation is performed to keep the overall number of microclusters in the set within the set range. As time goes by, new categories are generated, so new categories are sought in the anomaly candidate set; at the same time, stacked habituation autoencoders are trained using micro-clustering to decode and reduce the dimensionality of data in the subsequent window.

Images