An adaptive method for system anomaly detection by fusing multidimensional sequence data

Abstract

This invention discloses an adaptive method for detecting system anomalies by fusing multidimensional sequence data. The method first collects time-related raw monitoring data during the system execution phase, digitizes the non-numerical format raw monitoring data, and then normalizes it to obtain a multidimensional time series. The multidimensional time series is then reconstructed using a time series reconstruction model, and the difference between the data before and after reconstruction is calculated to obtain a reconstructed difference sequence X. A feature space is constructed based on the data distribution of the reconstructed difference sequence X, and this process is repeated t times. Based on the feature space constructed in t experiments, the feature vector of the reconstructed difference sequence X is calculated. A fixed-length sliding window is used to divide the feature vector of the reconstructed difference sequence X into multiple subsequences. Finally, kernel mean embedding is used to calculate the similarity between different subsequences, and the time corresponding to the subsequence with a similarity less than a threshold is determined as the time of abnormal system operation. This method effectively utilizes multidimensional time series data.

Description

Technical Field

[0001] This invention belongs to the field of data analysis technology and relates to the detection of abnormal operation of systems in large Internet service centers. Specifically, it relates to a system anomaly detection method that adaptively fuses multidimensional sequence data. Background Technology

[0002] Anomaly detection in large internet service centers faces two main challenges: 1. Monitoring entities within large service clusters typically possess vast amounts of monitoring data reflecting various aspects of their health. This data contains information across multiple dimensions; therefore, anomalies may manifest as complex interactions between these dimensions, such as shifts in data distribution, changes in cluster structure, or enhancements or weakening of correlations, rather than significant deviations in a single dimension. 2. Supervised anomaly detection algorithms require accurate anomaly labels, but due to the complexity of time-series anomaly labeling and the sheer volume of data, obtaining high-quality labels is a challenging problem.

[0003] Existing technologies have proposed various solutions to the problem of time series anomaly detection. For example, time series decomposition, which breaks down a one-dimensional time series into a two-dimensional time series with periodic information (a typical example being TimesNet), suffers from the problem of assigning equal importance to each dimension of a multi-dimensional time series, failing to utilize the advantages of multi-dimensional time series. Graph attention networks, which transform the time series into a graph and then use an attention mechanism to fuse information from key dimensions and neighboring time points for anomaly detection (a typical example being MAD-STA), heavily rely on the quality of the graph structure for learning. Constructing a graph capable of representing the complex relationships of a multi-dimensional time series is a challenging problem. Furthermore, graph neural networks suffer from training difficulties; the complexity of time series makes it difficult to obtain a model with strong generalization ability.

[0004] In summary, it is of great importance to design a system operation anomaly detection method for multidimensional time series, which fully utilizes the characteristics of multidimensional time series and does not use labels for anomaly detection, so as to promptly detect anomalies and take response measures before the anomalies have an adverse impact on the system, reduce the impact of anomalies on system performance, and improve the stability and reliability of the system. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an adaptive method for system anomaly detection by fusing multidimensional time series data. This method identifies potential anomalies in system operation by collecting and analyzing relevant monitoring data and calculating multidimensional time series similarity. Furthermore, it does not include anomaly labels, thus maintaining good performance even when the system performs different tasks, solving the problem of existing anomaly detection methods lacking effective utilization of multidimensional time series data.

[0006] An adaptive method for detecting system anomalies by fusing multidimensional sequence data includes the following steps:

[0007] Step 1: Collect time-related raw monitoring data during the system execution phase. This monitoring data includes multi-dimensional monitoring indicators, i.e., multi-dimensional sequence data. The raw monitoring data, which is not in digital format, is digitized and then normalized to obtain a multi-dimensional time series.

[0008] Step 2: Reconstruct the multidimensional time series using a time series reconstruction model. Reconstruct the data and calculate the difference between the data before and after reconstruction to obtain the reconstructed difference sequence X.

[0009] Step 3: Construct a feature space for the distribution of the reconstructed difference sequence X data, and repeat t times.

[0010] Step 4: Based on the feature space constructed from t experiments, calculate the isolated kernel feature vector of the reconstructed difference sequence X according to the position of the data points in the feature space.

[0011] Step 5: Using a fixed-length sliding window, divide the reconstructed difference sequence X feature vector into multiple subsequences.

[0012] Step 6: Calculate the similarity between different subsequences using kernel mean embedding, and determine the time corresponding to the subsequence with a similarity less than the threshold as the time when the system is running abnormally.

[0013] The present invention has the following beneficial effects:

[0014] 1. Compared with traditional methods that set a threshold for each dimension of the reconstructed data, this application fully utilizes the advantages of multidimensional time series by constructing a feature space for the reconstructed difference sequence X data distribution, thereby mining the rich information contained in the multidimensional data and solving the challenge of setting multidimensional thresholds.

[0015] 2. By not using label information, the cost of manual annotation is reduced, and human error introduced by the bias of labelers is avoided, thus improving the accuracy of the model. At the same time, not including outlier labels also allows the model to adapt more easily to changes in data, avoiding the need for frequent data relabeling in environments with dynamically changing data distributions.

[0016] 3. Using the difference after time series reconstruction algorithm instead of directly using the original time series for correlation calculation solves the noise problem of the original data and clearly represents the originally hidden data features, which helps to simplify the subsequent anomaly detection process and improve the accuracy of anomaly detection. Attached Figure Description

[0017] Figure 1 A schematic diagram of an anomaly detection method for adaptively fusing multidimensional sequence data;

[0018] Figure 2 This is a schematic diagram of the feature space established in one experiment in the embodiment;

[0019] Figure 3 This is a schematic diagram of the feature space established under the second experiment in the embodiment;

[0020] Figure 4 This is a schematic diagram of the time window to be detected in the embodiment;

[0021] Figure 5 This is a schematic diagram illustrating the historical data partitioning for online detection. Detailed Implementation

[0022] The invention will be further explained below with reference to the accompanying drawings. This embodiment uses Metrics data as an example to introduce a system anomaly detection method that adaptively fuses multidimensional sequence data. The specific steps for identifying potential anomalies in system operation by analyzing system monitoring data are as follows:

[0023] Step 1: Data Preparation Stage

[0024] The metrics data includes a series of system-level and user-perceived metrics, such as CPU utilization, memory utilization, and average response time, and is multi-dimensional numerical data collected at certain time intervals. (For multi-dimensional metrics time series...) First, normalization is performed to obtain the cleaned data.

[0025]

[0026] It is a time series The value at timestamp i and dimension j, where 1 ≤ i ≤ N, 1 ≤ j ≤ D, and N is a time series. The time length, D is the number of dimensions. Let j represent the sequence in the j-th dimension. yes The minimum value, yes The maximum value of e. e > 0 is a minimum value to prevent the denominator from being zero.

[0027] Step 2: Construct the data distribution of the multidimensional time series difference sequence

[0028] like Figure 1As shown, the normalized time series is reconstructed using the Time Series Reconstruction Model (TSM). Perform reconstruction and output a multidimensional reconstruction sequence. Next, in the reconstructed sequence Calculate the relationship on each dimension The reconstruction error between the two time series outputs the corresponding multidimensional time series reconstruction difference sequence.

[0029] The time series reconstruction model (TSM) is a deep learning model, specifically one of the following: autoencoder, variational autoencoder, recurrent neural network, long short-term memory network, diffusion model, temporal convolutional network, or generative adversarial network.

[0030] By generating the data distribution of the difference sequence, similarity comparison in subsequent anomaly detection is based on the data distribution of the difference sequence rather than the absolute size of a single data point. This can effectively handle sequences with different absolute sizes but similar data distributions, thus adaptively responding to changes in the difference sequence.

[0031] Step 3: Construct the feature space of the difference distribution

[0032] Reconstructing the difference sequence X = {x1,…,x} from a multidimensional time series N Randomly select ψ vectors Θ = {θ1,…,θ} from} ψ Each vector corresponds to the center coordinates of a hypersphere, and each hypersphere θ k radius r k To be with the supersphere θ k The minimum distance between the centers of all other spheres, i.e., r k =min(dist(θ) k ,Θ-{θ k})),in This represents the calculation of the distance between two vectors, m, k = 1, ..., ψ, and m ≠ k. Each hypersphere represents a partitioned subspace.

[0033] Based on the Monte Carlo method, the above method is repeated independently t times to obtain t different partitioning results. Each partitioning method can produce ψ hyperspheres.

[0034] This processing method allows the difference distribution to be mapped to the feature space, fully considering the semantic information of each dimension in the D-dimensional space. Compared with the traditional method of anomaly detection based on single-dimensional reconstruction of differences, this method can more comprehensively understand the data characteristics, thereby significantly improving the accuracy of anomaly detection.

[0035] Step 4: Adaptive feature mapping of the difference distribution

[0036] Define a partition H l This includes the range of ψ hyperspheres under a single segmentation method, l = 1, ..., t. Define a ψ-dimensional binary column vector Φ. l (x i )={Φ l,1 (x i ),…,Φ l,k (x i ),…Φ l,ψ (x i For the reconstructed difference sequence X = {x1, ..., x}, N Any vector x in} i Either it falls into only one of these ψ hyperspheres, or it does not fall into any hypersphere, if min(dist(x) i ,Θ)) <r k This indicates point x i Falling on the supersphere θ k In the middle, let Φ l,k (x i ) = 1, otherwise Φ l,k (x i ) = 0.

[0037] Given t possible segmentation results, It consists of t binary column vectors Φ1(x i ),…,Φ t (x i It is formed by connecting ) , M=t×ψ. As the feature vector of the reconstructed difference sequence X.

[0038] Figure 2 , 3 Two different spatial partitioning results are shown, where the red dots represent the centers of the hypersphere, and θ1 to θ3 are the three subspaces partitioned in one experiment. Figure 2 In the equation, point x1 falls into θ2, therefore Φ1(x1) = {0, 1, 0}, and point x2 falls into θ3, therefore Φ1(x2) = {0, 0, 1}. And... Figure 3Among them, the point x1 falls into θ1, so Φ2(x1) = {1, 0, 0}. After two experiments, Φ(x1) is composed of Φ1(x1) and Φ2(x1) spliced together, so Φ(x1) = {0, 1, 0, 1, 0, 0}.

[0039] This processing method represents the data distribution of the difference sequence as a vector in the feature space. When calculating the similarity, it is calculated based on different feature vectors mapped from different distributions. Therefore, the comparison strategy can be adaptively adjusted according to the difference distribution.

[0040] Step 5: Construct the difference sequence distribution of the time window to be detected

[0041] Construct a time window S to be detected with length L v , v = 1, 2, … N - L + 1, 0 < L < N, and the specific value of L is set according to actual needs. Through the time window S to be detected v Convert the feature vector Φ(X) of the reconstructed difference sequence X into a sliding window sequence As Figure 4 shown, set L = 4, then the reconstructed difference sequence X with length 8 can be divided into 5 time windows to be detected in sequence.

[0042] Through windowing processing, the feature vectors of the difference sequence for a continuous period of time are converted into a distribution. Compared with a single time point, this method considers the semantic information of the data in the time dimension, converts the difference distribution into a difference window with a fixed period, enables the system to adaptively respond to changes in the difference data distribution, and thus improves the sensitivity and accuracy of anomaly detection.

[0043] Step 6: Calculate the adaptive similarity of the difference sequence

[0044] For the feature vector of any given difference window sequence with length L Use the mean of the isolated kernel feature to replace the feature vector of the difference window sequence S v of

[0045] For any two feature vectors of the difference window sequence S v 、S u , their similarity The calculation method is:

[0046]

[0047] where u ≠ v, <,> represents the multiplication calculation between matrices.

[0048] By mapping the difference distribution of time windows to a feature vector in the feature space, kernel mean embedding is used to calculate the similarity between the difference distributions of two time windows. Since different time window difference distributions can be represented by different feature vectors in the feature space, the comparison strategy can be adaptively adjusted according to changes in the difference data. This method is suitable for handling differences with large absolute magnitudes that may occur in detection windows of different periods. Unlike traditional anomaly detection methods based on difference thresholds, anomaly detection methods based on similarity calculation can adaptively compare the distribution of difference data within a time window, thereby maintaining the relative stability of similarity calculation and improving the stability of anomaly detection.

[0049] Step 7: System Anomaly Detection

[0050] Anomaly detection based on similarity calculation results is performed using the following method:

[0051] S7.1, Offline Anomaly Detection

[0052] Given the anomalous data throughout the time series The proportion in time series is relatively low. The similarity to normal subsequences is high, while the similarity to abnormal subsequences is low. Therefore, the complete time series is obtained using the method in step 6. The reconstructed difference sequence X is divided into multiple subsequences, and then the similarity between each subsequence and the complete reconstructed difference sequence X is calculated sequentially. Set a threshold τ1, if Then determine the current time period of the window to be detected as abnormal.

[0053] As an example, the threshold τ1 is determined as follows: the similarity scores of all windows to be detected are arranged in ascending order to obtain an array Score of length N-L+1, and a similarity score is selected from Score as the threshold τ1 according to the preset anomaly rate.

[0054] s7.2 Online Anomaly Detection

[0055] For online monitoring of system operation data, the similarity between the data to be detected S and historical data T is calculated and compared with a set threshold τ2. Data to be detected with a similarity to historical data less than the threshold τ2 is identified as abnormal data.

[0056] It should be noted that the historical data T will also change over time, such as... Figure 5As shown, at the current moment, the historical data T = {x1, x2, x3, x4, x5}, and the data to be detected S = {x6, x7, x8}. At the next moment, the historical data T = {x1, x2, x3, x4, x5, x6}, and the data to be detected S = {x7, x8, x9}. By continuously adding the newly detected data to the historical data queue and updating the historical data information, adaptive optimization of online detection is achieved.

[0057] It should be noted that any content not described in detail in this specification belongs to the prior art known to those skilled in the art. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system anomaly detection method that adaptively fuses multidimensional sequence data, collecting time-related raw monitoring data during the system execution phase for anomaly detection, characterized in that: The specific steps are as follows: Step 1: Digitize and encode the raw monitoring data (which is not in digital format) to obtain a multidimensional time series. Where N is the length of the time series; Step 2: Reconstruct the multidimensional time series using a time series reconstruction model. Reconstruct the data and calculate the data difference before and after reconstruction to obtain the reconstructed difference sequence X = {x1, ..., x...} N }; Step 3: Construct a feature space based on the data distribution of the reconstructed difference sequence X; Step 4: Calculate the feature vector of the reconstructed difference sequence X based on the position of the data points in the feature space. Step 5: Using a fixed-length sliding window, divide the reconstructed difference sequence X feature vector into multiple subsequences and calculate the feature vectors of the subsequences; Step 6: Calculate the similarity between different subsequences, and determine the time corresponding to the subsequence with a similarity less than the threshold as the time when the system is running abnormally.

2. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 1, characterized in that: The original monitoring data after digitization Normalization is performed: It is a time series The value at timestamp i and dimension j, 1≤i≤N, 1≤j≤D, where D is the number of dimensions; Let j represent the sequence in the j-th dimension. yes The minimum value, yes The maximum value of ; e > 0, used to prevent the denominator from being 0.

3. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 1, characterized in that: The time series reconstruction model is one of the following: autoencoder, variational autoencoder, recurrent neural network, long short-term memory network, diffusion model, temporal convolutional network, or generative adversarial network.

4. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 1, characterized in that: The method for constructing the feature space based on the distribution of the reconstructed difference sequence X data is as follows: Randomly select ψ vectors Θ = {θ1,…,θ2} from the reconstructed difference sequence X. ψ Each vector corresponds to a hypersphere θ. k The coordinates of the center of the sphere, its radius And m≠k; Each hypersphere represents a partitioned subspace.

5. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 4, characterized in that: Based on the Monte Carlo method, the feature space is constructed independently t times to obtain t different segmentation results, and each segmentation method can produce ψ hyperspheres.

6. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 5, characterized in that: The method for calculating the eigenvector of the reconstructed difference sequence X is as follows: Define a partition H l This includes the range of ψ hyperspheres under a single partitioning method, l = 1, ..., t; and defines a ψ-dimensional binary column vector Φ. l (x i )={Φ l,1 (x i ),…,Φ l,k (x i ),…Φ l,ψ (x i For the reconstructed difference sequence X = {x1, ..., x}, N Any vector x in} i If min(dist(x) i ,Θ)) <r k This indicates point x i Falling on the supersphere θ k In the middle, let Φ l,k (x i ) = 1, otherwise Φ l,k (x i ) = 0; Given t possible segmentation results, It consists of t binary column vectors Φ1(x i ),…,Φ t (x i It is formed by connecting ) and M = t × ψ; As the feature vector of the reconstructed difference sequence X.

7. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 6, characterized in that: Construct a time window \(S\) to be detected with length \(L\). v where \(v = 1, 2, \ldots, N - L + 1\), \(0 < L < N\), and transform the feature vector \(\varPhi(X)\) of the reconstructed difference sequence \(X\) into a sliding window sequence through the time window \(S\) to be detected v S v =\{\varPhi(x v ), \ldots, \varPhi(x v+L-1 )\}; Replace the feature vector of the difference window sequence \(S\) with the isolated kernel feature mean v ​​​ For any two difference window sequences S v S u Their similarity The calculation method is as follows: Where u≠v, and <,> represent matrix multiplication.

8. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 7, characterized in that: For anomaly detection in the system's historical operation, multidimensional time series... After dividing the reconstructed difference sequence X into multiple subsequences, the value of each subsequence S is calculated sequentially. v Similarity between the complete reconstructed difference sequence X and the complete reconstructed difference sequence X Set a threshold τ1, if Then determine the subsequence S v The corresponding time period is the time when the system was abnormal.

9. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 8, characterized in that: The threshold τ1 is determined by the similarity between all subsequences and the complete reconstructed difference sequence X. Arrange the data in ascending order to obtain the array Score. Select a similarity score from the Score array as the threshold τ1 according to the preset anomaly rate.

10. The system anomaly detection method for adaptive fusion of multidimensional sequence data as described in claim 7, characterized in that: For anomaly detection during online system operation, the similarity between the data to be detected S and historical data T is calculated and compared with a set threshold τ2. Data to be detected with a similarity to historical data less than the threshold τ2 is identified as anomaly data. In the next step, the detected data is added to the historical data, the historical data information is updated, and the online detection is adaptively optimized.

Images