A cloud platform time series data anomaly detection method, system, device and medium

Abstract

This invention proposes a method, system, device, and medium for anomaly detection in cloud platform time-series data. The method includes: acquiring raw monitoring data from the cloud platform; segmenting and labeling the data using a sliding window, with the sample data including normal and abnormal samples; preprocessing the sample data; performing EMD processing on the preprocessed sample data to obtain modal components; smoothing and denoising each modal component using a filter to obtain a smoothed time series and construct a normal dataset; extracting feature vectors from the normal dataset and inputting these feature vectors into an anomaly detection model to output anomaly detection data; based on this method, a system, device, and medium for anomaly detection in cloud platform time-series data are also proposed. This invention employs a supervised machine learning method, which to some extent alleviates the trouble of manual labeling of anomaly data and also solves the impact of imbalance between abnormal and normal data on supervised learning.

Description

Technical Field

[0001] This invention belongs to the field of data detection technology, and specifically relates to a method, system, device and medium for detecting time-series data anomalies on a cloud platform. Background Technology

[0002] A cloud platform, also known as a cloud computing platform, refers to a service that provides computing, networking, and storage capabilities based on hardware and software resources. Cloud platform monitoring systems continuously collect a large number of time-series KPIs from the platform, such as CPU utilization and network throughput, to determine the platform's operational status. As cloud platforms mature, their scale has grown from a few or a dozen servers to hundreds or even thousands. Simultaneously, the platform's services have increased in number and complexity, resulting in massive and complex monitoring data. Traditional cloud platforms often use threshold methods for KPI anomaly detection. Operations personnel set thresholds based on experience, and when KPI data reaches this threshold, an anomaly alarm is generated. However, in practical applications, it has been found that threshold setting relies too heavily on experience, making it difficult to accurately set thresholds for a comprehensive range of complex KPIs. Thresholds that are too high may miss anomalies and fail to detect potential quality issues, while thresholds that are too low often trigger alarm storms, interfering with the operations personnel's judgment. Furthermore, this method cannot detect data fluctuations below the threshold, resulting in missed alarms.

[0003] Simply relying on thresholds to determine anomalies is insufficient for anomaly detection in cloud platforms. This necessitates the use of machine learning and data mining techniques for automated anomaly detection. However, in real-world cloud platform monitoring, the probability of anomalies is relatively low, making it difficult to accumulate a large number of anomaly samples. This necessitates that the machine learning framework possess the capability to automatically construct samples, generating sufficient data to support the training of the anomaly detection model. Furthermore, time-series monitoring data is complex, exhibiting characteristics specific to time-dependent changes and using numerical values ​​to reflect the degree of change. These changes can follow various patterns, with common time-series data patterns including stationary, fluctuating, and periodic patterns. In monitoring systems, data patterns change with business needs. This requires anomaly detection models for time-series monitoring data to have sufficient generalization ability to detect anomalies across different data patterns. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method, system, device, and medium for detecting anomalies in time-series data on cloud platforms. This approach alleviates the burden of manual annotation of anomaly data to a certain extent and also mitigates the impact of imbalance between anomalous and normal data on supervised learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for detecting anomalies in time-series data on a cloud platform includes the following steps:

[0007] The raw monitoring data from the cloud platform is acquired, segmented using a sliding window, and labeled as sample data, which includes normal samples and abnormal samples; then the sample data is preprocessed.

[0008] The preprocessed sample data is processed by EMD to obtain each modal component, and each modal component is smoothed and denoised by a filter to obtain a smoothed time series and construct a normal dataset.

[0009] The feature vectors of the normal dataset are extracted and input into the anomaly detection model to output anomaly detection data. The anomaly detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs anomaly detection data other than the normal data.

[0010] Furthermore, the process of obtaining raw monitoring data from the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes:

[0011] Collect monitoring data from the cloud platform over a period of time as raw monitoring data, and set the collection period;

[0012] Missing points in the original monitoring data are filtered out after segmentation using a sliding window.

[0013] Add data labels to the original monitoring data, where normal samples are labeled with 0 and abnormal samples are labeled with 1.

[0014] Furthermore, the preprocessing process for the sample data is as follows:

[0015] The sample data is defined as x = {x1, x2, ..., x...} m};

[0016] Using formula Perform normalization processing;

[0017] Where, x i Let x' represent the i-th sample data, i = 1, 2, ..., m; i These are the sample data after normalization.

[0018] Furthermore, the process of processing the preprocessed sample data into modal components using EMD includes:

[0019] All normalized sample data constitute the KPI time series data X(n); the KPI time series data X(n) is processed by EMD to obtain the sum of each modal component and the residual:

[0020]

[0021] Among them, C i (n) represents the i-th IMF component, N is the total number of IMFs, and R n is the margin, and n is the length of the data sample.

[0022] Furthermore, the process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows:

[0023]

[0024] Where X'(n) is the final smoothed time series; F is the SG filter.

[0025] Furthermore, the process of extracting feature vectors from normal datasets includes: extracting basic statistical features, time-domain features, and frequency-domain features from normal datasets; these basic statistical features, time-domain features, and frequency-domain features constitute the feature vectors of normal datasets.

[0026] The basic statistical features include mean, variance, extreme values, band and power spectrum characteristics;

[0027] The time-domain features include mean, variance, extreme values, zero crossings, boundary points, band length, and peak value features;

[0028] The frequency domain features include power spectrum, power density ratio, median frequency, and average power frequency features.

[0029] Furthermore, the abnormal data detection model adopts the OneClassSVM model.

[0030] This invention also proposes a cloud platform time-series data anomaly detection system, the system comprising a preprocessing module, a decomposition and denoising module, and a detection module;

[0031] The preprocessing module is used to acquire the original monitoring data of the cloud platform, segment it using a sliding window, and label it as sample data. The sample data includes normal samples and abnormal samples. Then, the sample data is preprocessed.

[0032] The decomposition and denoising module is used to decompose the preprocessed sample data into modal components through EMD processing, and to perform smoothing and denoising processing on each modal component through a filter to obtain a smoothed time series and construct a normal dataset.

[0033] The detection module is used to extract feature vectors from the normal dataset and input the feature vectors into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

[0034] The present invention also proposes an apparatus comprising:

[0035] Memory, used to store computer programs;

[0036] A processor for implementing the steps of the method when executing the computer program.

[0037] The present invention also proposes a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0038] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0039] This invention proposes a method, system, device, and medium for anomaly detection in cloud platform time-series data. The method includes: acquiring raw monitoring data from the cloud platform; segmenting the data using a sliding window and labeling it as sample data, which includes normal and abnormal samples; preprocessing the sample data; performing EMD processing on the preprocessed sample data to obtain modal components; smoothing and denoising each modal component using a filter to obtain a smoothed time series and constructing a normal dataset; extracting feature vectors from the normal dataset and inputting these feature vectors into an anomaly detection model to output anomaly detection data; and constructing a minimum hypersphere to enclose normal data within the normal dataset and outputting anomaly detection data excluding the normal data. Based on this cloud platform time-series data anomaly detection method, a cloud platform time-series data anomaly detection system, device, and medium are also proposed. This invention filters the sample set using EMD and SG filtering to construct a normal sample dataset. Then, it extracts the basic statistical features, time-domain features, and frequency-domain features of the normal sample dataset to construct a data feature vector. The constructed feature vector is then input into OneClassSVM, employing a supervised machine learning method. This method alleviates the trouble of manually labeling abnormal data to a certain extent and also solves the impact of the imbalance between abnormal and normal data on supervised learning. Attached Figure Description

[0040] like Figure 1 Here is a flowchart of a cloud platform time-series data anomaly detection method provided in Embodiment 1 of the present invention;

[0041] like Figure 2 This is a schematic diagram of a cloud platform time-series data anomaly detection system provided in Embodiment 2 of the present invention;

[0042] like Figure 3 This is a schematic diagram of an electronic device connection provided in Embodiment 3 of the present invention. Detailed Implementation

[0043] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0044] Example 1

[0045] Embodiment 1 of this invention proposes a method for detecting anomalies in time-series data on cloud platforms, applicable to performance optimization schemes for various cloud platform infrastructures, such as x86, ARM, and MIPS platforms. Figure 1 This is a flowchart of a cloud platform time-series data anomaly detection method according to Embodiment 1 of the present invention.

[0046] In step S100, the original monitoring data of the cloud platform is obtained, segmented by a sliding window and labeled as sample data, which includes normal samples and abnormal samples; then the sample data is preprocessed.

[0047] The process of acquiring raw monitoring data from the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes: collecting monitoring data from the cloud platform over a period of time as raw monitoring data and setting the collection period; filtering out missing points in the raw monitoring data after segmenting it using a sliding window; and adding data labels to the raw monitoring data, where normal samples are labeled with 0 and abnormal samples are labeled with 1.

[0048] The training data used in this invention consisted of four months of monitoring data from a cloud platform. The monitoring data was collected using Telegraf, with its built-in plugins for CPU, disk, and memory data collection, at a 60-second interval. The collected data included monitoring data for key performance indicators such as CPU and memory. Outliers were labeled using a combination of machine learning and manual judgment. The data was segmented using a sliding window approach to filter out missing data and add labels: 0 for normal samples and 1 for outlier samples. A total of 21,542 samples were extracted, including 20,074 normal samples and 1,468 outlier samples.

[0049] The process of preprocessing sample data is as follows:

[0050] The sample data is defined as x = {x1, x2, ..., x...} m};

[0051] Using formula Perform normalization processing;

[0052] Where, x i Let x' represent the i-th sample data, i = 1, 2, ..., m; i These are the sample data after normalization.

[0053] The transformed dimensionless time-series data ranges from [0,1]. All data used in this invention undergoes a standardized preprocessing step before being used as input data for the algorithm model to ensure data standardization.

[0054] In step S110, the preprocessed sample data is processed by EMD to obtain each modal component, and each modal component is smoothed and denoised by a filter to obtain a smoothed time series and construct a normal dataset.

[0055] Because KPI data is highly susceptible to external influences and exhibits significant fluctuations, it can be described as a non-linear, non-stationary signal. The EMD algorithm effectively processes this type of signal, decomposing the data into a finite number of linear, stable IMFs. Therefore, the EMD process can be described as follows: for a given KPI time series data X(n), after EMD processing, it can be represented as the sum of each modal component and the margin.

[0056] The process of converting preprocessed sample data into modal components using EMD includes:

[0057] All normalized sample data constitute the KPI time series data X(n); the KPI time series data X(n) is processed by EMD to obtain the sum of each modal component and the residual:

[0058]

[0059] Among them, C i (n) represents the i-th IMF component, N is the total number of IMFs, and R n is the margin, and n is the length of the data sample.

[0060] In this invention, each IMF component in the EMD component is smoothed and denoised using an SG filter. The principle of the SG filter is as follows: a neighborhood of length k for a data point is selected as a sliding window, and each data point in the neighborhood is fitted with a univariate p-order polynomial. The polynomial coefficients are obtained by the least squares method, and the best fitting value at the center point of the sliding window is obtained. This fitting value is the denoised value. The sliding window slides along each point in turn, thereby achieving smooth denoising.

[0061] The process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows:

[0062]

[0063] Where X'(n) is the final smoothed time series; F is the SG filter.

[0064] In this invention, data is segmented using a sliding window method, resulting in a data curve containing one outlier. This data is obtained through EMD decomposition. After EMD decomposition, the original data is decomposed into four components: the IMF curve and the residual R.

[0065] After EMD decomposition, KPI data yields 4th-order modal components. Based on the physical meaning of each EMD component, each component represents a frequency component in the original signal. Adding the individual IMF components represents the main changes in the original KPI data. Ultimately, the non-stationary signal is transformed into a stationary signal while preserving its instantaneous variations.

[0066] Then, SG filtering is applied to each mode separately to filter out noise in the instantaneous change curve, making the entire curve smoother, and finally reconstructing the curve.

[0067] The method in this invention extracts IMF components after EMD decomposition, and then reconstructs the components of each modality using SG filtering to finally construct a KPI curve that includes data variations and removes noise. The TCN-AE model used in this invention is essentially an AE model, which fits normal data patterns. Therefore, this invention uses a method based on EMD and SG filtering to construct a normal dataset that only includes data variations.

[0068] In step S120, feature vectors of the normal dataset are extracted and input into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

[0069] The data features are extracted, encompassing three aspects: first, basic statistical features: mean, variance, extreme values, bands, and power spectrum characteristics; second, time-domain features: mean, variance, extreme values, zero-crossing points, boundary points, and band length and peak values; and third, frequency-domain features: power spectrum, power density ratio, median frequency, and average power frequency characteristics. This constitutes a 15-dimensional feature vector.

[0070] After feature vector extraction, OneClassSVM is used to learn the data patterns of normal data. The idea of ​​OneClassSVM is very simple: find a minimum hypersphere to circle the positive examples in the samples, and make predictions using this hypersphere. Samples within the circle are considered positive samples.

[0071] After training the OneClassSVM anomaly detection model using training set samples, extract basic statistical features, time-domain features, and frequency-domain features from the test set or online data. Input these features into the trained model and output the anomaly detection results.

[0072] Embodiment 1 of this invention proposes a method for detecting anomalies in time-series data on a cloud platform. The sample set is filtered by EMD and SG filtering to construct a normal sample dataset. Then, the basic statistical features, time-domain features and frequency-domain features of the normal sample dataset are extracted to construct a data feature vector. The constructed feature vector is input into OneClassSVM, which is a supervised machine learning method. It eliminates the trouble of manual annotation of abnormal data to a certain extent and also solves the impact of imbalance between abnormal and normal data on supervised learning.

[0073] Example 2

[0074] Based on Embodiment 1 of this invention, a method for detecting time-series data anomalies in a cloud platform is proposed. Embodiment 2 of this invention proposes a system for detecting time-series data anomalies in a cloud platform, such as... Figure 2 This is a schematic diagram of a cloud platform time-series data anomaly detection system according to Embodiment 2 of the present invention. The system includes a preprocessing module, a decomposition and denoising module, and a detection module.

[0075] The preprocessing module is used to acquire the raw monitoring data from the cloud platform, segment it using a sliding window, and label it as sample data, which includes normal samples and abnormal samples; then the sample data is preprocessed.

[0076] The decomposition and denoising module is used to decompose the preprocessed sample data into modal components through EMD processing, and to smooth and denoise each modal component through a filter to obtain a smoothed time series and construct a normal dataset.

[0077] The detection module is used to extract feature vectors from the normal dataset and input the feature vectors into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

[0078] The preprocessing module process is as follows: the process of acquiring raw monitoring data from the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes: collecting monitoring data from the cloud platform for a period of time as raw monitoring data and setting the collection period; filtering out missing points in the raw monitoring data after segmenting it using a sliding window; and adding data labels to the raw monitoring data, where normal samples are labeled with 0 and abnormal samples are labeled with 1.

[0079] The training data used in this invention consisted of four months of monitoring data from a cloud platform. The monitoring data was collected using Telegraf, with its built-in plugins for CPU, disk, and memory data collection, at a 60-second interval. The collected data included monitoring data for key performance indicators such as CPU and memory. Outliers were labeled using a combination of machine learning and manual judgment. The data was segmented using a sliding window approach to filter out missing data and add labels: 0 for normal samples and 1 for outlier samples. A total of 21,542 samples were extracted, including 20,074 normal samples and 1,468 outlier samples.

[0080] The process of preprocessing sample data is as follows:

[0081] The sample data is defined as x = {x1, x2, ..., x...} m};

[0082] Using formula Perform normalization processing;

[0083] Where, x i Let x' represent the i-th sample data, i = 1, 2, ..., m; i These are the sample data after normalization.

[0084] The transformed dimensionless time series data ranges from [0,1]. All data used in this invention undergoes a standardized preprocessing process before being used as input data for the algorithm model to ensure standardized data processing.

[0085] The decomposition and denoising module process includes the following: Since KPI data is significantly affected by external factors and exhibits large data fluctuations, it can be described as a nonlinear, non-stationary signal. The EMD algorithm can effectively process this type of signal, decomposing the data into a finite number of linear, stable IMFs. Therefore, the EMD process can be described as follows: for a given KPI time series data X(n), after EMD processing, it can be represented as the sum of each modal component and the margin.

[0086] The process of converting preprocessed sample data into modal components using EMD includes:

[0087] All normalized sample data constitute the KPI time series data X(n); the KPI time series data X(n) is processed by EMD to obtain the sum of each modal component and the residual:

[0088]

[0089] Among them, C i (n) represents the i-th IMF component, N is the total number of IMFs, and R n is the margin, and n is the length of the data sample.

[0090] In this invention, each IMF component in the EMD component is smoothed and denoised using an SG filter. The principle of the SG filter is as follows: a neighborhood of length k for a data point is selected as a sliding window, and each data point in the neighborhood is fitted with a univariate p-order polynomial. The polynomial coefficients are obtained by the least squares method, and the best fitting value at the center point of the sliding window is obtained. This fitting value is the denoised value. The sliding window slides along each point in turn, thereby achieving smooth denoising.

[0091] The process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows:

[0092]

[0093] Where X'(n) is the final smoothed time series; F is the SG filter.

[0094] In this invention, data is segmented using a sliding window method, resulting in a data curve containing one outlier. This data is obtained through EMD decomposition. After EMD decomposition, the original data is decomposed into four components: the IMF curve and the residual R.

[0095] After EMD decomposition, KPI data yields 4th-order modal components. Based on the physical meaning of each EMD component, each component represents a frequency component in the original signal. Adding the individual IMF components represents the main changes in the original KPI data. Ultimately, the non-stationary signal is transformed into a stationary signal while preserving its instantaneous variations.

[0096] Then, SG filtering is applied to each mode separately to filter out noise in the instantaneous change curve, making the entire curve smoother, and finally reconstructing the curve.

[0097] The system in this invention extracts IMF components after EMD decomposition, and then reconstructs the components of each modality using SG filtering to finally construct a KPI curve that includes data variations and removes noise. The TCN-AE model used in this invention is essentially an AE model, which fits normal data patterns. Therefore, this invention uses a method based on EMD and SG filtering to construct a normal dataset that only includes data variations.

[0098] The detection module operates as follows: it extracts data features, which include three aspects: first, basic statistical features: mean, variance, extreme values, bands, and power spectrum characteristics; second, time-domain features: mean, variance, extreme values, zero-crossing points, boundary points, and band length and peak value characteristics; and third, frequency-domain features: power spectrum, power density ratio, median frequency, and average power frequency characteristics. This constitutes a 15-dimensional feature vector.

[0099] After feature vector extraction, OneClassSVM is used to learn the data patterns of normal data. The idea of ​​OneClassSVM is very simple: find a minimum hypersphere to circle the positive examples in the samples, and make predictions using this hypersphere. Samples within the circle are considered positive samples.

[0100] After training the OneClassSVM anomaly detection model using training set samples, extract basic statistical features, time-domain features, and frequency-domain features from the test set or online data. Input these features into the trained model and output the anomaly detection results.

[0101] Embodiment 2 of this invention proposes a cloud platform time-series data anomaly detection system. The system filters the sample set through EMD and SG filtering to construct a normal sample dataset. Then, it extracts the basic statistical features, time-domain features and frequency-domain features of the normal sample dataset to construct a data feature vector. The constructed feature vector is input into OneClassSVM, which is a supervised machine learning method. This method alleviates the trouble of manual annotation of abnormal data to a certain extent and also solves the impact of imbalance between abnormal and normal data on supervised learning.

[0102] Example 3

[0103] The present invention also proposes a device, such as Figure 3 The diagram shown is a schematic representation of an electronic device connection according to Embodiment 3 of the present invention, including:

[0104] Memory, used to store computer programs;

[0105] When a processor executes the computer program, the method steps are as follows:

[0106] like Figure 1 This is a flowchart of a cloud platform time-series data anomaly detection method according to Embodiment 1 of the present invention.

[0107] In step S100, the original monitoring data of the cloud platform is obtained, segmented by a sliding window and labeled as sample data, which includes normal samples and abnormal samples; then the sample data is preprocessed.

[0108] The process of acquiring raw monitoring data from the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes: collecting monitoring data from the cloud platform over a period of time as raw monitoring data and setting the collection period; filtering out missing points in the raw monitoring data after segmenting it using a sliding window; and adding data labels to the raw monitoring data, where normal samples are labeled with 0 and abnormal samples are labeled with 1.

[0109] The training data used in this invention consisted of four months of monitoring data from a cloud platform. The monitoring data was collected using Telegraf, with its built-in plugins for CPU, disk, and memory data collection, at a 60-second interval. The collected data included monitoring data for key performance indicators such as CPU and memory. Outliers were labeled using a combination of machine learning and manual judgment. The data was segmented using a sliding window approach to filter out missing data and add labels: 0 for normal samples and 1 for outlier samples. A total of 21,542 samples were extracted, including 20,074 normal samples and 1,468 outlier samples.

[0110] The process of preprocessing sample data is as follows:

[0111] The sample data is defined as x = {x1, x2, ..., x...} m};

[0112] Using formula Perform normalization processing;

[0113] Where, x i Let x' represent the i-th sample data, i = 1, 2, ..., m; i These are the sample data after normalization.

[0114] The transformed dimensionless time series data ranges from [0,1]. All data used in this invention undergoes a standardized preprocessing process before being used as input data for the algorithm model to ensure standardized data processing.

[0115] In step S110, the preprocessed sample data is processed by EMD to obtain each modal component, and each modal component is smoothed and denoised by a filter to obtain a smoothed time series and construct a normal dataset.

[0116] Because KPI data is highly susceptible to external influences and exhibits significant fluctuations, it can be described as a non-linear, non-stationary signal. The EMD algorithm effectively processes this type of signal, decomposing the data into a finite number of linear, stable IMFs. Therefore, the EMD process can be described as follows: for a given KPI time series data X(n), after EMD processing, it can be represented as the sum of each modal component and the margin.

[0117] The process of converting preprocessed sample data into modal components using EMD includes:

[0118] All normalized sample data constitute the KPI time series data X(n); the KPI time series data X(n) is processed by EMD to obtain the sum of each modal component and the residual:

[0119]

[0120] Among them, C i (n) represents the i-th IMF component, N is the total number of IMFs, and R n is the margin, and n is the length of the data sample.

[0121] In this invention, each IMF component in the EMD component is smoothed and denoised using an SG filter. The principle of the SG filter is as follows: a neighborhood of length k for a data point is selected as a sliding window, and each data point in the neighborhood is fitted with a univariate p-order polynomial. The polynomial coefficients are obtained by the least squares method, and the best fitting value at the center point of the sliding window is obtained. This fitting value is the denoised value. The sliding window slides along each point in turn, thereby achieving smooth denoising.

[0122] The process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows:

[0123]

[0124] Where X'(n) is the final smoothed time series; F is the SG filter.

[0125] In this invention, data is segmented using a sliding window method, resulting in a data curve containing one outlier. This data is obtained through EMD decomposition. After EMD decomposition, the original data is decomposed into four components: the IMF curve and the residual R.

[0126] After EMD decomposition, KPI data yields 4th-order modal components. Based on the physical meaning of each EMD component, each component represents a frequency component in the original signal. Adding the individual IMF components represents the main changes in the original KPI data. Ultimately, the non-stationary signal is transformed into a stationary signal while preserving its instantaneous variations.

[0127] Then, SG filtering is applied to each mode separately to filter out noise in the instantaneous change curve, making the entire curve smoother, and finally reconstructing the curve.

[0128] The method in this invention extracts IMF components after EMD decomposition, and then reconstructs the components of each modality using SG filtering to finally construct a KPI curve that includes data variations and removes noise. The TCN-AE model used in this invention is essentially an AE model, which fits normal data patterns. Therefore, this invention uses a method based on EMD and SG filtering to construct a normal dataset that only includes data variations.

[0129] In step S120, feature vectors of the normal dataset are extracted and input into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

[0130] The data features are extracted, encompassing three aspects: first, basic statistical features: mean, variance, extreme values, bands, and power spectrum characteristics; second, time-domain features: mean, variance, extreme values, zero-crossing points, boundary points, and band length and peak values; and third, frequency-domain features: power spectrum, power density ratio, median frequency, and average power frequency characteristics. This constitutes a 15-dimensional feature vector.

[0131] After feature vector extraction, OneClassSVM is used to learn the data patterns of normal data. The idea of ​​OneClassSVM is very simple: find a minimum hypersphere to circle the positive examples in the samples, and make predictions using this hypersphere. Samples within the circle are considered positive samples.

[0132] After training the OneClassSVM anomaly detection model using training set samples, extract basic statistical features, time-domain features, and frequency-domain features from the test set or online data. Input these features into the trained model and output the anomaly detection results.

[0133] Embodiment 3 of this invention proposes a device that filters the sample set using EMD and SG filtering to construct a normal sample dataset. Then, it extracts the basic statistical features, time-domain features, and frequency-domain features of the normal sample dataset to construct a data feature vector. The constructed feature vector is then input into OneClassSVM, which employs a supervised machine learning method. This method alleviates the trouble of manually labeling abnormal data to a certain extent and also solves the problem of the imbalance between abnormal and normal data on supervised learning.

[0134] It should be noted that the present invention also provides an electronic device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute a cloud platform timing data anomaly detection method provided by one or more of the above technical solutions when running a computer program, wherein the computer program is stored in a memory. In practical applications, the various components of the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0135] Example 4

[0136] The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps:

[0137] like Figure 1 This is a flowchart of a cloud platform time-series data anomaly detection method according to Embodiment 1 of the present invention.

[0138] In step S100, the original monitoring data of the cloud platform is obtained, segmented by a sliding window and labeled as sample data, which includes normal samples and abnormal samples; then the sample data is preprocessed.

[0139] The process of acquiring raw monitoring data from the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes: collecting monitoring data from the cloud platform over a period of time as raw monitoring data and setting the collection period; filtering out missing points in the raw monitoring data after segmenting it using a sliding window; and adding data labels to the raw monitoring data, where normal samples are labeled with 0 and abnormal samples are labeled with 1.

[0140] The training data used in this invention consisted of four months of monitoring data from a cloud platform. The monitoring data was collected using Telegraf, with its built-in plugins for CPU, disk, and memory data collection, at a 60-second interval. The collected data included monitoring data for key performance indicators such as CPU and memory. Outliers were labeled using a combination of machine learning and manual judgment. The data was segmented using a sliding window approach to filter out missing data and add labels: 0 for normal samples and 1 for outlier samples. A total of 21,542 samples were extracted, including 20,074 normal samples and 1,468 outlier samples.

[0141] The process of preprocessing sample data is as follows:

[0142] The sample data is defined as x = {x1, x2, ..., x...} m};

[0143] Using formula Perform normalization processing;

[0144] Where, x i Let x' represent the i-th sample data, i = 1, 2, ..., m; i These are the sample data after normalization.

[0145] The transformed dimensionless time series data ranges from [0,1]. All data used in this invention undergoes a standardized preprocessing process before being used as input data for the algorithm model to ensure standardized data processing.

[0146] In step S110, the preprocessed sample data is processed by EMD to obtain each modal component, and each modal component is smoothed and denoised by a filter to obtain a smoothed time series and construct a normal dataset.

[0147] Because KPI data is highly susceptible to external influences and exhibits significant fluctuations, it can be described as a non-linear, non-stationary signal. The EMD algorithm effectively processes this type of signal, decomposing the data into a finite number of linear, stable IMFs. Therefore, the EMD process can be described as follows: for a given KPI time series data X(n), after EMD processing, it can be represented as the sum of each modal component and the margin.

[0148] The process of converting preprocessed sample data into modal components using EMD includes:

[0149] All normalized sample data constitute the KPI time series data X(n); the KPI time series data X(n) is processed by EMD to obtain the sum of each modal component and the residual:

[0150]

[0151] Among them, C i (n) represents the i-th IMF component, N is the total number of IMFs, and R n is the margin, and n is the length of the data sample.

[0152] In this invention, each IMF component in the EMD component is smoothed and denoised using an SG filter. The principle of the SG filter is as follows: a neighborhood of length k for a data point is selected as a sliding window, and each data point in the neighborhood is fitted with a univariate p-order polynomial. The polynomial coefficients are obtained by the least squares method, and the best fitting value at the center point of the sliding window is obtained. This fitting value is the denoised value. The sliding window slides along each point in turn, thereby achieving smooth denoising.

[0153] The process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows:

[0154]

[0155] Where X'(n) is the final smoothed time series; F is the SG filter.

[0156] In this invention, data is segmented using a sliding window method, resulting in a data curve containing one outlier. This data is obtained through EMD decomposition. After EMD decomposition, the original data is decomposed into four components: the IMF curve and the residual R.

[0157] After EMD decomposition, KPI data yields 4th-order modal components. Based on the physical meaning of each EMD component, each component represents a frequency component in the original signal. Adding the individual IMF components represents the main changes in the original KPI data. Ultimately, the non-stationary signal is transformed into a stationary signal while preserving its instantaneous variations.

[0158] Then, SG filtering is applied to each mode separately to filter out noise in the instantaneous change curve, making the entire curve smoother, and finally reconstructing the curve.

[0159] The method in this invention extracts IMF components after EMD decomposition, and then reconstructs the components of each modality using SG filtering to finally construct a KPI curve that includes data variations and removes noise. The TCN-AE model used in this invention is essentially an AE model, which fits normal data patterns. Therefore, this invention uses a method based on EMD and SG filtering to construct a normal dataset that only includes data variations.

[0160] In step S120, feature vectors of the normal dataset are extracted and input into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

[0161] The data features are extracted, encompassing three aspects: first, basic statistical features: mean, variance, extreme values, bands, and power spectrum characteristics; second, time-domain features: mean, variance, extreme values, zero-crossing points, boundary points, and band length and peak values; and third, frequency-domain features: power spectrum, power density ratio, median frequency, and average power frequency characteristics. This constitutes a 15-dimensional feature vector.

[0162] After feature vector extraction, OneClassSVM is used to learn the data patterns of normal data. The idea of ​​OneClassSVM is very simple: find a minimum hypersphere to circle the positive examples in the samples, and make predictions using this hypersphere. Samples within the circle are considered positive samples.

[0163] After training the OneClassSVM anomaly detection model using training set samples, extract basic statistical features, time-domain features, and frequency-domain features from the test set or online data. Input these features into the trained model and output the anomaly detection results.

[0164] Embodiment 1 of this invention proposes a storage medium that filters the sample set using EMD and SG filtering to construct a normal sample dataset. Then, it extracts the basic statistical features, time-domain features, and frequency-domain features of the normal sample dataset to construct a data feature vector. The constructed feature vector is then input into OneClassSVM, which employs a supervised machine learning method. This method alleviates the trouble of manually labeling abnormal data to a certain extent and also solves the problem of the imbalance between abnormal and normal data on supervised learning.

[0165] This application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0166] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0167] For a description of the relevant parts of the processing device and medium for detecting time-series data anomalies on a cloud platform provided in this application embodiment, please refer to the detailed description of the corresponding parts in the cloud platform time-series data anomaly detection method provided in embodiment 1 of this application, and will not be repeated here.

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0169] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in time-series data on a cloud platform, characterized in that, Includes the following steps: The raw monitoring data from the cloud platform is obtained, segmented using a sliding window, and labeled as sample data, which includes normal samples and abnormal samples. Then, the sample data is preprocessed; the process of obtaining the original monitoring data of the cloud platform, segmenting it using a sliding window, and labeling it as sample data includes: collecting monitoring data of the cloud platform for a period of time as the original monitoring data, and setting the collection period; filtering out missing points in the original monitoring data after segmenting it using a sliding window; adding data labels to the original monitoring data, where the label for normal samples is 0 and the label for abnormal samples is 1; The preprocessing process for the sample data is as follows: The sample data is determined to be ; Using formula Perform normalization processing; in, Indicates the first One sample data, ; The sample data has been normalized. The preprocessed sample data is processed by EMD to obtain modal components. Each modal component is then smoothed and denoised using a filter to obtain a smoothed time series, which is then used to construct a normal dataset. The process of processing the preprocessed sample data into modal components by EMD includes: all normalized sample data forming KPI time series data. ; KPI time series data After EMD processing, it is the sum of each modal component and the margin: ； in, For the first One IMF component, For the total number of IMF funds, As a margin, The length of the data sample; The process of smoothing and denoising each modal component using an SG filter to obtain the smoothed time series is as follows: ; in, The final smoothed time series is obtained; F is the SG filter; The process of extracting feature vectors from a normal dataset includes: extracting basic statistical features, time-domain features, and frequency-domain features from the normal dataset; these features constitute the feature vectors of the normal dataset; the basic statistical features include mean, variance, extreme values, bands, and power spectrum features; the time-domain features include mean, variance, extreme values, zero-crossing points, boundary points, band length, and peak value features; the frequency-domain features include power spectrum, power density ratio, median frequency, and average power frequency features. The feature vectors of the normal dataset are extracted and input into the anomaly detection model to output anomaly detection data. The anomaly detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs anomaly detection data other than the normal data.

2. The cloud platform time-series data anomaly detection method according to claim 1, characterized in that, The abnormal data detection model adopts the OneClassSVM model.

3. A cloud platform time-series data anomaly detection system, used to execute the cloud platform time-series data anomaly detection method according to any one of claims 1 to 2, characterized in that, The system includes a preprocessing module, a decomposition and denoising module, and a detection module; The preprocessing module is used to acquire the original monitoring data of the cloud platform, segment it using a sliding window, and label it as sample data. The sample data includes normal samples and abnormal samples. Then, the sample data is preprocessed. The decomposition and denoising module is used to decompose the preprocessed sample data into modal components through EMD processing, and to perform smoothing and denoising processing on each modal component through a filter to obtain a smoothed time series and construct a normal dataset. The detection module is used to extract feature vectors from the normal dataset and input the feature vectors into the abnormal data detection model to output abnormal detection data. The abnormal data detection model constructs a minimum hypersphere to circle the normal data in the normal dataset and outputs abnormal detection data other than the normal data.

4. A device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 2.

5. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 2.

Images