A method and device for detecting abnormal data, a computer device and a storage medium

Abstract

The application discloses a method and device for detecting abnormal data, a computer device and a storage medium, and aims at solving the problem that when detecting abnormal time series data, the efficiency and accuracy of detection cannot be ensured, and the consumption of device performance is reduced. The method comprises the following steps: judging whether the time series data contains a trend; if yes, removing the trend in the time series data to obtain intermediate processing data; judging whether the intermediate processing data contains periodicity; if yes, removing the periodicity in the intermediate processing data to obtain residual series data; and inputting the residual series data into an unsupervised model based on Bayesian inference to detect whether the residual series data is abnormal.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection, and more particularly to a method, apparatus, computer device, and storage medium for detecting anomalous data. Background Technology

[0002] In today's technologically advanced world, many computer systems utilize machine learning algorithms to process and identify massive amounts of data in order to deduce desired results. Among this data, a sequence of values ​​for the same statistical indicator arranged chronologically is called time series data. Because time series data contains a series of timestamps and numerical values, it possesses strong unique characteristics and greater diversity in form compared to other types of data used in machine learning. Therefore, in data screening for machine learning, the ability to quickly and accurately detect outliers in time series data is crucial.

[0003] Existing technologies include many techniques for supervising the detection of time series data. Supervised model detection methods typically involve collecting a large amount of time series data with anomaly labels and then training a model capable of detecting anomalous time series data. However, in practical applications, this method requires extensive data collection and labeling, resulting in excessive consumption of human resources and equipment performance. Currently, supervised model detection of anomalous time series data cannot simultaneously guarantee high efficiency and accuracy while minimizing equipment resource consumption. Summary of the Invention

[0004] This invention provides a method, apparatus, computer device, and storage medium for detecting abnormal data, in order to solve the problem that when detecting abnormal time series data, it is impossible to ensure both high efficiency and accuracy in detection while reducing the consumption of equipment performance.

[0005] In a first aspect, the present invention provides a method for detecting abnormal data, comprising:

[0006] Determine whether time series data contains a trend;

[0007] If so, the trend in the time series data is removed to obtain intermediate processed data;

[0008] Determine whether the intermediate processed data contains periodicity;

[0009] If so, the periodicity in the intermediate processed data is removed to obtain the residual sequence data;

[0010] The residual sequence data is input into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0011] In one possible design, determining whether time series data contains a trend includes:

[0012] Calculate the difference in variable relationships between different data points in the time series data;

[0013] Construct the order column of the time series data using the difference in the relationships between the variables;

[0014] Calculate the variance of the ordered series;

[0015] The variance is used to transform the ordered column into a test statistic of a standard normal distribution;

[0016] The test statistic is used to determine whether the time series data contains the trend.

[0017] In one possible design, removing the trend from the time series data to obtain the intermediate processed data includes:

[0018] Calculate the sum of squared errors of the data points in the time series data that deviate from the trend term;

[0019] Multiply the penalty parameter by the sum of squared differences of the second difference of the trend term;

[0020] The intermediate processing data is obtained by calculating the minimum value of the sum of squared errors and the sum of squared differences.

[0021] In one possible design, determining whether the intermediate processed data contains periodicity includes:

[0022] The intermediate processed data is subjected to Fourier transform to convert the intermediate processed data into frequency domain data;

[0023] Calculate the energy density spectrum of the frequency domain data;

[0024] Filter out all frequency data preceding the expected sequence in the energy density spectrum.

[0025] The frequency data is subjected to an inverse Fourier transform to obtain the time-domain data of the frequency data.

[0026] The autocorrelation sequence of the time-domain data is calculated using the autocorrelation function;

[0027] Calculate the cluster centers of the autocorrelation sequences;

[0028] By using the data points in the clusters of the cluster centers, it can be determined whether the intermediate processed data contains periodicity.

[0029] In one possible design, calculating the cluster centers of the autocorrelation sequence includes:

[0030] Obtain the peak value of the autocorrelation sequence;

[0031] Calculate the interval between two adjacent peaks;

[0032] Filter out the low intervals whose length is lower than the expected length from the intervals;

[0033] All data points within the low interval are grouped into the same cluster;

[0034] Clusters in which the number of data points is greater than the expected number are designated as cluster centers.

[0035] In one possible design, removing the periodicity from the intermediate processed data to obtain the residual sequence data includes:

[0036] Obtain the data points from the intermediate processing data;

[0037] Obtain a plurality of preceding data points before the data point, and a plurality of subsequent data points after the data point, also a plurality of expected numbers of preceding data points.

[0038] Calculate the mean of the multiple preceding data points and multiple subsequent data points within the expected time period;

[0039] The periodic term in the intermediate processed data is calculated using the mean.

[0040] The periodic term is removed from the intermediate processed data to obtain the residual sequence data.

[0041] In one possible design, inputting the residual sequence data into an unsupervised model based on Bayesian inference to detect whether the residual sequence data is abnormal includes:

[0042] Obtain the data points from the residual sequence data;

[0043] Obtain historical sequence data within the expected window prior to the data point;

[0044] The probability of an anomaly in the data points is calculated using the unsupervised model obtained through Bayesian inference.

[0045] Determine whether the anomaly probability is higher than the expected anomaly threshold;

[0046] If the value is higher, then the data point is considered to be abnormal.

[0047] Secondly, an apparatus for detecting abnormal data is provided, comprising:

[0048] The first judgment module is used to determine whether time series data contains a trend;

[0049] The first processing module is configured to remove the trend from the time series data if the trend is present, and obtain the intermediate processed data.

[0050] The second judgment module is used to determine whether the intermediate processed data contains periodicity;

[0051] The second processing module is used to remove the periodicity from the intermediate processed data if the condition is met, and obtain residual sequence data.

[0052] The detection module is used to input the residual sequence data into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0053] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method for detecting abnormal data described above.

[0054] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for detecting abnormal data.

[0055] The aforementioned method, apparatus, computer equipment, and storage medium for detecting abnormal data determine whether the time series data contains a trend. If so, the trend is removed from the time series data to obtain intermediate processing data. The reason for removing the trend is that the trend will affect the periodicity determination in subsequent steps; that is, it will suppress the frequency corresponding to the periodicity in the time series data, making it impossible to accurately determine whether the intermediate processing data contains periodicity. Therefore, it is necessary to first remove the trend from the intermediate processing data, and then determine whether the intermediate processing data contains periodicity. If so, the periodic component is removed from the intermediate processing data to obtain residual sequence data. The reason for removing the periodicity from the intermediate processing data is that both periodicity and trend will affect the accuracy of anomaly detection by the subsequent unsupervised model. Furthermore, the idea of ​​data classification is incorporated into the time series data detection process. Compared to the existing method of directly inputting unjudged time series data into a supervised model, this invention pre-classifies and filters the time series data, enabling the final detection model to better adapt to different data types, thereby achieving a more accurate and efficient anomaly detection effect. Finally, the residual sequence data is input into an unsupervised model based on Bayesian inference to detect anomalies in the residual sequence data. This unsupervised model, compared to existing supervised models, is more efficient at detecting anomalies in time series data. Unsupervised models do not require training with a large amount of labeled data, thus effectively reducing device performance consumption. Furthermore, since this unsupervised model is based on Bayesian inference, the inference algorithm can maintain inference accuracy while requiring minimal manual parameter tuning, ensuring both efficiency and accuracy in detecting anomalous time series data. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of an application environment for a method for detecting abnormal data according to an embodiment of the present invention;

[0058] Figure 2 This is a flowchart illustrating a method for detecting abnormal data in one embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of a device for detecting abnormal data in one embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The method for detecting abnormal data provided in this invention can be applied to, for example... Figure 1 In this application environment, terminal devices communicate with servers via a network. The terminal devices can acquire time-series data and then send it to the server, which performs anomaly detection on the data. Terminal devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers.

[0063] In one embodiment, such as Figure 2 As shown, a method for detecting abnormal data is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0064] S10: Determine whether the time series data contains a trend.

[0065] In this embodiment, the acquired time series data is first sent to the server. The time series data refers to a sequence of data points arranged in chronological order. The time series data is composed of various components, including but not limited to trend, period, and residual components.

[0066] The server then determines whether the time series data contains a trend. This trend refers to the entire process of deviation from the baseline over time, including but not limited to trend terms and trend components. For example, stock data over the past year will show an upward or downward trend; this trend is the trend of the stock data. Methods for determining trends include, but are not limited to, the slope method, the Cox-Stuart trend test, and the Mann-Kendall trend test.

[0067] S20: If so, remove the trend from the time series data to obtain intermediate processed data.

[0068] In the path where the time series data contains a trend, the server removes the trend from the time series data to obtain intermediate processed data. The methods used by the server to remove the trend from the time series data include, but are not limited to, differencing, logarithmic transformation, and linear regression.

[0069] It is worth noting that in this embodiment, removing the trend from the time series data in step S20 is because the trend will affect the periodicity determination in the subsequent step S30. That is, it will suppress the frequency corresponding to the periodicity in the time series data, making it impossible to accurately determine whether the intermediate processed data contains periodicity. Therefore, it is necessary to first remove the trend from the intermediate processed data, and then determine whether the intermediate processed data contains periodicity.

[0070] S30: Determine whether the intermediate processing data contains periodicity.

[0071] After obtaining the intermediate processed data from which trends have been removed, the server needs to determine whether the intermediate processed data contains periodicity. Periodicity refers to a wave-like or oscillating variation in time series data around a long-term trend. For example, network traffic peaks in the early morning of each day, and daily network traffic data exhibits periodicity. Methods for servers to detect whether intermediate processed data contains periodicity include, but are not limited to, periodogram-type algorithms and trend decomposition algorithms.

[0072] S40: If so, remove the periodicity from the intermediate processed data to obtain residual sequence data.

[0073] In the path where the intermediate processing data contains periodicity, the server removes the periodicity from the intermediate processing data to obtain residual sequence data. The methods for removing the periodicity from the intermediate processing data include, but are not limited to, classical decomposition methods, STL (Seasonal and Trend decomposition using Loess) decomposition methods, etc. The residual sequence data refers to the irregular data after removing periodicity and trend.

[0074] It is worth noting that step S40 in this embodiment removes the periodicity from the intermediate processed data because both periodic and trend components can affect the accuracy of anomaly detection by the subsequent unsupervised model. Furthermore, the idea of ​​data classification is incorporated into the detection of time series data here. Compared to the existing method of directly inputting unjudged time series data into a supervised model, this invention pre-classifies and filters the time series data, enabling the final detection model to better adapt to different data formats, thereby achieving more accurate and efficient anomaly detection.

[0075] S50: Input the residual sequence data into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0076] After obtaining the residual sequence data, the server inputs it into a statistical model based on Bayesian inference, i.e., an unsupervised model. The statistical model infers whether anomalies exist in the residual sequence, thus determining whether there are anomalies in the time series data. Bayesian inference is a method in inferential statistics. This method uses Bayes' theorem to update the probability of a specific hypothesis as more evidence and information become available. Bayesian inference is one of the most important techniques in statistics (especially mathematical statistics).

[0077] It is worth noting that a Bayesian inference statistical model is used as the unsupervised model in this embodiment because Bayesian inference only requires a few parameter fine-tunings when judging outliers, avoiding the heavy work of parameter tuning.

[0078] It should be noted that this embodiment employs an unsupervised model for data detection. Compared to supervised models in existing technologies, unsupervised models are more efficient at detecting anomalies in time series data. Unlike supervised models, unsupervised models do not require accumulating large amounts of labeled data for training, thus effectively reducing device performance consumption. Furthermore, compared to other unsupervised models in existing technologies, the Ye Bayes inference-based unsupervised model in this embodiment can maintain inference accuracy while requiring minimal manual parameter tuning, reducing the amount of parameter tuning work.

[0079] In addition, since the residual sequence data has had its trends and periodicity removed, the impact of periodicity and trends on the unsupervised model is reduced, making the unsupervised model better adaptable to different types of data, thus ensuring the efficiency and accuracy of detecting abnormal time series data.

[0080] In one embodiment, before step S10, i.e. before determining whether the time series data contains a trend, the server can preprocess the time series data. The preprocessing methods include, but are not limited to, handling missing values ​​and removing noise.

[0081] In one embodiment, step S10, namely determining whether the time series data contains a trend, specifically includes the following steps:

[0082] S11: Calculate the difference in variable relationships between different data points in the time series data.

[0083] S12: Construct the ordered column of the time series data by using the difference in the relationship between the variables.

[0084] S13: Calculate the variance of the ordered series.

[0085] S14: Using the variance, transform the ordered sequence into a test statistic of a standard normal distribution.

[0086] S15: Determine whether the time series data contains the trend by using the test statistic.

[0087] In this embodiment, firstly, the server calculates the difference in the variable relationship between two different data points in the time series data. Then, using this difference, an ordered column of the time series data is constructed. The ordered column refers to the cumulative number of data points at one time point that are greater than data points at another time point; that is, the ordered column represents the number of positive sign function values ​​minus the number of negative sign function values. Next, the variance of this ordered column is calculated, and by calculating the variance, the ordered column is transformed into a test statistic for a standard normal distribution. Finally, the test statistic is used to determine whether the time series data contains a trend.

[0088] Specifically, if the length of the time series data X is n, then we first calculate the difference in the variable relationships between different data points, d[j,k] = X. j -X k .

[0089] Among them, X j Let X represent the data point of the j-th time series data. k Represents the k-th data point

[0090] Next, the ordered column S of the time series data is constructed, using the following formula:

[0091]

[0092] The range of values ​​for sgn is:

[0093] Next, the variance VAR(S) of the ordered series is calculated using the following formula:

[0094]

[0095] Where, q k This represents the value of the data point in the k-th group, where k represents the group number of the same data point, and p represents the number of groups formed by the same data point.

[0096] For example, X = {12, 56, 23, 12, 67, 45, 56, 56, 10} contains identical data points. These identical data points are grouped into two groups, i.e., p = 2. In this case, the first group is {12} with q1 = 2, and the second group is {56} with q2 = 3.

[0097] Finally, the above data are used to form an approximate standard normal distribution using the test statistic Z, the calculation formula of which is as follows:

[0098]

[0099] The above formula determines whether the time series contains a trend. In other words, the test statistic Z indicates whether the time series data has a trend of increasing or decreasing over time, that is, whether the test statistic Z is positive or negative.

[0100] It should be noted that this embodiment essentially examines the sign difference between early and late data points in the time series data, thereby detecting the monotonicity of the time series data. Monotonicity includes, but is not limited to, upward trends, downward trends, and no trend. If an upward or downward trend exists, the sign value will continuously increase or decrease. Each value is compared with every preceding value in the time series data, where n is the number of observations in the time series data. This method requires no assumptions about the time series data and can be used for various types of time series data, including non-normally distributed data. This embodiment improves the accuracy of trend detection in time series data, effectively preventing trend detection errors from affecting subsequent periodicity judgments, thus ensuring the efficiency of anomaly detection and improving detection speed while maintaining accuracy.

[0101] In one embodiment, step S20, namely removing the trend from the time series data, specifically includes the following steps:

[0102] S21: Calculate the sum of squared errors of the data points in the time series data that deviate from the trend term.

[0103] S22: Multiply the penalty parameter by the sum of squared differences of the second difference of the trend term.

[0104] S23: The intermediate processing data is obtained by calculating the minimum value of the sum of the squared errors and the sum of the squared differences.

[0105] In this embodiment, the error of data points deviating from the trend term in the time series data is first calculated, and the sum of squares of the errors is obtained. Then, the penalty parameter is multiplied by the sum of squares of the second difference of the trend term, and the two are added together to obtain the minimum value. The target trend term is obtained from the minimum value, and the final intermediate processing data can be obtained from the target trend term. Here, the penalty parameter refers to a preset positive number.

[0106] Specifically, this method is similar to time series decomposition, letting {y i Let i = 1, 2, ..., g be the logarithm of a time series dataset, then {y} i} consisting of a series of trend terms g iIt consists of a periodic term j and an error term k, which is y i =(g i +j+k). If a suitable penalty parameter λ is preset, then there exists a trend term that satisfies the following objective function formula:

[0107]

[0108] In the formula above, the first term represents the sum of squares of the errors in which data points deviate from the trend term, thus controlling the magnitude of the trend term. The second term is multiplied by the sum of squares of the second difference of the trend term, thus controlling the drastic change in the trend term. The larger λ is, the stronger the control of the trend term.

[0109] Among them, y i Let g represent the i-th data point in the time series data. i Let λ represent the i-th data point indicating a trend, and let λ represent the penalty term parameter used to penalize changes in the trend. i+1 The i+1th data point represents the trend, and N is the length of the time series data.

[0110] It should be noted that this embodiment essentially extracts the trend portion from the time series data and obtains a smooth curve to represent the entire time series, which is the final intermediate processing data. In other words, it transforms time series data, which is more sensitive to short-term fluctuations, into intermediate processing data, which is more sensitive to long-term fluctuations. Formulaically, the sensitivity is adjusted by a penalty parameter. This method is both accurate and efficient, and it can effectively prevent trend detection errors from affecting subsequent judgments of periodicity, thereby ensuring the efficiency of anomaly detection. While ensuring the accuracy of anomaly detection, it also improves the detection speed.

[0111] In one embodiment, step S30, namely determining whether the intermediate processing data contains periodicity, specifically includes the following steps:

[0112] S31: Perform a Fourier transform on the intermediate processed data to convert the intermediate processed data into frequency domain data.

[0113] S32: Calculate the energy density spectrum of the frequency domain data.

[0114] S33: Filter out all frequency data preceding the expected sequence in the energy density spectrum.

[0115] S34: Perform an inverse Fourier transform on the frequency data to obtain the time-domain data of the frequency data.

[0116] S35: Calculate the autocorrelation sequence of the time-domain data using the autocorrelation function.

[0117] S36: Calculate the cluster centers of the autocorrelation sequence.

[0118] S37: Determine whether the intermediate processing data contains periodicity by using the data points in the clusters of the cluster center.

[0119] In this embodiment, firstly, the intermediate processing data is subjected to a Fourier transform, converting it to the frequency domain, i.e., into frequency domain data. Then, the energy density spectrum of the frequency domain data is calculated, and the frequency data preceding the expected sequence in the energy density spectrum is found; that is, the top n frequencies with the highest energy percentage in the energy density spectrum are found. For example, when the period of the time series data is daily, weekly, or monthly, n in the top n can usually be set to 3, representing the top three frequencies in the time series data. The formula for calculating the energy density spectrum of the frequency domain data is:

[0120] P(f k / N )=‖X(f k / N )‖ 2

[0121] in, f k / N X(f) represents the frequency at k / N in the spectrum, where k represents the k-th data point and N represents the data length. k / N )‖ represents f k / N The spectral function value, P(f k / N ) represents the energy density spectrum of frequency domain data.

[0122] Next, the top n frequency data are extracted one by one using inverse Fourier transform and transformed into time-domain data. Simultaneously, the autocorrelation function (ACF) is calculated to obtain the autocorrelation sequence. Then, the cluster centers of the autocorrelation sequence are calculated, and the periodicity of the intermediate processed data is determined by the data points in the clusters of these centers. Methods for calculating the cluster centers of the autocorrelation sequence include, but are not limited to, Mean Shift, K-means, and DBSCAN (Density-Based Spatial Clustering of Applications with Noise). Clustering is a technique for finding the inherent structure among data points. Clustering divides all data points into multiple similar groups, and these similar groups are called clusters. Data points in the same cluster are identical to each other, and data points in different clusters are different from each other.

[0123] It should be noted that the periodicity of the intermediate processed data is determined in this embodiment to prevent the periodicity from affecting the accuracy of anomaly detection by the subsequent unsupervised model. Furthermore, the idea of ​​data classification is incorporated into the detection of time series data. Compared to the existing method of directly inputting unjudged time series data into a supervised model, this invention pre-classifies and filters the time series data, enabling the final detection model to better adapt to different data types, thereby achieving more accurate and efficient anomaly detection.

[0124] In one embodiment, step S37, namely determining whether the intermediate processing data contains periodicity based on the data points in the clusters of the cluster center, includes:

[0125] S371: Set the expected quantity.

[0126] S372: Determine whether the number of data points in the cluster is higher than the expected number.

[0127] S373: If it is greater than or equal to, then the intermediate processing data contains periodicity.

[0128] S374: If it is lower than that, then the intermediate processing data does not contain periodicity.

[0129] In this embodiment, it is determined whether the total number of data points in the cluster is higher than the expected number. If it is lower, it is considered that the cycle corresponding to the data processing of the center does not exist.

[0130] It should be noted that this embodiment classifies intermediate processing data based on whether the number of data points in the cluster is higher than the expected number, thereby removing redundant data and directly filtering the data that meets the conditions. This makes the overall periodicity judgment more efficient, eliminating the need for redundant calculations. While improving the efficiency of periodicity detection, it also ensures accuracy, effectively preventing periodicity detection errors from affecting the subsequent unsupervised model's anomaly detection of residual sequence data. This ensures the efficiency of anomaly detection and improves detection speed while maintaining the accuracy of anomaly detection.

[0131] In one embodiment, step S36, which involves calculating the cluster centers of the autocorrelation sequence, specifically includes the following steps:

[0132] S361: Obtain the peak value of the autocorrelation sequence.

[0133] S362: Calculate the interval between two adjacent peaks.

[0134] S363: Filter out low intervals whose length is lower than the expected length from the intervals.

[0135] S364: Aggregate all data points within the low interval into the same cluster.

[0136] S365: The cluster with more than the expected number of data points in the cluster is taken as the cluster center.

[0137] In this embodiment, the peak values ​​of the autocorrelation sequence are first obtained, and the intervals between the peak values ​​are calculated; this interval is the period length. Then, intervals shorter than the expected length are designated as low intervals. All data points within the low intervals are clustered into the same cluster. Finally, clusters with more data points than the expected number are designated as cluster centers. The autocorrelation sequence is an ACF sequence, which is the sequence calculated using the autocorrelation function.

[0138] Specifically, the interval between peaks is calculated because the peak intervals of an autocorrelation sequence exhibit fluctuations of inconsistent length. The method for obtaining the peaks of the autocorrelation sequence involves first calculating the first difference of the autocorrelation sequence, resulting in a first-order difference sequence. Then, consecutive pairs of values ​​greater than zero and less than zero are searched within the first-order difference sequence. This leads to the derivation of the autocorrelation sequence, finding the midpoint among the three points constituting the value pair. If the selected value point is greater than the mean of the autocorrelation sequence, then that value point is the peak of the autocorrelation sequence.

[0139] Next, a clustering algorithm is used to group peak intervals (i.e., period lengths) with similar lengths into the same cluster. By calculating the cluster center of the cluster with the most data points—that is, calculating the data point density—the accurate period length, or accurate interval, can be obtained. This method effectively enhances the robustness of period extraction. The formula for calculating the mean center of the data points is as follows:

[0140]

[0141] Among them, M h (x) represents the calculated cluster centers, and x represents the data point currently being calculated. i Let w(x) represent the i-th data point in the autocorrelation sequence. i ) represents the weight of the i-th data point, n represents the number of data points, and G H This refers to a kernel function, which includes, but is not limited to, Gaussian kernels and polynomial kernels.

[0142] It should be noted that this embodiment essentially uses a data point density calculation method to determine the periodicity of intermediate processed data. This method not only enhances the robustness of periodic extraction but also performs more accurate and efficient classification and screening of time series data, thereby effectively improving the accuracy and efficiency of periodic detection. It effectively prevents periodic detection errors from affecting the subsequent unsupervised model's anomaly detection of residual sequence data, thus ensuring the efficiency of anomaly detection and improving the detection speed while ensuring the accuracy of anomaly detection.

[0143] In one embodiment, step S40, namely removing the periodicity from the intermediate processed data to obtain residual sequence data, specifically includes the following steps:

[0144] S61: Obtain the data points in the intermediate processing data.

[0145] S62: Obtain a plurality of preceding data points of an expected number before the data point, and a plurality of following data points of the expected number after the data point.

[0146] S63: Calculate the mean of the plurality of previous data points and the plurality of subsequent data points within the expected time period.

[0147] S64: Calculate the periodic term in the intermediate processing data using the mean.

[0148] S65: Remove the periodic term from the intermediate processing data to obtain the residual sequence data.

[0149] In this embodiment, the server needs to remove periodicity from the intermediate processing data, that is, remove periodic terms. Specifically, any data point in the intermediate processing data is obtained, and a predetermined number of data points preceding this data point are designated as the preceding data points, and a predetermined number of data points following this data point are designated as the following data points. The mean of the preceding and following data points within a predetermined time period is calculated. Using this mean, the periodic terms in the intermediate processing data are calculated. Finally, the periodic terms in the intermediate processing data are removed, yielding the residual sequence data.

[0150] For example, for monthly data, the periodic term for March is the average of all March data after removing any trend. These periodic terms are then adjusted so that their sum is zero. The periodic term is simply a combination of these data points arranged together.

[0151] Expressing the above steps using a formula yields the following formula:

[0152]

[0153] Where m = 2k + 1, which means the moving average at time t is the average of the previous data points and the next data points.

[0154] It should be noted that the periodicity of the intermediate processing data was removed in this embodiment to prevent periodicity from affecting the accuracy of anomaly detection by the subsequent unsupervised model. This makes the data input to the unsupervised model more accurate, prevents periodicity from causing unpredictable effects on the unsupervised model, and enables the final detection model to better adapt to different data formats, thereby achieving more accurate and efficient anomaly detection results.

[0155] In one embodiment, step S50 involves inputting the residual sequence data into an unsupervised model based on Bayesian inference to detect whether the residual sequence data is abnormal. This specifically includes the following steps:

[0156] S51: Obtain the data points in the residual sequence data.

[0157] S52: Obtain historical sequence data within the expected window prior to the data point.

[0158] S53: Calculate the probability of anomalies in the data points using the unsupervised model obtained through Bayesian inference.

[0159] S54: Determine whether the abnormal probability is higher than the expected abnormal threshold.

[0160] S55: If it is higher, then the data point is determined to be abnormal.

[0161] In this embodiment, the server acquires any data point from the residual sequence data. Then, it acquires historical sequence data within a predetermined window preceding that data point. The predetermined window refers to the time series data of a preset unit length, used to calculate statistical indicators within the window. Next, the server calculates the probability that the data point is an anomalous data point using a statistical model obtained through Bayesian inference, i.e., an unsupervised model. Finally, it determines whether the probability of an anomalous data point in the residual sequence data is higher than a predetermined anomalous threshold. If it is higher, the data point is determined to be anomalous; if it is lower than or equal to the predetermined anomalous threshold, the data point is determined to be normal. The predetermined window and the predetermined anomalous threshold are model parameters that can be adjusted.

[0162] In addition, the formula for calculating the outlier probability of data points in the T-distribution statistical model obtained by Bayesian inference is as follows:

[0163]

[0164]

[0165]

[0166] Among them, D newLet D represent the residual sequence data, x represent the historical sequence data within the expected window preceding the current data point, Γ represent the data point in the residual sequence data, and α represent the gamma function. n β represents the shape parameter in the posterior probability distribution (gamma distribution). n μ represents the scaling parameter in the posterior probability distribution (gamma distribution). n ∧ represents the expected parameter in the posterior probability distribution (gamma distribution), and ∧ represents the accuracy.

[0167] It should be noted that using a statistical model derived from Bayesian inference, i.e., an unsupervised model, to detect anomalies in residual sequence data is more efficient than supervised models in existing technologies. Unsupervised models do not require training with a large amount of labeled data, thus effectively reducing equipment performance consumption. Furthermore, since this unsupervised model is based on Bayesian inference, the inference algorithm can maintain inference accuracy with minimal manual parameter tuning, ensuring both efficiency and accuracy in detecting anomalous time series data. Additionally, the model derived from Bayesian inference requires minimal manual parameter tuning; in this embodiment, only the expected window and expected anomaly threshold need to be adjusted, significantly saving labor costs and improving anomaly detection efficiency.

[0168] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0169] In one embodiment, an apparatus for detecting abnormal data is provided, which corresponds one-to-one with the method for detecting abnormal data in the above embodiments. For example... Figure 3 As shown, the device for detecting abnormal data includes a first judgment module 10, a first processing module 20, a second judgment module 30, a second processing module 40, and a detection module 50. Detailed descriptions of each functional module are as follows:

[0170] The first judgment module 10 is used to determine whether the time series data contains a trend.

[0171] The first processing module 20 is configured to remove the trend from the time series data if the trend is present, and obtain the intermediate processed data.

[0172] The second judgment module 30 is used to determine whether the intermediate processed data contains periodicity;

[0173] The second processing module 40 is used to remove the periodicity from the intermediate processed data if the condition is met, and obtain residual sequence data.

[0174] The detection module 50 is used to input the residual sequence data into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0175] Specific limitations regarding the device for detecting abnormal data can be found in the limitations of the method for detecting abnormal data described above, and will not be repeated here. Each module in the aforementioned device for detecting abnormal data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0176] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores various data generated during the implementation of methods for detecting anomalous data, such as time-series data, intermediate processing data, residual sequence data, and unsupervised models. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting anomalous data.

[0177] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0178] Determine whether time series data contains a trend;

[0179] If so, the trend in the time series data is removed to obtain intermediate processed data;

[0180] Determine whether the intermediate processed data contains periodicity;

[0181] If so, the periodicity in the intermediate processed data is removed to obtain the residual sequence data;

[0182] The residual sequence data is input into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0184] Determine whether time series data contains a trend;

[0185] If so, the trend in the time series data is removed to obtain intermediate processed data;

[0186] Determine whether the intermediate processed data contains periodicity;

[0187] If so, the periodicity in the intermediate processed data is removed to obtain the residual sequence data;

[0188] The residual sequence data is input into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence data.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0190] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0191] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting abnormal network traffic data, characterized in that, include: Determine whether time-series network traffic data contains a trend; If so, the trend in the time series network traffic data is removed to obtain intermediate processed network traffic data; Determine whether the intermediate network traffic data contains periodicity; If so, the periodicity in the intermediate network traffic data is removed to obtain the residual sequence network traffic data; The residual sequence network traffic data is input into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence network traffic data. The step of inputting the residual sequence network traffic data into an unsupervised model based on Bayesian inference to detect whether the residual sequence network traffic data is abnormal includes: obtaining data points in the residual sequence network traffic data; obtaining historical sequence network traffic data within the expected window before the data points; calculating the probability of the data points being abnormal using the T-distribution statistical model obtained by Bayesian inference; determining whether the probability of abnormality is higher than the expected abnormality threshold; if it is higher, then determining that the data points are abnormal. The formula for calculating the anomaly probability of data points in the T-distribution statistical model obtained by Bayesian inference is as follows: in, Let represent the residual sequence network traffic data, D represent the historical sequence network traffic data within the expected window preceding the current data point, and x represent the data point in the residual sequence network traffic data. Represents the gamma function. This represents the shape parameter in the posterior probability distribution. This represents the scaling parameter in the posterior probability distribution. Denotes the expected parameter in the posterior probability distribution. Indicates accuracy.

2. The method for detecting abnormal network traffic data as described in claim 1, characterized in that, The determination of whether time-series network traffic data contains a trend includes: Calculate the difference in variable relationships between different data points in the time series network traffic data; Construct the ordered column of the time series network traffic data by using the difference in the variable relationships; Calculate the variance of the ordered series; The variance is used to transform the ordered column into a test statistic of a standard normal distribution; The test statistic is used to determine whether the time series network traffic data contains the trend.

3. The method for detecting abnormal network traffic data as described in claim 1, characterized in that, The step of removing the trend from the time-series network traffic data to obtain the intermediate-processed network traffic data includes: Calculate the sum of squared errors of the data points in the time series network traffic data that deviate from the trend term; Multiply the penalty parameter by the sum of squared differences of the second difference of the trend term; The intermediate network traffic data is obtained by calculating the minimum value of the sum of squared errors and the sum of squared differences.

4. The method for detecting abnormal network traffic data as described in claim 1, characterized in that, The step of determining whether the intermediate network traffic data contains periodicity includes: The intermediate processing network traffic data is subjected to Fourier transform to convert the intermediate processing network traffic data into frequency domain network traffic data. Calculate the energy density spectrum of the frequency domain network traffic data; Filter out all frequency network traffic data preceding the expected sequence in the energy density spectrum; The frequency network traffic data is subjected to inverse Fourier transform to obtain the time-domain network traffic data of the frequency network traffic data. The autocorrelation sequence of the time-domain network traffic data is calculated using the autocorrelation function. Calculate the cluster centers of the autocorrelation sequences; By using the data points in the clusters of the cluster centers, it can be determined whether the intermediate processing network traffic data contains periodicity.

5. The method for detecting abnormal network traffic data as described in claim 4, characterized in that, The calculation of the cluster centers of the autocorrelation sequence includes: Obtain the peak value of the autocorrelation sequence; Calculate the interval between two adjacent peaks; Filter out the low intervals whose length is lower than the expected length from the intervals; All data points within the low interval are grouped into the same cluster; Clusters in which the number of data points is greater than the expected number are designated as cluster centers.

6. The method for detecting abnormal network traffic data as described in claim 1, characterized in that, The step of removing the periodicity from the intermediate network traffic data to obtain residual sequence network traffic data includes: Obtain data points from the intermediate network traffic data; Obtain a plurality of preceding data points before the data point, and a plurality of subsequent data points after the data point, also a plurality of expected numbers of preceding data points. Calculate the mean of the multiple preceding data points and multiple subsequent data points within the expected time period; The periodicity term in the intermediate processing network traffic data is calculated using the mean value. The periodic term is removed from the intermediate network traffic data to obtain the residual sequence network traffic data.

7. A device for detecting abnormal network traffic data, characterized in that, include: The first judgment module is used to determine whether the time series network traffic data contains a trend; The first processing module is configured to remove the trend from the time-series network traffic data if the trend is present, and obtain intermediate processed network traffic data. The second judgment module is used to determine whether the intermediate processed network traffic data contains periodicity. The second processing module is used to remove the periodicity from the intermediate network traffic data if the condition is met, and obtain residual sequence network traffic data. The detection module is used to input the residual sequence network traffic data into an unsupervised model based on Bayesian inference to detect whether there are any anomalies in the residual sequence network traffic data. The step of inputting the residual sequence network traffic data into an unsupervised model based on Bayesian inference to detect whether the residual sequence network traffic data is abnormal includes: obtaining data points in the residual sequence network traffic data; obtaining historical sequence network traffic data within the expected window before the data points; calculating the probability of the data points being abnormal using the T-distribution statistical model obtained by Bayesian inference; determining whether the probability of abnormality is higher than the expected abnormality threshold; if it is higher, then determining that the data points are abnormal. The formula for calculating the anomaly probability of data points in the T-distribution statistical model obtained by Bayesian inference is as follows: in, Let represent the residual sequence network traffic data, D represent the historical sequence network traffic data within the expected window preceding the current data point, and x represent the data point in the residual sequence network traffic data. Represents the gamma function. This represents the shape parameter in the posterior probability distribution. This represents the scaling parameter in the posterior probability distribution. Denotes the expected parameter in the posterior probability distribution. Indicates accuracy.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting abnormal network traffic data as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for detecting abnormal network traffic data as described in any one of claims 1 to 6.

Images