Transition point detection system and method based on large-scale network traffic type data

Abstract

The present invention discloses a jump point detection system and method based on large-scale network traffic type data, which specifically includes the following steps: preprocessing the data of the reference window, the preprocessing includes three steps of missing value filling, data normalization and smoothing; using the discrete Fourier transform combined with the autocorrelation coefficient method to classify the preprocessed time series; converting different categories of time series into binary sequences with 0 and 1, and identifying the jump point by the density change of the front and rear windows 1, and using a bidirectional detection method to determine whether the data jumps up or down. The method of the present invention does not need to assume the distribution of the data in advance, has good robustness, and the complexity of the algorithm is low, and can be well applied in large-scale data processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of time series anomaly detection, and in particular relates to a time series jump point detection method. Background Art

[0002] Jump point detection is a basic problem in the field of data analysis. Its purpose is to find the moment when the system behavior suddenly changes in the time series, so as to timely discover and solve the actual problems hidden behind the data jump. In real life, many systems will generate massive amounts of network traffic type data (such as wireless communication network load data, power grid power demand data, etc.). These data are affected by human behavior and activities and usually show certain regularity. Jump point detection for such data with certain regularity or seasonality has always been a hot direction in the field of data analysis.

[0003] At present, many methods related to jump point detection have been proposed. The traditional parameterization method needs to assume the distribution of data in advance. Its principle is to identify data that deviates from the model distribution as jump data. This method strongly relies on the assumed model. When the data does not obey the assumed distribution, the performance of the algorithm is poor. Later, some people used optimization methods to segment time series and determine the location of the jump point by selecting the segmentation method with the smallest loss function value. However, such methods have high complexity and are difficult to be well applied in large-scale data processing. With the successful application of deep learning methods in various fields, many deep learning models are also used to detect jump points, but this method also has high complexity, and some models require a large number of data labels. In the absence of labeled data, the model is difficult to achieve satisfactory results. Summary of the invention

[0004] In view of the problems existing in the above methods, the present invention provides a system and method for detecting jump points based on large-scale network traffic type data. The core idea of ​​the method is to convert the original time series into a binary sequence (a time series composed of 0 and 1), and identify the jump points by finding the moment when the density of 1 (or 0) in the binary sequence changes. The method does not need to assume the distribution of data in advance, has good robustness, and has low complexity, and can be well applied in the detection of jump points in massive time series.

[0005] The present invention discloses a time series jump point detection method with low complexity and good performance for large-scale network traffic type data. The method uses part of the historical data before the period to be detected as a reference window, and the data of the period to be detected as a test window, and detects whether the original time series data has a jump by judging whether there is a difference in the density of the binary sequence 1 of the front and back windows.

[0006] The jump point detection method based on large-scale network traffic type data specifically includes the following steps:

[0007] Step 1: Preprocessing of reference window data;

[0008] Preprocessing includes two steps: filling missing values ​​of time series, data normalization and smoothing. The processed time series helps to improve the accuracy of sequence classification.

[0009] Step 2: Classification of reference window sequences;

[0010] According to human behavior patterns, the data is divided into three different types of time series: one-day time series, one-week time series, and non-periodic time series. Discrete Fourier transform combined with autocorrelation coefficient is used to classify time series.

[0011] Step 3: Detection of density differences in binary time series;

[0012] For different types of time series, the steps of jump point detection are the same, but the parameters of different types of sequences are selected differently. This step first converts the original time series into a binary sequence, and then identifies whether there is a difference in the density of 1 in the binary sequence of the reference window and the test window to detect the change point. Density difference detection is divided into forward detection and backward detection. Forward detection uses the reference window to detect whether the test window has a jump, while reverse detection uses the test window to detect whether the reference window has a jump.

[0013] Preferably, step 1 is as follows:

[0014] Let the time series be x = {x1, x2, ..., x L}, where L is the length of the sequence, x t is the specific value of sequence x at a certain moment, and t∈{1, 2, …, L}, and let the reference window and test window be W respectively ref , W test , the window lengths are l1 and l2 respectively, then for the data x t For example, its reference window data can be expressed as The test window data can be represented as

[0015] Step 1.1, fill in missing values.

[0016] The traffic monitoring system collects system data at regular intervals (e.g., every hour), but sometimes the system is affected by internal or external factors, resulting in partial data loss. Generally, the proportion of system data loss is very small, so the linear interpolation method is used here to fill in the missing data of the reference window.

[0017] Step 1.2, data normalization.

[0018] Since there are many types of network traffic data and the ranges of different types of data vary greatly, the MIN-MAX method is used to normalize the data and limit the reference window data to the range of [0, 1] to facilitate data comparison and processing.

[0019] Step 1.3, smoothing.

[0020] In order to avoid the influence of noise and outliers on the classification of sequences, the exponentially weighted moving average algorithm is used to smooth the reference window data to improve the accuracy of sequence classification.

[0021] Preferably, step 2 is as follows:

[0022] Network traffic type data, driven by human activities, usually shows strong regularity with one-day and one-week cycles. Usually, the size of candidate cycles can be obtained through visual analysis. Here, according to human behavior patterns, the data is divided into three different types of time series: one-day time series, one-week time series, and non-periodic time series. Discrete Fourier transform combined with autocorrelation coefficient is used to classify time series.

[0023] Step 2.1, performing Discrete Fourier Transform (DFT) on the reference window data.

[0024] For a given reference window W of length l1 ref , assuming that X is the discrete Fourier transform of the preprocessed time series, then the periodogram Determined by the square of each Fourier coefficient:

[0025]

[0026] in:

[0027]

[0028] Where k is the harmonic component order, f k is the magnitude of the kth harmonic component, j is an imaginary unit, and n∈{0, 1, …, l1-1}. Let the magnitudes of the period with one day and one week be T1 and T2 respectively, then the magnitudes of the corresponding frequencies in the frequency domain are 1 / T1 and 1 / T2 respectively. Select the frequency with the largest amplitude in the spectrum except the DC component. If the frequency is not 1 / T1 or 1 / T2, the sequence is determined to be a non-periodic sequence, otherwise continue to step 2.2.

[0029] Step 2.2, calculate the autocorrelation coefficient of the reference window data.

[0030] The calculation formula of the autocorrelation coefficient (ACF) with a lag phase of l is:

[0031]

[0032] Where i∈{0, 1, …, l1-1}, for a sequence with a period of T (T∈{T1, T2}), when the lag phase l is T, 2T, 3T, …, the time series will show strong correlation. According to this principle, ACF(m1T1), ACF(m2T2) and the predetermined autocorrelation coefficient threshold r are used to calculate the correlation coefficient. ts Compare, where m1 and m2 are positive integers, and when selecting, it should be ensured that m1T1 and m2T2 are not much different. If ACF(m1T1)>r ts or ACF(m2T2)>r ts , then the sequence is identified as a periodic sequence, otherwise, it is marked as a non-periodic sequence. If the sequence is identified as a periodic sequence, it is necessary to compare the size of ACF(m1T1) and ACF(m2T2) to determine the size of the period. The comparison formula is:

[0033]

[0034] Preferably, step 3 is as follows:

[0035] After step 2, the type of time series is determined. For different types of time series, the steps of jump point detection are the same, but the parameters of different types of sequences are selected differently.

[0036] First, the original time series is converted into a binary sequence, and then the density of 1 in the binary sequence of the reference window and the test window is identified to detect the change point. Density difference detection is divided into forward detection and backward detection. Forward detection uses the reference window to detect whether the test window has a jump, while backward detection uses the test window to detect whether the reference window has a jump. Forward detection and backward detection have the same principle. The following is a detailed introduction to the principle of forward detection.

[0037] Step 3.1, convert the reference window and test window sequences into binary sequences.

[0038] Since the transition point may appear in the form of an upward or downward jump, two thresholds are used, namely the upper threshold u ts and the lower threshold l ts , converting the original time series into two different binary sequences to detect the upward and downward jump points. Here, the upper threshold u tsSet to:

[0039] u ts =Q3+k1*IQR (5)

[0040] Where IQR = Q3-Q1, Q3 and Q1 are the upper and lower quartiles of the reference window, respectively, and k1 is a hyperparameter. Get the upper threshold u ts Finally, the original data of the reference window and the test window are converted into binary values. The conversion formula is:

[0041]

[0042] where y t is the binary value of the time series at time t. ts Set to:

[0043] l ts =Quantile(k2) (7)

[0044] Quantile(·) represents the quantile of the time series, k2 is a hyperparameter and needs to be set manually based on the data characteristics of the sequence. The corresponding conversion formula is:

[0045]

[0046] Since a time series with a cycle of one week usually has different data characteristics on weekdays and weekends, different upper thresholds u are set for weekday and weekend data respectively. ts and the lower threshold l ts For the other two types of series, the threshold settings for weekdays and weekends are the same.

[0047] Step 3.2, determine whether there is a difference in the density of 1s in the binary sequences of the reference window and the test window.

[0048] Once the binary sequences of the reference and test windows are obtained (based on the upper or lower threshold conversion), the reference sequence Y ref The density ρ of 1 is calculated as follows:

[0049]

[0050] where sum(Y ref ) represents the label sequence Y ref Sum, assuming that the samples in the binary sequence are independently drawn from the same distribution, that is, follow the binomial distribution, then the probability that there are at least q 1 samples in the test window is calculated as follows:

[0051]

[0052] When the probability of f(q|ρ) is close to 0 (the left tail probability or right tail probability of the probability distribution), it means that observing q 1 samples in the test window is a rare event, indicating that the density of 1 samples has changed. The increase in the density of 1 samples means an upward change. At this time, the right tail probability is calculated by formula (10) to detect the upward jump point. Similarly, the decrease in the density of 1 samples means a downward change, and the left tail probability can be calculated to detect the downward jump point.

[0053] Since it is numerically difficult to directly calculate f(q|ρ), the normal distribution is used to approximate the binomial distribution according to the central limit theorem. The mean μ and standard deviation σ of the normal distribution are calculated using the following formula:

[0054] μ=l2*ρ (11)

[0055]

[0056] After obtaining the mean μ and standard deviation σ of the normal distribution, the N-sigma criterion is used to detect the upward or downward jump point. That is, for the binary sequence generated by applying the upper threshold uts to detect upward changes, the test threshold is set to c ts =μ+N1*σ, where N1 is a hyperparameter. test The number of samples with 1 in the above formula exceeds c ts , then an upward change is detected. For applying the lower threshold l ts The binary sequence generated by detecting downward changes, the test threshold is set to c ts =μ-N2*σ, where N2 is also a hyperparameter. test The number of samples in 1 is less than c ts , then a downward change is detected. It is worth noting that when ρ is very small, it means that Y ref The density of 1 is very small, so Y test When the density of 1 changes slightly, it is easy to misjudge it as jump data, so it is necessary to set a density threshold ρ for ρ ts , when ρ is less than ρ ts When ts Replace.

[0057] The above steps are explained for forward detection. The same steps also apply to backward detection. Forward detection uses the reference window to detect whether the test window has a jump, while backward detection uses the test window to detect whether the reference window has a jump (that is, the test window is equivalent to the forward reference window, the reference window is equivalent to the forward test window, and the two windows exchange corners). In order to reduce the chance of false alarms, data will only be identified as a jump point if the forward and backward detections give consistent alarms. For example, if the forward detection finds an upward jump and the backward detection finds a downward jump, then the test window data is regarded as the jump data.

[0058] A change point detection system based on large-scale network traffic type data includes a preprocessing module, a sequence classification module and a difference detection module.

[0059] The preprocessing module preprocesses the time series by filling missing values, normalizing data, and smoothing to improve the accuracy of sequence classification;

[0060] The sequence classification module is used to classify data into two different types of time series according to human behavior patterns, namely, a time series with a one-day cycle, a time series with a one-week cycle, and a non-periodic time series; discrete Fourier transform combined with autocorrelation coefficients is used to classify time series;

[0061] The difference detection module is used to detect the classified time series to determine whether there is a jump;

[0062] First, the original time series is converted into a binary sequence, and then the density of 1 in the binary sequences of the reference window and the test window is identified to detect the change point; density difference detection is divided into forward detection and backward detection. The forward detection uses the reference window to detect whether the test window has a jump, while the backward detection uses the test window to detect whether the reference window has a jump.

[0063] Preferably, the specific method of the preprocessing module is as follows:

[0064] Let the time series be x = {x1, x2, ..., x L}, where L is the length of the sequence, x t is the specific value of sequence x at a certain moment, and t∈{1, 2, …, L}, and let the reference window and test window be W respectively ref , W test , the window lengths are l1 and l2 respectively, then for the data x t For example, its reference window data can be expressed as The test window data can be represented as

[0065] Step 1.1, fill in missing values.

[0066] Linear interpolation method is used to fill the missing data in the reference window.

[0067] Step 1.2, data normalization.

[0068] Since there are many types of network traffic data and the ranges of different types of data vary greatly, the MIN-MAX method is used to normalize the data and limit the reference window data to the range of [0, 1] to facilitate data comparison and processing.

[0069] Step 1.3, smoothing.

[0070] In order to avoid the influence of noise and outliers on the classification of sequences, the exponentially weighted moving average algorithm is used to smooth the reference window data to improve the accuracy of sequence classification.

[0071] Preferably, the specific method of the sequence classification module is as follows:

[0072] According to human behavior patterns, the data is divided into two different types of time series, namely, time series with a one-day cycle, time series with a one-week cycle, and non-periodic time series. Discrete Fourier transform combined with autocorrelation coefficient is used to classify time series.

[0073] Step 2.1, perform discrete Fourier transform (DFT) on the reference window data.

[0074] For a given reference window W of length l1 ref , assuming that X is the discrete Fourier transform of the preprocessed time series, then the periodogram Determined by the square of each Fourier coefficient:

[0075]

[0076] in:

[0077]

[0078] Where k is the harmonic component order, f k is the magnitude of the kth harmonic component, j is an imaginary unit, and n∈{0, 1, …, l1-1}. Let the magnitudes of the period with one day and one week be T1 and T2 respectively, then the magnitudes of the corresponding frequencies in the frequency domain are 1 / T1 and 1 / T2 respectively. Select the frequency with the largest amplitude in the spectrum except the DC component. If the frequency is not 1 / T1 or 1 / T2, the sequence is determined to be a non-periodic sequence, otherwise continue to step 2.2.

[0079] Step 2.2, calculate the autocorrelation coefficient of the reference window data.

[0080] The calculation formula of the autocorrelation coefficient (ACF) with a lag phase of l is:

[0081]

[0082] Where i∈{0, 1, …, l1-1}, for a sequence with a period of T (T∈{T1, T2}), when the lag phase l is T, 2T, 3T, …, the time series will show strong correlation. According to this principle, ACF(m1T1), ACF(m2T2) and the predetermined autocorrelation coefficient threshold r are used to calculate the correlation coefficient. ts Compare, where m1 and m2 are positive integers, and when selecting, it should be ensured that m1T1 and m2T2 are not much different. If ACF(m1T1)>r ts or ACF(m2T2)>r ts , then the sequence is identified as a periodic sequence, otherwise, it is marked as a non-periodic sequence. If the sequence is identified as a periodic sequence, it is necessary to compare the size of ACF(m1T1) and ACF(m2T2) to determine the size of the period. The comparison formula is:

[0083]

[0084] Preferably, the specific method of the difference detection module is as follows:

[0085] For different types of time series, the steps of jump point detection are the same, but the parameters of different types of sequences are selected differently.

[0086] First, the original time series is converted into a binary sequence, and then the density of 1 in the binary sequence of the reference window and the test window is identified to detect the change point. Density difference detection is divided into forward detection and backward detection. Forward detection uses the reference window to detect whether the test window has a jump, while backward detection uses the test window to detect whether the reference window has a jump. Forward detection and backward detection have the same principle. The following is a detailed introduction to the principle of forward detection.

[0087] Step 3.1, convert the reference window and test window sequences into binary sequences.

[0088] Since the transition point may appear in the form of an upward or downward jump, two thresholds are used, namely the upper threshold u ts and the lower threshold l ts , converting the original time series into two different binary sequences to detect the upward and downward jump points. Here, the upper threshold u tsSet to:

[0089] u ts =Q3+k1*IQR (17)

[0090] Where IQR = Q3-Q1, Q3 and Q1 are the upper and lower quartiles of the reference window, respectively, and k1 is a hyperparameter. After obtaining the upper threshold uts, the original data of the reference window and the test window are converted into binary values. The conversion formula is:

[0091]

[0092] where y t is the binary value of the time series at time t. ts Set to:

[0093] l ts =Quantile(k2) (19)

[0094] Quantile(·) represents the quantile of the time series, k2 is a hyperparameter and needs to be set manually based on the data characteristics of the sequence. The corresponding conversion formula is:

[0095]

[0096] Since a time series with a cycle of one week usually has different data characteristics on weekdays and weekends, different upper thresholds u are set for weekday and weekend data respectively. ts and the lower threshold l ts For the other two types of series, the threshold settings for weekdays and weekends are the same.

[0097] Step 3.2, determine whether there is a difference in the density of 1s in the binary sequences of the reference window and the test window.

[0098] Once the binary sequences of the reference and test windows are obtained (based on the upper or lower threshold conversion), the reference sequence Y ref The density ρ of 1 is calculated as follows:

[0099]

[0100] Where sum(Y ref ) represents the label sequence Y ref Sum, assuming that the samples in the binary sequence are independently drawn from the same distribution, that is, follow the binomial distribution, then the probability that there are at least q 1 samples in the test window is calculated as follows:

[0101]

[0102] When the probability of f(q|ρ) is close to 0 (the left tail probability or right tail probability of the probability distribution), it means that observing q 1 samples in the test window is a rare event, indicating that the density of 1 samples has changed. The increase in the density of 1 samples means an upward change. At this time, the right tail probability is calculated by formula (22) to detect the upward jump point. Similarly, the decrease in the density of 1 samples means a downward change, and the left tail probability can be calculated to detect the downward jump point.

[0103] Since it is numerically difficult to directly calculate f(q|ρ), the normal distribution is used to approximate the binomial distribution according to the central limit theorem. The mean μ and standard deviation σ of the normal distribution are calculated using the following formula:

[0104] μ=l2*ρ (23)

[0105]

[0106] After obtaining the mean μ and standard deviation σ of the normal distribution, the N-sigma criterion is used to detect the upward or downward jump point. That is, for the application of the upper threshold u ts The binary sequence generated by detecting upward changes, the test threshold is set to c ts =μ+N1*σ, where N1 is a hyperparameter. test The number of samples with 1 in the above formula exceeds c ts , then an upward change is detected. For applying the lower threshold l ts The binary sequence generated by detecting downward changes, the test threshold is set to c ts =μ-N2*σ, where N2 is also a hyperparameter. test The number of samples in 1 is less than c ts , then a downward change is detected. It is worth noting that when ρ is very small, it means that Y ref The density of 1 is very small, so Y test When the density of 1 changes slightly, it is easy to misjudge it as jump data, so it is necessary to set a density threshold ρ for ρ ts , when ρ is less than ρ ts When , the density threshold ρts is used instead.

[0107] The principle of backward detection is the same as that of forward detection. Forward detection uses the reference window to detect whether the test window has a jump, while backward detection uses the test window to detect whether the reference window has a jump (that is, the test window is equivalent to the forward reference window, the reference window is equivalent to the forward test window, and the two windows exchange corners). In order to reduce the chance of false alarms, data will only be identified as a jump point if the forward and backward detections give consistent alarms. For example, if the forward detection finds an upward jump and the backward detection finds a downward jump, then the test window data is regarded as the jump data.

[0108] The beneficial effects of the present invention are as follows:

[0109] The present invention proposes a jump point detection method for detecting the jump points of large-scale network traffic type data. The method converts the time series into a binary sequence, and then identifies the jump point by detecting the density change of sample 1 in the converted binary sequence. The method has low complexity and does not require the assumption of data distribution in advance, and has good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 is an algorithm flow chart of an embodiment of the present invention;

[0111] Figure 2 The spectrum diagram and correlation coefficient diagram of the sequence of the embodiment of the present invention;

[0112] Figure 3 This is a short-time jump detection diagram of an embodiment of the present invention;

[0113] Figure 4 This is a long-time jump detection diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0114] The method disclosed in the present invention first preprocesses and classifies the time series, then uses a simple and efficient threshold determination method to quickly obtain the binary sequences of the reference window and the test window, and finally detects the jump point by judging whether there is a difference in the density of 1 samples in the binary sequence. This method has achieved good results in terms of efficiency and accuracy, and is suitable for jump point detection of large-scale network traffic type data.

[0115] This algorithm uses base station data collected by a foreign communications department as the data set of the embodiment. The data set records the system resource (counter) data and key performance indicator (KPI) data of base stations in different regions during a certain period of time. The monitoring system collects data every hour from January 29, 2018 to October 21, 2018. Through visual analysis of the data set, it can be found that some jumps last for a short time (only one day), while some jumps last for a long time (more than one week). In order to identify these changes, we apply the proposed detection method on two time scales. In the fast time scale, the forward test window length is set to 1 day (l2=24), while in the slow time scale, the test window is set to 1 week (l2=168). In both time scales, the forward reference window length is set to 6 weeks (l1=1008).

[0116] The method proposed by the present invention is further described below in conjunction with the accompanying drawings and embodiments:

[0117] Figure 1 The algorithm flow chart of the present invention is shown in FIG. 1. The algorithm first preprocesses the data of the reference window, and the preprocessing includes two steps: filling missing values ​​in the sequence, normalizing the data, and smoothing. Then, the preprocessed time series is classified. Here, the discrete Fourier transform combined with the autocorrelation coefficient method is mainly used to determine whether the time series is a periodic sequence and the period size of the periodic sequence. Finally, the density change of the binary sequence is used to determine whether the data has a jump.

[0118] Step 1: Preprocessing of reference window data.

[0119] Specifically, execute steps 1.1 to 1.3, first use the linear interpolation method to calculate W ref The missing values ​​are filled, and then the Min-Max normalization method is used to unify the data dimension, and finally the exponentially weighted moving average algorithm is used to smooth the sequence. In this embodiment, the weighted reduction coefficient of the moving weighted average algorithm is set to 0.25.

[0120] Step 2: Classify the reference window sequence.

[0121] In this embodiment, the base station data is collected every hour, so the period lengths of one day and one week are T1=24 and T2=168 respectively. Specifically, execute steps 2.1 to 2.2, first perform discrete Fourier transform on the reference window data, and when the frequency spectrum is excluding the DC component, or When the amplitude is not the maximum value, the sequence is judged as a non-periodic sequence. Otherwise, continue to observe the autocorrelation coefficient diagram of the sequence. Here, the autocorrelation coefficient threshold r tsSet to 0.3, set m1 and m2 to 8 and 1 respectively, then m1T1 = 192, m2T2 = 168, if one or more of ACF(168) and ACF(192) is greater than the threshold r ts , then the sequence is determined to be a periodic sequence, and the sizes of ACF(168) and ACF(192) are compared. If ACF(168) is greater than ACF(192), the sequence is determined to be a sequence with a period of one week, otherwise the sequence is determined to be a sequence with a period of one day. Figure 2 The spectrogram and autocorrelation coefficient plot of the reference window sequence are shown.

[0122] Step 3: Detect density differences in binary time series.

[0123] Specifically, execute step 3.1 to determine W ref Window and W test The upper and lower thresholds of the window are used to quickly generate binary label sequences for the reference window and the test window. Since a time series with a cycle of one week usually has different data characteristics on weekdays and weekends, we set different upper thresholds u for weekday and weekend data respectively. ts and the lower threshold l ts For the other two types of sequences, the threshold settings for weekdays and weekends are the same. Then execute step 3.2 to determine whether the data of the original time series has a jump by using the density difference between the reference window and the test window binary sequence. Table 1-3 gives the parameter values ​​of two different types of time series, and Figure 3 , Figure 4 The following diagrams show the detection examples of short-time jumps and long-time jumps respectively.

[0124] Table 1: Time series with a period of one day

[0125]

[0126] Table 2: Non-periodic time series

[0127]

[0128] Table 3: Time series with a weekly period

[0129]

[0130]

[0131] In this example, we selected 4 names counter data and use the accuracy and recall The algorithm effect is evaluated, where TP is true positive, FP is false positive, and FN is false negative. The following table shows the statistics of the four counter data. As can be seen from the table, the overall accuracy can reach 91.4%, and the overall recall rate can reach 94.2%.

[0132] Table 4: Transition point detection statistics

[0133]

[0134] In summary, the present invention proposes a non-parametric jump point detection method suitable for processing large-scale network traffic type data. The core idea of ​​the method is to convert the original time series into a binary sequence (a time series composed of 0 and 1), and identify the jump point by finding the moment when the density of 1 (or 0) in the binary sequence changes. The method does not need to assume the distribution of data in advance, and the complexity of the algorithm is low, and good results are obtained in the embodiment data set.

Claims

1. A method for detecting a change point based on large-scale network traffic type data, characterized in that: The following steps are involved: Step 1: Preprocessing of reference window data; Preprocessing includes three steps: filling missing values ​​in time series, data normalization, and smoothing. The processed time series helps improve the accuracy of sequence classification; Step 2: Classification of reference window sequences; According to human behavior patterns, the data is divided into three different types of time series: time series with a one-day cycle, time series with a one-week cycle, and non-periodic time series; discrete Fourier transform combined with autocorrelation coefficient is used to classify time series; Step 3: Detection of density differences in binary time series; For different types of time series, the steps of jump point detection are the same, but the parameters of different types of sequences are selected differently. This step first converts the original time series into a binary sequence, and then identifies whether there is a difference in the density of 1 in the binary sequence of the reference window and the test window, so as to detect the change point. The density difference detection is divided into forward detection and backward detection. The forward detection uses the reference window to detect whether the test window has a jump, while the backward detection uses the test window to detect whether the reference window has a jump. Step 3 is as follows: After step 2, the type of time series is determined. For different types of time series, the steps of jump point detection are the same, but the parameters of different types of sequences are selected differently. First, the original time series is converted into a binary sequence, and then the density of 1 in the binary sequence of the reference window and the test window is identified to detect the change point. The density difference detection is divided into forward detection and backward detection. The forward detection uses the reference window to detect whether the test window has a jump, while the backward detection uses the test window to detect whether the reference window has a jump. Forward detection and backward detection have the same principle. The following is a detailed introduction to the principle of forward detection. Step 3.1, converting the reference window and test window sequences into binary sequences; Since the transition point may appear in the form of an upward or downward jump, two thresholds are used, namely the upper threshold u ts and the lower threshold l ts , convert the original time series into two different binary sequences to detect the upward and downward jump points; here the upper threshold u ts Set to: in ts =Q3+k1*IQR (5) Where IQR = Q3-Q1, Q3 and Q1 are the upper and lower quartiles of the reference window, respectively, and k1 is a hyperparameter; the upper threshold u is obtained. ts Finally, the original data of the reference window and the test window are converted into binary values. The conversion formula is: where y t is the binary value of the time series at time t; at the same time, the lower threshold l ts Set to: l ts =Quantile(k2) (7) Quantile(·) represents the quantile of the time series, k2 is a hyperparameter and needs to be set manually based on the data characteristics of the sequence. The corresponding conversion formula is: Since a time series with a cycle of one week usually has different data characteristics on weekdays and weekends, different upper thresholds u are set for weekday and weekend data respectively. ts and the lower threshold l ts ;For the other two types of sequences, the threshold settings for weekdays and weekends are the same; Step 3.2, determine whether there is a difference in the density of 1s in the binary sequences of the reference window and the test window; Once the binary sequences of the reference and test windows are obtained, the reference sequence Y ref The density ρ of 1 is calculated as follows: Where sum(Y ref ) represents the label sequence Y ref Sum, assuming that the samples in the binary sequence are independently drawn from the same distribution, that is, follow the binomial distribution, then the probability that there are at least q 1 samples in the test window is calculated as follows: When the probability of f(q|ρ) is close to 0, it means that observing q 1 samples in the test window is a rare event, indicating that the density of 1 samples has changed; the increase in the density of 1 samples means an upward change, and the right tail probability is calculated by formula (10) to detect the upward jump point; similarly, the decrease in the density of 1 samples means a downward change, and the left tail probability is calculated to detect the downward jump point; Since it is numerically difficult to directly calculate f(q|ρ), the normal distribution is used to approximate the binomial distribution according to the central limit theorem. The mean μ and standard deviation σ of the normal distribution are calculated using the following formula: μ=l2*ρ (11) After obtaining the mean μ and standard deviation σ of the normal distribution, the N-sigma criterion is used to detect the upward or downward jump point; that is, for the application of the upper threshold u ts The binary sequence generated by detecting upward changes, the test threshold is set to c ts =μ+N1*σ, where N1 is a hyperparameter; if Y test The number of samples with 1 in the above formula exceeds c ts , then an upward change is detected; for the application of the lower threshold l ts The binary sequence generated by detecting downward changes, the test threshold is set to c ts =μ-N2*σ, where N2 is also a hyperparameter; if Y test The number of samples in 1 is less than c ts , then a downward change is detected; it is worth noting that when ρ is very small, it means that Y ref The density of 1 is very small, so Y test When the density of 1 changes slightly, it is easy to misjudge it as jump data, so it is necessary to set a density threshold ρ for ρ ts , when ρ is less than ρ ts When ts to make a substitution; The principle of backward detection is the same as that of forward detection. Forward detection uses the reference window to detect whether the test window has a jump, while backward detection uses the test window to detect whether the reference window has a jump. To reduce the chance of false alarms, data is identified as a transition point only if both forward and backward detection give consistent alarms.

2. The method for detecting a change point based on large-scale network traffic type data according to claim 1, characterized in that: Step 1 is as follows: Let the time series be x = {x1, x2, ..., x L }, where L is the length of the sequence, x t is the specific value of sequence x at a certain moment, and t∈{1, 2, …, L}, and let the reference window and test window be W respectively ref , W test , the window lengths are l1 and l2 respectively, then for the data x t For example, its reference window data can be expressed as The test window data can be represented as Step 1.1, fill in missing values; Linear interpolation method was used to fill the missing data of the reference window; Step 1.2, data normalization; Since there are many types of network traffic data and the ranges of different types of data vary greatly, the MIN-MAX method is used to normalize the data and limit the reference window data to the range of [0,1] to facilitate data comparison and processing; Step 1.3, smoothing; In order to avoid the influence of noise and outliers on the classification of sequences, the exponentially weighted moving average algorithm is used to smooth the reference window data to improve the accuracy of sequence classification.

3. The method for detecting a change point based on large-scale network traffic type data according to claim 2, characterized in that: Step 2 is as follows: According to human behavior patterns, the data is divided into three different types of time series: time series with a one-day cycle, time series with a one-week cycle, and non-periodic time series; discrete Fourier transform combined with autocorrelation coefficient is used to classify time series; Step 2.1, performing discrete Fourier transform on the reference window data; For a given reference window W of length l1 ref , assuming that X is the discrete Fourier transform of the preprocessed time series, then the periodogram Determined by the square of each Fourier coefficient: in: Where k is the harmonic component order, f k is the size of the kth harmonic component, j is an imaginary unit, n∈{0, 1, …, l1-1}; let the period sizes of one day and one week be T1 and T2 respectively, then the corresponding frequency sizes in the frequency domain are 1 / T1 and 1 / T2 respectively, select the frequency with the largest amplitude in the spectrum except the DC component, if the frequency is not 1 / T1 or 1 / T2, then determine the sequence as a non-periodic sequence, otherwise continue to step 2.2; Step 2.2, calculate the autocorrelation coefficient of the reference window data; The calculation formula of the autocorrelation coefficient Auto-Correlation Function with a lag phase of l, ACF is: Where i∈{0, 1, …, l1-1}, for a sequence T∈{T1,T2} with a period of T, when the lag phase l is T, 2T, 3T, …, the time series will show strong correlation; according to this principle, ACF(m1T1), ACF(m2T2) and the predetermined autocorrelation coefficient threshold r are ts Compare, where m1 and m2 are positive integers, and when selecting, they should ensure that m1T1 and m2T2 are not much different; if ACF(m1T1)>r ts or ACF(m2T2)>r ts , then the sequence is identified as a periodic sequence, otherwise, it is marked as a non-periodic sequence; if the sequence is identified as a periodic sequence, it is necessary to compare the sizes of ACF(m1T1) and ACF(m2T2) to determine the size of the period, and the comparison formula is:

4. The system constructed by the method for detecting the jumping point based on large-scale network traffic type data according to claim 1 is characterized in that: It includes a preprocessing module, a sequence classification module and a difference detection module; The preprocessing module preprocesses the time series by filling missing values, normalizing data, and smoothing to improve the accuracy of sequence classification; The sequence classification module is used to classify data into three different types of time series according to human behavior patterns, namely, a time series with a one-day cycle, a time series with a one-week cycle, and a non-periodic time series; discrete Fourier transform combined with autocorrelation coefficients is used to classify the time series; The difference detection module is used to detect the classified time series to determine whether there is a jump; First, the original time series is converted into a binary sequence, and then the density of 1 in the binary sequences of the reference window and the test window is identified to detect the change point; density difference detection is divided into forward detection and backward detection. The forward detection uses the reference window to detect whether the test window has a jump, while the backward detection uses the test window to detect whether the reference window has a jump.

Images