Network traffic anomaly detection method based on big data analysis

Abstract

The present invention relates to the field of data processing, and in particular to a network traffic anomaly detection method based on big data analysis. The method seasonally decomposes a network traffic data sequence to obtain a seasonal component of each data; when trending the network traffic data sequence, a short-term local window and a long-term local window of the first data are obtained to obtain corresponding trend components and a degree of data change trend; a short-term local window of the next data is obtained according to the degree of data change trend, the next data is taken as the first data, and the method for obtaining the trend component and the degree of data change trend corresponding to the first data is repeated until the last data in the network traffic data sequence is traversed to obtain the trend component of each data; a residual component of each data is obtained according to the seasonal component and the trend component, and then abnormal data is screened in the network traffic data sequence, thereby improving the accuracy of detecting abnormal network traffic data.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a network traffic anomaly detection method based on big data analysis. Background Art

[0002] With the increasing complexity of network environment and the continuous growth of network traffic data, traditional network traffic anomaly detection methods are gradually unable to cope with existing challenges. The application of big data technology makes it possible to process and analyze massive network traffic data, improving the efficiency and accuracy of anomaly detection. At the same time, the real-time demand for network traffic anomaly detection methods is increasing.

[0003] In the prior art, the traditional STL algorithm is used to perform anomaly detection on network traffic data. According to a fixed-size window, local weighted regression is used to perform trend decomposition and seasonal decomposition on the network traffic data sequence, and the trend component and seasonal component of each data in the network traffic data sequence are obtained. The residual component of each data is obtained based on the trend component and the seasonal component, and then anomaly detection is performed on the data in the network traffic data sequence. However, when using a fixed-size window for STL decomposition, the efficiency and accuracy of detecting abnormal data are limited due to the real-time, large fluctuation, and instability characteristics of the network traffic data sequence.

[0004] Therefore, how to improve the accuracy of anomaly detection in network traffic data has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a network traffic anomaly detection method based on big data analysis to solve the problem of how to improve the accuracy of anomaly detection of network traffic data.

[0006] An embodiment of the present invention provides a method for detecting network traffic anomalies based on big data analysis, the method comprising the following steps:

[0007] Acquire a network traffic data sequence within a preset period, and perform seasonal decomposition on the network traffic data sequence according to a sliding window of a preset length to obtain a seasonal component of each data in the network traffic data sequence;

[0008] When trend decomposing the network traffic data sequence, a short-term local window and a long-term local window of the first data in the network traffic data sequence are obtained according to the sliding window, and a trend component and a degree of data change trend corresponding to the short-term local window of the first data are obtained according to the data difference between the short-term local window and the long-term local window of the first data;

[0009] According to the data change trend degree corresponding to the short-term local window of the first data, the short-term local window of the next data after the first data is obtained, the next data is used as the first data, and the method of obtaining the trend component and the data change trend degree corresponding to the short-term local window of the first data is repeated until the last data in the network traffic data sequence is traversed to obtain the trend component of each data in the network traffic data sequence;

[0010] According to the seasonal component and the trend component of each data in the network traffic data sequence, the residual component of each data in the network traffic data sequence is obtained respectively, and according to the residual component of each data in the network traffic data sequence, the abnormal data is screened in the network traffic data sequence.

[0011] Preferably, the step of obtaining the short-term local window and the long-term local window of the first data in the network traffic data sequence according to the sliding window includes:

[0012] According to the sliding window, an initial short-term local window of the first data is obtained in the network traffic data sequence, wherein the length of the initial short-term local window is equal to the length of the sliding window;

[0013] According to the data difference in the initial short-term local window of the first data, an initial trend component corresponding to the initial short-term local window of the first data is obtained;

[0014] According to the initial short-term local window of the first data and the preset sliding step length, a sliding traversal is performed on the network traffic data sequence to obtain a preset number of target short-term local windows, and the initial short-term local window and the preset number of target short-term local windows are combined to form the initial long-term local window of the first data;

[0015] Obtaining the degree of initial data change trend corresponding to the initial short-term local window of the first data according to the data difference in the initial long-term local window of the first data and the initial trend component;

[0016] According to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data, the short-term local window and the long-term local window of the first data are obtained.

[0017] Preferably, obtaining the initial trend component corresponding to the initial short-term local window of the first data according to the data difference in the initial short-term local window of the first data comprises:

[0018] Obtaining a volatility coefficient corresponding to the initial short-term local window of the first data according to a standard deviation corresponding to the initial short-term local window of the first data;

[0019] Respectively calculating the first-order difference value of every two adjacent data in the initial short-term local window of the first data to obtain an average first-order difference value, and obtaining the gradient change coefficient corresponding to the initial short-term local window of the first data according to the average first-order difference value;

[0020] The fluctuation coefficient and the gradient change coefficient are weightedly summed to obtain an initial trend component corresponding to an initial short-term local window of the first data.

[0021] Preferably, obtaining the initial data change trend degree corresponding to the initial short-term local window of the first data according to the data difference in the initial long-term local window of the first data and the initial trend component includes:

[0022] Respectively obtain the initial trend components corresponding to the preset number of target short-term local windows, form a set with the initial trend components corresponding to the initial short-term local window, perform weighted summation on the initial trend components in the set, and obtain the local long-term trend component corresponding to the initial long-term local window of the first data;

[0023] Obtain the maximum data and the minimum data in the initial short-term local window of the first data, calculate the mean of the initial trend component and the local long-term trend component, obtain a first difference between the mean and the minimum data, obtain a second difference between the maximum data and the minimum data, and use the ratio of the first difference to the second difference as the degree of initial data change trend corresponding to the initial short-term local window of the first data.

[0024] Preferably, obtaining the short-term local window and the long-term local window of the first data according to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data comprises:

[0025] If the initial trend component is greater than the preset trend component threshold, or the initial data change trend degree is greater than the preset first data change trend degree threshold, the length of the initial short-term local window of the first data is doubled to obtain a new short-term local window, and the new short-term local window is used as the initial short-term local window. The method of obtaining the initial trend component and the initial data change trend degree corresponding to the initial short-term local window of the first data is repeated until the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, so as to obtain the short-term local window and long-term local window of the first data.

[0026] Preferably, the step of obtaining the short-term local window and the long-term local window of the first data according to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data further includes:

[0027] If the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, then the initial short-term local window and initial long-term local window of the first data are used as the short-term local window and long-term local window of the first data.

[0028] Preferably, the step of obtaining the short-term local window of next data after the first data according to the degree of data change trend corresponding to the short-term local window of the first data includes:

[0029] If the data change trend degree is greater than or equal to a preset second data change trend degree threshold, then calculating a first product of the data change trend degree and a preset first adjustment weight, obtaining the sum of the first product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data;

[0030] According to the initial trend component corresponding to the initial short-term local window of the next data and the degree of change trend of the initial data, the short-term local window of the next data is obtained.

[0031] Preferably, the step of obtaining the short-term local window of next data after the first data according to the data change trend degree corresponding to the short-term local window of the first data further includes:

[0032] If the data change trend degree is less than a preset second data change trend degree threshold, then calculating a second product of the data change trend degree and a preset second adjustment weight, obtaining the sum of the second product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data;

[0033] According to the initial trend component corresponding to the initial short-term local window of the next data and the degree of change trend of the initial data, the short-term local window of the next data is obtained.

[0034] Preferably, the step of screening the network traffic data sequence to obtain abnormal data according to the residual component of each data in the network traffic data sequence comprises:

[0035] For any data in the network traffic data sequence, the absolute value of the residual component of the data is obtained, and if the absolute value of the residual component is greater than a preset residual threshold, the data is marked as abnormal data.

[0036] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0037] The present invention obtains a network traffic data sequence. Since the network traffic data has significant periodicity, for example, the daily traffic starts to be low in the early morning, gradually reaches a peak during the day, and then decreases again at night. This pattern will appear repeatedly in each cycle. Therefore, when performing seasonal decomposition on the network traffic data sequence, a fixed-size sliding window is still used, and local weighted regression is used to obtain the seasonal component of each data in the network traffic data sequence. However, since the network traffic data has the characteristics of real-time, large fluctuation, and instability, for data with a large change trend, due to the window setting being too large, some rapidly changing data features cannot be captured in time, so that the data with a large change amplitude is smoothed out. For data with a stable change trend, due to the window setting being too small, the calculation overhead increases and the efficiency of trend decomposition of the network traffic data sequence decreases. Therefore, in an embodiment of the present invention, when performing trend decomposition on the network traffic data sequence, first, according to a fixed-size sliding window, the first data in the network traffic data sequence is obtained. According to the short-term local window and long-term local window of the network traffic data sequence, the trend component and the degree of data change trend corresponding to the short-term local window of the first data are obtained according to the data difference between the short-term local window and the long-term local window of the first data. Then, according to the degree of data change trend corresponding to the short-term local window of the first data, the short-term local window of the next data after the first data is obtained. The next data is taken as the first data, and the method of obtaining the degree of data change trend and the trend component corresponding to the short-term local window of the first data is repeated until the last data in the network traffic data sequence is traversed to obtain the trend component of each data in the network traffic data sequence. Finally, according to the seasonal component and the trend component of each data in the network traffic data sequence, the residual component of each data in the network traffic data sequence is obtained respectively. According to the residual component of each data in the network traffic data sequence, the abnormal data is screened in the network traffic data sequence, thereby improving the accuracy of detecting abnormal network traffic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 This is a method flow chart of a network traffic anomaly detection method based on big data analysis provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0040] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0041] It should be noted that the terms "first", "second", etc. in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0042] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0043] See also Figure 1 , is a method flow chart of a network traffic anomaly detection method based on big data analysis provided by the first embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0044] Step S101, obtaining a network traffic data sequence within a preset period, performing seasonal decomposition on the network traffic data sequence according to a sliding window of a preset length, and obtaining a seasonal component of each data in the network traffic data sequence.

[0045] First, a network flow data sequence within a preset period is obtained. The specific method is as follows: the period set in the embodiment of the present invention is 1 day. There is no limit on the preset period, which can be set according to the implementation scenario. Through network monitoring tools (such as Wireshark, NetFlow, etc.) or network device logs, the network flow data within one day is collected in real time. After that, the collected network flow data is preprocessed. The specific data preprocessing method is as follows:

[0046] a. Through the data cleaning process, invalid data in the network traffic data is removed, missing values ​​are filled, and the data format is standardized. For example, duplicate network traffic records are eliminated, outliers are corrected, and the timestamp format is unified, so as to ensure the accuracy and reliability of subsequent network traffic data analysis.

[0047] b. Normalize different features in network traffic data to eliminate the impact between different dimensions.

[0048] At this point, according to the above data preprocessing steps, a network traffic data sequence within a day is obtained.

[0049] In the process of performing anomaly detection on the data in the acquired network traffic data sequence, accurately distinguishing the trend component and seasonal component in the network traffic data sequence to obtain accurate residual components is the key to achieving accurate abnormal data detection. The traditional STL algorithm uses a fixed-size sliding window and local weighted regression to perform trend decomposition and seasonal decomposition on the network traffic data sequence, obtain the trend component and seasonal component of each data in the network traffic data sequence, and then obtain the residual component of each data based on the trend component and seasonal component, and then perform anomaly detection on the data in the network traffic data sequence.

[0050] Among them, when performing seasonal decomposition on the network traffic data sequence, since the network traffic data has significant periodicity, for example, the daily traffic starts to be low in the early morning, gradually reaches a peak during the day, and drops again at night. This pattern will repeat in each cycle. Therefore, in the embodiment of the present invention, a sliding window with a preset fixed length is still used, and local weighted regression is used to obtain the seasonal component of each data in the network traffic data sequence. Among them, seasonal decomposition belongs to the existing technology and will not be described in detail here.

[0051] It should be noted that the length of the sliding window set in the embodiment of the present invention is an empirical value, that is, the sliding window contains 15 data points. There is no restriction on the preset length of the sliding window, which can be set according to the specific implementation scenario.

[0052] Step S102, when performing trend decomposition on the network traffic data sequence, obtain the short-term local window and the long-term local window of the first data in the network traffic data sequence according to the sliding window, and obtain the trend component and the degree of data change trend corresponding to the short-term local window of the first data according to the data difference between the short-term local window and the long-term local window of the first data.

[0053] When the traditional STL algorithm uses a fixed-size sliding window and local weighted regression to perform trend decomposition on the network traffic data series, due to the real-time nature of network traffic data, large data change trend, instability and other characteristics, the use of a fixed-size sliding window cannot accurately capture the trend direction and change speed of local data. That is, for data with a large change trend, the sliding window is set too large, which makes it impossible to capture some rapidly changing data features in time, causing data with large changes to be smoothed out. For data with a stable change trend, the sliding window is set too small, which increases the computational overhead and reduces the efficiency of decomposing the network traffic data series.

[0054] In order to solve the above problems, in an embodiment of the present invention, when performing trend decomposition on a network traffic data sequence, an adaptive sliding window of each data in the network traffic data sequence is obtained based on the sliding window of the above preset length, that is, a short-term local window of each data, and then the trend component of each data in the network traffic data sequence is obtained according to the short-term local window of each data, thereby optimizing the traditional trend decomposition.

[0055] Among them, the method for obtaining an adaptive sliding window for each data in a network traffic data sequence includes: first, according to the sliding window of the preset length, obtaining the short-term local window and the long-term local window of the first data in the network traffic data sequence; then, according to the data difference between the short-term local window and the long-term local window of the first data, respectively obtaining the trend component and the degree of data change trend corresponding to the short-term local window of the first data; finally, according to the degree of data change trend corresponding to the short-term local window of the first data, obtaining the short-term local window of the next data after the first data.

[0056] The short-term local window and the long-term local window of the first data in the network traffic data sequence are obtained, including:

[0057] (1) According to the sliding window, an initial short-term local window of the first data is obtained in the network traffic data sequence, wherein the length of the initial short-term local window is equal to the length of the sliding window.

[0058] According to the sliding window obtained in step S101, the first data in the network traffic data sequence is used as the first data in the sliding window to obtain the initial short-term local window of the first data in the network traffic data sequence, wherein the length of the initial short-term local window of the first data is equal to the length of the sliding window, that is, the initial short-term local window of the first data contains 15 data points.

[0059] (2) According to the data difference in the initial short-term local window of the first data, an initial trend component corresponding to the initial short-term local window of the first data is obtained.

[0060] Specifically, according to the standard deviation corresponding to the initial short-term local window of the first data, the volatility coefficient corresponding to the initial short-term local window of the first data is obtained;

[0061] Respectively calculating the first-order difference value of every two adjacent data in the initial short-term local window of the first data to obtain an average first-order difference value, and obtaining the gradient change coefficient corresponding to the initial short-term local window of the first data according to the average first-order difference value;

[0062] The fluctuation coefficient and the gradient change coefficient are weightedly summed to obtain an initial trend component corresponding to an initial short-term local window of the first data.

[0063] In one implementation, the standard deviation corresponding to the initial short-term local window of the first data is calculated, and the first-order difference values ​​of each two adjacent data in the initial short-term local window of the first data are respectively obtained to obtain the corresponding average first-order difference value. According to the standard deviation and the average first-order difference value corresponding to the initial short-term local window of the first data, the initial trend component corresponding to the initial short-term local window of the first data is obtained. Then, the expression of the initial trend component corresponding to the initial short-term local window of the first data is:

[0064]

[0065] Among them, H represents the initial trend component corresponding to the initial short-term local window of the first data, S represents the volatility coefficient corresponding to the initial short-term local window of the first data, represents the first-order difference between the i-th data and the i+1-th data in the initial short-term local window of the first data, Z represents the length of the initial short-term local window of the first data, 1 represents a constant, and || represents an absolute value. represents the weight of the volatility coefficient, Represents the weight of the gradient change coefficient.

[0066] It should be noted that, if the value of S is larger, it means that the fluctuation degree of the data in the initial short-term local window of the first data is larger, indicating that the local change trend of the data in the initial short-term local window of the first data is more obvious, and the initial trend component of the initial short-term local window of the first data is larger; The larger the value is, the greater the degree of gradient change between two adjacent data in the initial short-term local window of the first data is, indicating that the local change trend of the data in the initial short-term local window of the first data is more obvious, and the corresponding initial trend component of the initial short-term local window of the first data is larger.

[0067] Preferably, the weight of the fluctuation coefficient and the weight of the gradient change coefficient set in the embodiment of the present invention are empirical values, that is, There is no restriction on this, and it can be set according to the implementation scenario.

[0068] (3) According to the initial short-term local window of the first data and a preset sliding step, a sliding traversal is performed on the network traffic data sequence to obtain a preset number of target short-term local windows, and the initial short-term local window and the preset number of target short-term local windows are combined to form the initial long-term local window of the first data.

[0069] In one implementation, the initial short-term local window of the first data and a preset sliding step are used to perform a sliding traversal on the network traffic data sequence to obtain three target short-term local windows. The three target short-term local windows and the initial short-term local window are combined to form the initial long-term local window of the first data.

[0070] It should be noted that the sliding step size and the number of target short-term local windows set in the embodiment of the present invention are empirical values, that is, the sliding step size is 1 data point, which is not limited and can be set according to the implementation scenario.

[0071] (4) Obtaining the degree of initial data change trend corresponding to the initial short-term local window of the first data based on the data difference in the initial long-term local window of the first data and the initial trend component.

[0072] Specifically, respectively obtain the initial trend components corresponding to the preset number of target short-term local windows, form a set with the initial trend components corresponding to the initial short-term local window, perform weighted summation on the initial trend components in the set, and obtain the local long-term trend component corresponding to the initial long-term local window of the first data;

[0073] Obtain the maximum data and the minimum data in the initial short-term local window of the first data, calculate the mean of the initial trend component and the local long-term trend component, obtain a first difference between the mean and the minimum data, obtain a second difference between the maximum data and the minimum data, and use the ratio of the first difference to the second difference as the degree of initial data change trend corresponding to the initial short-term local window of the first data.

[0074] In one implementation, since the degree of data change trends in short-term local windows of different data may be different, the short-term local window of the next data cannot be accurately adjusted based on the degree of data change trends in the short-term local window of the first data. Therefore, by obtaining the data change trend in the initial long-term local window of the first data, the trend direction and change speed of the local data after the short-term local window of the first data are captured, and then based on the degree of data change trends in the short-term local window and the long-term local window of the first data, the short-term local window of the next data is more accurately obtained.

[0075] According to the method for obtaining the initial trend component corresponding to the initial short-term local window of the first data, the initial trend components corresponding to the three target short-term local windows are obtained respectively, and a set is formed with the initial trend component corresponding to the initial short-term local window of the first data. According to the initial trend component in the set, the local long-term trend component corresponding to the initial long-term local window of the first data is obtained. Then, the expression of the local long-term trend component corresponding to the initial long-term local window of the first data is:

[0076]

[0077] Among them, L represents the local long-term trend component corresponding to the initial long-term local window of the first data, and H represents the initial trend component corresponding to the initial short-term local window of the first data. Represents the initial trend component corresponding to the first target short-term local window, Represents the initial trend component corresponding to the second target short-term local window, Represents the initial trend component corresponding to the third target short-term local window, represents the first weight, represents the second weight, represents the third weight, Represents the fourth weight.

[0078] It should be noted that The larger the value of , the greater the fluctuation degree and gradient change degree of the data in the initial short-term local window of the first data and the three target short-term local windows, and the greater the fluctuation degree and gradient change degree of the data in the initial long-term local window of the first data, that is, the more obvious the local change trend in the initial long-term local window of the first data, the greater the local long-term trend component corresponding to the initial long-term local window of the first data.

[0079] Preferably, the first weight, the second weight, the third weight and the fourth weight set in the embodiment of the present invention are There is no restriction on this, and it can be set according to the implementation scenario.

[0080] Get the maximum data and minimum data in the initial short-term local window of the first data. According to the initial trend component, local long-term trend component, maximum data and minimum data corresponding to the initial short-term local window of the first data, get the initial data change trend degree corresponding to the initial short-term local window of the first data. Then the expression of the initial data change trend degree corresponding to the initial short-term local window of the first data is:

[0081]

[0082] Among them, T represents the degree of change trend of the initial data corresponding to the initial short-term local window of the first data, H represents the initial trend component corresponding to the initial short-term local window of the first data, and L represents the local long-term trend component corresponding to the initial long-term local window of the first data. represents the minimum data in the initial short-term local window of the first data, Represents the maximum data in the initial short-term local window of the first data.

[0083] It should be noted that the larger the value of H is, the greater the degree of fluctuation and gradient change of the data in the initial short-term local window of the first data is, that is, the more obvious the local change trend is, and the greater the degree of change trend of the initial data corresponding to the initial short-term local window of the first data is; the smaller the value of L is, the more stable the local long-term change trend of the data in the initial long-term local window of the first data is, and the smaller the degree of change trend of the initial data corresponding to the initial short-term local window of the first data is.

[0084] (5) Obtaining a short-term local window and a long-term local window of the first data according to an initial trend component corresponding to the initial short-term local window of the first data and a degree of change trend of the initial data.

[0085] Specifically, if the initial trend component is greater than the preset trend component threshold, or the initial data change trend degree is greater than the preset first data change trend degree threshold, the length of the initial short-term local window of the first data is doubled to obtain a new short-term local window, and the new short-term local window is used as the initial short-term local window. The method of obtaining the initial trend component and the initial data change trend degree corresponding to the initial short-term local window of the first data is repeated until the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, so as to obtain the short-term local window and long-term local window of the first data.

[0086] If the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, then the initial short-term local window and initial long-term local window of the first data are used as the short-term local window and long-term local window of the first data.

[0087] In one embodiment, when the degree of change trend of the data in the initial short-term local window is too large, it indicates that the possibility of more local abnormal data existing in the initial short-term local window is greater. Therefore, the initial short-term local window needs to be adaptively adjusted to balance the impact of more abnormal data on the degree of data change trend. When the degree of change trend of the data in the initial short-term local window of the first data is moderate, it indicates that the possibility of more local abnormal data existing in the initial short-term local window is smaller, indicating that more data show a trend of continuous growth or decline, that is, the local change trend is more obvious.

[0088] The trend component threshold and the first data change trend degree threshold set in the embodiment of the present invention are 0.8 and 0.8 respectively. There is no restriction on the preset trend component threshold and the first data change trend degree threshold, and they can be set according to the implementation scenario. If the initial trend component corresponding to the initial short-term local window of the first data is greater than the trend component threshold, or the initial data change trend degree corresponding to the initial short-term local window of the first data is greater than the first data change trend degree threshold, that is, H>0.8 or T>0.8, then the length of the initial short-term local window of the first data is doubled to obtain a new short-term local window, and the initial short-term local window of the first data is obtained according to the initial short-term local window of the first data. The initial trend component and initial data change trend degree corresponding to the new short-term local window are obtained according to the method for obtaining the initial trend component and initial data change trend degree corresponding to the initial short-term local window, and the initial trend component and initial data change trend degree corresponding to the new short-term local window are obtained, and they are compared with the corresponding set thresholds respectively. If H>0.8 or T>0.8 is still satisfied, the length of the new short-term local window is doubled to obtain a second new short-term local window. According to the method for obtaining the initial trend component and initial data change trend degree corresponding to the initial short-term local window of the first data, the initial trend component and initial data change trend degree corresponding to the second new short-term local window are obtained, and they are compared with the corresponding set thresholds respectively, and so on, until and , the corresponding new short-term local window and new long-term local window are used as the short-term local window and long-term local window of the first data.

[0089] If the initial trend component corresponding to the initial short-term local window of the first data is less than or equal to the trend component threshold, and the initial data change trend degree corresponding to the initial short-term local window of the first data is less than or equal to the first data change trend degree threshold, that is, and , the initial short-term local window and initial long-term local window of the first data are used as the short-term local window and long-term local window of the first data.

[0090] After obtaining the short-term local window and the long-term local window of the first data, the trend component and the degree of data change trend corresponding to the short-term local window of the first data are obtained according to the method for obtaining the initial trend component and the initial data change trend degree corresponding to the initial short-term local window of the first data.

[0091] Furthermore, after determining the short-term local window of the first data, the degree of data change trend corresponding to the short-term local window of the first data can be , obtain the short-term local window of the next data after the first data (that is, the second data in the network traffic sequence), then the specific method for obtaining the short-term local window of the second data is:

[0092] If the data change trend degree is greater than or equal to a preset second data change trend degree threshold, then calculating a first product of the data change trend degree and a preset first adjustment weight, obtaining the sum of the first product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data;

[0093] If the data change trend degree is less than a preset second data change trend degree threshold, then calculating a second product of the data change trend degree and a preset second adjustment weight, obtaining the sum of the second product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data;

[0094] According to the initial trend component corresponding to the initial short-term local window of the next data and the degree of change trend of the initial data, the short-term local window of the next data is obtained.

[0095] In one embodiment, the second data change trend degree threshold value set in the embodiment of the present invention is 0.5. There is no restriction on the preset second data change trend degree threshold value, which can be set according to the implementation scenario. , the embodiment of the present invention sets the first adjustment weight to -0.1, that is, There is no restriction on the preset first adjustment weight, which can be set according to the implementation scenario. According to the degree of data change trend, the first adjustment weight and the length of the short-term local window of the first data, the length of the initial short-term local window of the second data is obtained. The expression of the length of the initial short-term local window of the second data is:

[0096]

[0097] in, represents the length of the initial short-term local window of the second data, W represents the length of the short-term local window of the first data, Indicates the degree of data change trend corresponding to the short-term local window of the first data. Represents the first adjustment weight.

[0098] It should be noted that The larger the value is, the greater the degree of change trend of the data in the short-term local window and the long-term local window of the first data is, and the more it is necessary to shorten the length of the initial short-term local window of the second data, which helps to more accurately capture the data features of the local change trend in the initial short-term local window of the second data. The smaller the length of the initial short-term local window of the second data is.

[0099] like <0.5, the second adjustment weight is set to 0.1 in the embodiment of the present invention, that is, There is no restriction on the preset second adjustment weight, which can be set according to the implementation scenario. Then, according to the degree of data change trend, the preset second adjustment weight and the length of the short-term local window of the first data, the length of the initial short-term local window of the second data is obtained. The expression of the length of the initial short-term local window of the second data is:

[0100]

[0101] in, represents the length of the initial short-term local window of the second data, W represents the length of the short-term local window of the first data, Indicates the degree of data change trend corresponding to the short-term local window of the first data. Represents the second adjustment weight.

[0102] It should be noted that The smaller the value, the smaller the change trend of the data in the short-term local window and the long-term local window of the first data, indicating that the data is more stable and the more it is necessary to expand the initial short-term local window length of the second data, which helps to reduce the computational overhead and improve the processing efficiency. The larger the initial short-term local window length of the second data is, the smaller the change trend of the data in the short-term local window and the long-term local window of the first data is.

[0103] After the length of the initial short-term local window of the second data is determined, the short-term local window of the second data can be obtained according to the method for obtaining the short-term local window of the first data.

[0104] Step S103, taking the next data as the first data, repeating the method of obtaining the data change trend degree and trend component corresponding to the short-term local window of the first data, until traversing to the last data in the network traffic data sequence, and obtaining the trend component of each data in the network traffic data sequence.

[0105] In step S102, the trend component corresponding to the short-term local window of the first data and the short-term local window of the second data are obtained. Similarly, for the short-term local window of the second data, according to the method for obtaining the trend component corresponding to the short-term local window of the first data and the short-term local window of the second data, the trend component corresponding to the short-term local window of the second data and the short-term local window of the third data are obtained. For the short-term local window of the third data, according to the method for obtaining the trend component corresponding to the short-term local window of the first data and the short-term local window of the second data, the trend component corresponding to the short-term local window of the third data and the short-term local window of the fourth data are obtained, and so on, until the trend component corresponding to the short-term local window of the last data in the network traffic data sequence is obtained, thereby obtaining the trend component corresponding to each data in the network traffic data sequence.

[0106] At this point, the trend component corresponding to each data in the network traffic data series is obtained.

[0107] Step S104, according to the seasonal component and trend component of each data in the network traffic data sequence, respectively obtain the residual component of each data in the network traffic data sequence, and according to the residual component of each data in the network traffic data sequence, screen out abnormal data in the network traffic data sequence.

[0108] According to step S101 to step S103, the seasonal component and trend component of each data in the network traffic data sequence are obtained. By eliminating the seasonal component and trend component of the data, the corresponding residual component is obtained. The residual component reflects the random fluctuation of the data, and the residual component contains a lot of random noise and outliers. By analyzing the difference between the residual component and the preset residual threshold, the abnormal data can be effectively detected. Therefore, in the embodiment of the present invention, first, according to the seasonal component and trend component of each data in the network traffic data sequence, the residual component of each data in the network traffic data sequence is obtained respectively, and then, according to the residual component of each data in the network traffic data sequence, the abnormal data is screened in the network traffic data sequence.

[0109] Among them, according to the seasonal component and trend component of each data in the network traffic data series, the residual component of each data in the network traffic data series is obtained respectively, including:

[0110] In one embodiment, taking the t-th data as an example, the residual component of the t-th data is obtained according to the seasonal component and the trend component of the t-th data in the network traffic data sequence, and the expression of the residual component of the t-th data is:

[0111]

[0112] in, represents the residual component of the tth data, represents the value of the tth data, Represents the trend component corresponding to the short-term local window of the t-th data, represents the seasonal component of the t-th data.

[0113] It should be noted that The smaller the value is, the less obvious the trend of the data in the short-term local window of the t-th data is, indicating that the random fluctuation of the data is more significant, which helps to identify more local abnormal data, and the larger the residual component corresponding to the t-th data is; The smaller the value, the less obvious the seasonal variation of the data in the short-term local window of the t-th data, indicating that the random fluctuation of the data is more significant, which helps to identify more local abnormal data, and the larger the residual component corresponding to the t-th data.

[0114] Among them, according to the residual component of each data in the network traffic data sequence, the abnormal data is screened in the network traffic data sequence, including:

[0115] For any data in the network traffic data sequence, the absolute value of the residual component of the data is obtained, and if the absolute value of the residual component is greater than a preset residual threshold, the data is marked as abnormal data.

[0116] In one embodiment, taking the tth data as an example, the absolute value of the residual component of the tth data is obtained. , if the absolute value of the residual component of the tth data is greater than the preset residual threshold ,Right now , then the tth data is marked as abnormal data.

[0117] Repeat the above method of determining whether the t-th data is abnormal data, obtain the absolute value of the residual component of each data in the network traffic data sequence, and screen out abnormal data in the network traffic data sequence according to the absolute value of the residual component of each data.

[0118] At this point, all abnormal data in the network traffic data sequence are detected based on the residual component of each data in the network traffic data sequence.

[0119] In summary, the present invention performs seasonal decomposition on the acquired network traffic data sequence according to a sliding window of a preset length to obtain the seasonal component of each data in the network traffic data sequence. Since the amount of network traffic data is often very large, the traditional STL calculation efficiency is not high, and when a fixed-size sliding window is used for STL decomposition, the efficiency and accuracy of detecting abnormal data are limited due to the large fluctuation and instability of the network traffic data sequence. Therefore, in an embodiment of the present invention, when trend decomposition is performed on the network traffic data sequence, first, according to the sliding window, a short-term local window and a long-term local window of the first data in the network traffic data sequence are obtained, and according to the data difference between the short-term local window and the long-term local window of the first data, the trend component and data change trend corresponding to the short-term local window of the first data are obtained. Then, according to the data change trend degree corresponding to the short-term local window of the first data, the short-term local window of the next data after the first data is obtained, and the next data is used as the first data. The method of obtaining the data change trend degree and trend component corresponding to the short-term local window of the first data is repeated until the last data in the network traffic data sequence is traversed to obtain the trend component of each data in the network traffic data sequence. Finally, according to the seasonal component and trend component of each data in the network traffic data sequence, the residual component of each data in the network traffic data sequence is obtained respectively. According to the residual component of each data in the network traffic data sequence, the abnormal data is screened in the network traffic data sequence, thereby improving the accuracy of detecting abnormal network traffic data.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A network traffic anomaly detection method based on big data analysis, characterized in that: The method comprises: Acquire a network traffic data sequence within a preset period, and perform seasonal decomposition on the network traffic data sequence according to a sliding window of a preset length to obtain a seasonal component of each data in the network traffic data sequence; When trend decomposing the network traffic data sequence, a short-term local window and a long-term local window of the first data in the network traffic data sequence are obtained according to the sliding window, and a trend component and a degree of data change trend corresponding to the short-term local window of the first data are obtained according to the data difference between the short-term local window and the long-term local window of the first data; According to the data change trend degree corresponding to the short-term local window of the first data, the short-term local window of the next data after the first data is obtained, the next data is used as the first data, and the method of obtaining the trend component and the data change trend degree corresponding to the short-term local window of the first data is repeated until the last data in the network traffic data sequence is traversed to obtain the trend component of each data in the network traffic data sequence; According to the seasonal component and the trend component of each data in the network traffic data sequence, respectively obtain the residual component of each data in the network traffic data sequence, and according to the residual component of each data in the network traffic data sequence, screen out abnormal data in the network traffic data sequence; The step of obtaining the trend component and the degree of data change trend corresponding to the short-term local window of the first data according to the data difference between the short-term local window and the long-term local window of the first data comprises: According to the standard deviation corresponding to the short-term local window of the first data, the fluctuation coefficient corresponding to the short-term local window of the first data is obtained; the first-order difference values ​​of every two adjacent data in the short-term local window of the first data are respectively calculated to obtain the average first-order difference value, and according to the average first-order difference value, the gradient change coefficient corresponding to the short-term local window of the first data is obtained; the fluctuation coefficient and the gradient change coefficient are weightedly summed to obtain the trend component corresponding to the short-term local window of the first data; According to the short-term local window of the first data and the preset sliding step, a sliding traversal is performed on the network traffic data sequence to obtain a preset number of target short-term local windows, and trend components corresponding to the preset number of target short-term local windows are respectively obtained, and a set is formed with the trend components corresponding to the short-term local windows, and the trend components in the set are weighted summed to obtain the local long-term trend component corresponding to the long-term local window of the first data; the maximum data and the minimum data in the short-term local window of the first data are obtained, the trend component and the mean of the local long-term trend component are calculated, a first difference between the mean and the minimum data is obtained, a second difference between the maximum data and the minimum data is obtained, and the ratio of the first difference to the second difference is used as the degree of data change trend corresponding to the short-term local window of the first data.

2. According to the method for detecting network traffic anomalies based on big data analysis according to claim 1, it is characterized in that: The step of obtaining a short-term local window and a long-term local window of the first data in the network traffic data sequence according to the sliding window includes: According to the sliding window, an initial short-term local window of the first data is obtained in the network traffic data sequence, wherein the length of the initial short-term local window is equal to the length of the sliding window; According to the data difference in the initial short-term local window of the first data, an initial trend component corresponding to the initial short-term local window of the first data is obtained; According to the initial short-term local window of the first data and the preset sliding step length, a sliding traversal is performed on the network traffic data sequence to obtain a preset number of target short-term local windows, and the initial short-term local window and the preset number of target short-term local windows are combined to form the initial long-term local window of the first data; Obtaining the degree of initial data change trend corresponding to the initial short-term local window of the first data according to the data difference in the initial long-term local window of the first data and the initial trend component; According to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data, the short-term local window and the long-term local window of the first data are obtained.

3. According to the method for detecting network traffic anomalies based on big data analysis according to claim 2, it is characterized in that: The step of obtaining the initial trend component corresponding to the initial short-term local window of the first data according to the data difference in the initial short-term local window of the first data comprises: Obtaining a volatility coefficient corresponding to the initial short-term local window of the first data according to a standard deviation corresponding to the initial short-term local window of the first data; Respectively calculating the first-order difference value of every two adjacent data in the initial short-term local window of the first data to obtain an average first-order difference value, and obtaining the gradient change coefficient corresponding to the initial short-term local window of the first data according to the average first-order difference value; The fluctuation coefficient and the gradient change coefficient are weightedly summed to obtain an initial trend component corresponding to an initial short-term local window of the first data.

4. According to the method for detecting network traffic anomalies based on big data analysis according to claim 2, it is characterized in that: The step of obtaining the initial data change trend degree corresponding to the initial short-term local window of the first data according to the data difference in the initial long-term local window of the first data and the initial trend component comprises: Respectively obtain the initial trend components corresponding to the preset number of target short-term local windows, form a set with the initial trend components corresponding to the initial short-term local window, perform weighted summation on the initial trend components in the set, and obtain the local long-term trend component corresponding to the initial long-term local window of the first data; Obtain the maximum data and the minimum data in the initial short-term local window of the first data, calculate the mean of the initial trend component and the local long-term trend component, obtain a first difference between the mean and the minimum data, obtain a second difference between the maximum data and the minimum data, and use the ratio of the first difference to the second difference as the degree of initial data change trend corresponding to the initial short-term local window of the first data.

5. The method for detecting network traffic anomalies based on big data analysis according to claim 2 is characterized in that: The step of obtaining the short-term local window and the long-term local window of the first data according to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data comprises: If the initial trend component is greater than the preset trend component threshold, or the initial data change trend degree is greater than the preset first data change trend degree threshold, the length of the initial short-term local window of the first data is doubled to obtain a new short-term local window, and the new short-term local window is used as the initial short-term local window. The method of obtaining the initial trend component and the initial data change trend degree corresponding to the initial short-term local window of the first data is repeated until the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, so as to obtain the short-term local window and long-term local window of the first data.

6. The method for detecting network traffic anomalies based on big data analysis according to claim 2 is characterized in that: The step of obtaining the short-term local window and the long-term local window of the first data according to the initial trend component corresponding to the initial short-term local window of the first data and the degree of change trend of the initial data further includes: If the initial trend component is less than or equal to the preset trend component threshold, and the initial data change trend degree is less than or equal to the preset first data change trend degree threshold, then the initial short-term local window and initial long-term local window of the first data are used as the short-term local window and long-term local window of the first data.

7. The method for detecting network traffic anomalies based on big data analysis according to claim 2 is characterized in that: The step of obtaining the short-term local window of next data after the first data according to the data change trend degree corresponding to the short-term local window of the first data includes: If the data change trend degree is greater than or equal to a preset second data change trend degree threshold, then calculating a first product of the data change trend degree and a preset first adjustment weight, obtaining the sum of the first product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data; According to the initial trend component corresponding to the initial short-term local window of the next data and the degree of change trend of the initial data, the short-term local window of the next data is obtained.

8. The method for detecting network traffic anomalies based on big data analysis according to claim 2 is characterized in that: The step of obtaining the short-term local window of next data after the first data according to the data change trend degree corresponding to the short-term local window of the first data further includes: If the data change trend degree is less than a preset second data change trend degree threshold, then calculating a second product of the data change trend degree and a preset second adjustment weight, obtaining the sum of the second product and the length of the short-term local window of the first data as the length of the short-term local window of the next data after the first data, and obtaining an initial short-term local window of the next data; According to the initial trend component corresponding to the initial short-term local window of the next data and the degree of change trend of the initial data, the short-term local window of the next data is obtained.

9. The method for detecting network traffic anomalies based on big data analysis according to claim 1, characterized in that: The step of screening the network traffic data sequence to obtain abnormal data according to the residual component of each data in the network traffic data sequence includes: For any data in the network traffic data sequence, the absolute value of the residual component of the data is obtained, and if the absolute value of the residual component is greater than a preset residual threshold, the data is marked as abnormal data.

Images