A network traffic anomaly detection method and system

Abstract

The present invention discloses a network traffic anomaly detection method and system, which belongs to the field of network traffic anomaly detection technology. In view of the context dependency of network traffic and the correlation between different feature space dimensions, an adaptive sliding window mechanism is designed. When there are a large number of anomalies in the data, by maintaining the distribution of some historical normal samples, the missed alarm rate of anomaly detection can be effectively reduced. At the same time, the mechanism can dynamically update the joint distribution of normal network traffic features, thereby being able to fully capture the change pattern of normal network traffic and effectively reduce the false alarm rate of anomaly detection. In this process, considering that the Vine Copula function does not require the network traffic features to conform to a specific distribution type, the prior assumption is eliminated and the actual distribution of network traffic features is closer to the actual distribution. By introducing the VineCopula function to fit the distribution of normal samples in the historical window, the coupling relationship between different network traffic features can be accurately established, and network traffic anomaly detection can be performed efficiently and accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of network traffic anomaly detection, and more specifically, relates to a network traffic anomaly detection method and system. Background Art

[0002] Network traffic is a multivariate time series generated by parsing network packets at equal time intervals, implicitly reflecting various user behavior patterns. Anomalies in network traffic are those that disrupt the normal and stable operation of the network, degrade network performance, or even severely impact network availability. These anomalies typically differ from normal traffic patterns.

[0003] Existing anomaly detection algorithms ignore the connection between the contextual associations of multidimensional network traffic features and their local spatial distribution, leading to false positives. Specifically, current unsupervised network traffic anomaly detection methods can be categorized into statistical learning methods and deep learning models. The former fail to consider how the spatial distribution of network traffic feature samples changes over time. Furthermore, some methods require extensive distance calculations on high-dimensional data, which incurs significant time overhead. Some models of the latter require that positive and negative samples have the same spatial distribution, are sensitive to random noise, have complex model structures, and incur high training overhead, failing to meet the requirements for efficient and accurate anomaly detection. Summary of the Invention

[0004] In response to the above defects or improvement needs of the existing technology, the present invention provides a network traffic anomaly detection method and system to solve the technical problem that the existing technology cannot efficiently and accurately perform network traffic anomaly detection.

[0005] To achieve the above object, the present invention provides a method for detecting network traffic anomalies, comprising the following steps:

[0006] S1. After preprocessing the network traffic to be detected for anomalies, a sequence sample with d feature dimensions is obtained. d is a positive integer. The sequence under each feature dimension is obtained by splicing the records under the same feature of the network traffic in time sequence.

[0007] S2. Construct a test window T composed of the historical window W and the test window w in chronological order; in the initial state, the historical window W stores a normal sequence sample with d feature dimensions; the test window w is placed at the beginning of the sequence sample;

[0008] S3. Use the Vine Copula function to calculate the multivariate joint distribution between all feature dimensions of the samples in the inspection window T, and then obtain the marginal distribution of each feature dimension; based on the marginal distribution of each feature dimension, calculate the anomaly score matrix of the test window w in the inspection window T, and determine whether there is an abnormal point in the test window w based on the anomaly score matrix. If so, the samples in the historical window W remain unchanged, slide the test window w on the time axis of the sequence sample, and update the inspection window T; otherwise: move the samples in the test window w into the historical window W, and when the total length of the samples in the historical window W after the shift exceeds the window size of the historical window W, move out the samples with a longer time in the historical window W, and the length of the moved samples is the excess length; slide the test window w on the time axis of the sequence sample, and update the inspection window T;

[0009] S4, repeat step S3 until the window w to be tested slides out of the sequence sample;

[0010] The value of the i-th row and j-th column in the above abnormal score matrix is ​​the sample point x at the j-th moment in the i-th feature dimension in the test window w within the inspection window T. ij The anomaly score is:

[0011]

[0012] M i (x ij ) is the sample point x under the i-th feature dimension ij The marginal distribution value of b i is the sample skewness of the i-th feature dimension in the historical window W within the inspection window T.

[0013] Further preferably, the method of using the Vine Copula function to calculate the multivariate joint distribution between all feature dimensions of sequence samples in the inspection window T includes:

[0014] After calculating the marginal probability density of each feature dimension of the sequence samples in the inspection window T respectively, the binary Copula function is further used to fit the binary joint distribution between each feature dimension. Based on the Cantor tree structure of the Vine Copula function, starting from the leaf nodes of the Cantor tree structure, the binary joint distribution is constructed and merged layer by layer until the root node, thereby obtaining the multivariate joint distribution between all feature dimensions of the sequence samples in the inspection window T.

[0015] Further preferably, the binary Copula function is a binary Copula function with minimum Akaike information and Bayesian information in the binary Copula function candidate set; the binary Copula functions in the binary Copula function candidate set include: Gaussian function, t-Copula function and Frank function.

[0016] Further preferably, the method for determining whether there is an abnormal point in the test window w based on the abnormality score matrix includes:

[0017] Compare the abnormality score corresponding to each sample point in the test window w with the preset threshold. If it is greater than, the sample point is determined to be abnormal; otherwise, the sample point is determined to be normal.

[0018] Further preferably, the method for setting the preset threshold τ includes:

[0019] Get the minimum value min, maximum value max and average value Avg of the anomaly score in the anomaly score matrix;

[0020] The value of the preset threshold τ is increased from min to max. In this process, the probability P of the abnormal score in the abnormal score matrix being less than τ is calculated respectively. lower The probability P that the sum is greater than τ upper , and the mean of the anomaly scores less than τ Avg lower and the mean of the anomaly scores greater than τAvg upper , and calculate the corresponding 2 2result=P lower ×(Avg-Avg lower )+P upper ×(Avg-Avg upper );

[0021] The maximum value of the result is used as the final preset threshold τ.

[0022] Further preferably, the method for preprocessing network traffic to be subjected to anomaly detection includes: deleting duplicate values ​​in the network traffic, and using the values ​​in the deleted missing value timestamp field as a reference, performing linear interpolation and normalization on them.

[0023] Further preferably, the above d features are obtained by screening all features contained in the network traffic based on correlation analysis; the specific screening method includes:

[0024] The Pearson coefficient is used to measure the correlation between two network traffic features. When the correlation between two features is greater than a first preset value, only one feature is retained.

[0025] Spearman is used to measure the correlation between each feature dimension and the degree of abnormality, and the obtained correlation degrees are sorted from large to small, and the first d features with higher correlation degrees are retained.

[0026] Further preferably, the time window size and sliding step size of the window to be measured w are both p; the time window size k of the historical window W is an integer multiple of p.

[0027] Further preferably, the above-mentioned network traffic anomaly detection method further includes step S5, which specifically includes: after the abnormal point is detected, obtaining abnormal time series segments with d feature dimensions within the neighborhood of the abnormal point, and inputting them into the classification model to obtain the state category of the abnormal point;

[0028] The classification model includes a cascaded feature extraction network and a classification network; the feature extraction network is the encoding network in the pre-trained SimSiam model;

[0029] When an outlier is detected, the outlier time series segments with d feature dimensions in the neighborhood of the outlier are saved in a buffer pool. When the number of time series segments in the buffer pool exceeds a preset threshold, the SimSiam model is trained. The training method of the SimSiam model includes:

[0030] For each abnormal time series segment in the buffer pool, under the condition of retaining the abnormal time series segment, sample forward a constant time step on the sequence sample time axis to obtain an abnormal sample V, and sample backward the same constant time step to obtain an abnormal sample V', and obtain the corresponding positive sample pair<V,V'> ; Each positive sample pair<V,V'> Input into SimSiam module for comparative learning;

[0031] The training method of the above classification model includes: inputting the pre-collected training sample set into the classification model for training; the training sample set includes: time series segments with different category labels and corresponding state category labels; the state categories include normal categories and different abnormal categories.

[0032] Further preferably, the encoding network in the SimSiam module includes: a cascaded encoding module and a projection module; the encoding module includes a cascaded three-layer encoder, and the projection module includes a cascaded two-layer projection perceptron.

[0033] Further preferably, the above encoder is a GRU network.

[0034] Further preferably, after obtaining the positive sample pair<V,V'> Afterwards, the common parts of V and V' are transformed in various ways by random masking, and the data is expanded with positive samples.

[0035] In a second aspect, the present invention provides a network traffic anomaly detection system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the network traffic anomaly detection method provided by the first aspect of the present invention when executing the computer program.

[0036] In a third aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by the processor, the device where the storage medium is located is controlled to execute the network traffic anomaly detection method provided in the first aspect of the present invention.

[0037] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0038] 1. The present invention provides a network traffic anomaly detection method. Aiming at the contextual dependence of network traffic and the correlation between different feature space dimensions, an adaptive sliding window mechanism is designed. Specifically, by maintaining a historical window with an upper bound containing historical normal samples, a joint distribution of normal samples within the nearest time range to the test window is constructed. After detecting anomalies in the samples within the test window, the historical sliding window is dynamically updated based on the results. Since the distribution of the most recent historical normal samples is maintained, when a large number of abnormal samples exist within a certain test window, the mechanism can effectively reduce the false negative rate of anomaly detection. At the same time, the mechanism can dynamically update the joint distribution of normal network traffic features, thereby fully capturing the changing patterns of normal network traffic and effectively reducing the false positive rate of anomaly detection. In this process, considering that the Vine Copula function does not require network traffic features to conform to a specific distribution type, the prior assumption is eliminated and the actual distribution of network traffic features is closer to the actual distribution. By fitting the distribution of normal samples within the historical window with the Vine Copula function, the coupling relationship between different network traffic features can be accurately established, and network traffic anomaly detection can be performed efficiently and accurately.

[0039] 2. Furthermore, the network traffic anomaly detection method provided by the present invention, after the abnormal samples are detected through steps S1-S4, further classifies the obtained samples based on the state classification of step S5 to reduce the impact of false alarms on the network traffic detection effect. In this process, the present invention designs a reasonable data enhancement method based on the time series characteristics of abnormal network traffic to avoid the wrong negative sample pair division method, and constructs a classification model based on the encoding network in the SimSiam model. By inputting the abnormal network traffic fragments after disturbance into the SimSiam module one by one, the representations generated after the projection of the encoding network mapping of the same type of abnormal samples in the SimSiam module are closer on the hypersphere, and the generated disturbances are more evenly distributed on the manifold. The whole process does not require negative sample pairs; in addition, the key homogeneous information of the same type of positive sample pairs can be extracted through comparative learning. Compared with the method of using autoencoders or generative adversarial networks, the training overhead is smaller and it is not easy to fall into the dilemma of overfitting or mode collapse. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Flowchart of the network traffic anomaly detection method provided in Example 1 of the present invention;

[0041] Figure 2 A schematic diagram of a feature correlation matrix provided in Example 1 of the present invention;

[0042] Figure 3 This is a flow chart for calculating the anomaly score matrix provided in Example 1 of the present invention;

[0043] Figure 4 A schematic diagram of the structure of the SimSiam module provided in Example 1 of the present invention;

[0044] Figure 5 A schematic diagram of a method for constructing erroneous negative sample pairs provided in Example 1 of the present invention;

[0045] Figure 6 A schematic diagram of a method for constructing positive sample pairs provided in Example 1 of the present invention;

[0046] Figure 7 This is a flow chart of a network traffic anomaly detection method provided by an optional implementation method of Example 1 of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0048] The present invention provides a network traffic anomaly detection method and system. Using an anomaly detection algorithm, the system identifies anomalous network traffic generated by network attacks, clearly defining the boundaries between normal and abnormal network user behavior patterns. This allows for efficient detection of suspected anomalous network traffic and prompts timely alerts. Furthermore, the system can input suspected anomalous traffic into a classification model, enabling the model to learn the manifold embedding representations of various malicious behaviors reflected in network traffic carriers and promptly correct false positives in anomaly detection results. Specifically, the classification model divides suspected anomalous traffic into anomalous data generated by different attack methods and a small number of normal samples that have been misidentified due to anomaly detection errors.

[0049] Example 1

[0050] A network traffic anomaly detection method, such as Figure 1 As shown, the following steps are included:

[0051] S1. After preprocessing the network traffic to be detected for anomalies, a sequence sample with d feature dimensions is obtained. d is a positive integer. The sequence under each feature dimension is obtained by splicing the records under the same feature of the network traffic in time sequence.

[0052] Specifically, due to network fluctuations, delays, and process communication blockages, raw network traffic data contains some duplicate values ​​with the same timestamp and missing values ​​at different times. If these erroneous sample data are not processed, incorrect patterns will be learned later, which will affect the final anomaly analysis results. Therefore, before performing anomaly analysis, it is necessary to remove duplicate values ​​from the raw data and fill in missing values. In one optional embodiment, the method for preprocessing network traffic to be detected for anomaly includes: denoising and normalization processing, specifically: deleting duplicate values ​​in the network traffic, and using the values ​​in the deleted missing value timestamp area as a reference, performing linear interpolation to fill in the missing values, and then performing normalization processing. This embodiment uses a set data structure to eliminate duplicate values, and uses the feature values ​​in the missing value timestamp area as a reference to perform linear interpolation to fill in the missing values, thereby eliminating the negative impact of erroneous samples on subsequent work. Furthermore, because some attributes have different dimensions and have large differences in numerical ranges, different features have different degrees of influence on network weights during the model learning process, which cannot accurately reflect the true state of network traffic. Therefore, the above preprocessing method also includes normalization processing. This embodiment compares the Min-Max normalization method with the Z-Score normalization method and finds that using the Min-Max normalization method to process all values ​​within the sample attribute range will produce more accurate anomaly analysis results.

[0053] Based on the KDD Cup 99 dataset (a widely used dataset for network traffic anomaly analysis), the network traffic features are shown in Table 1:

[0054] Table 1

[0055] Serial number Feature Name meaning 1 duration Duration of the connection 2 protocol_type Protocol Type 3 service Service Type 4 flag Connection status code 5 src_bytes Number of bytes sent by the source address 6 dst_bytes Number of bytes received by the source address 7 land Connection status code 8 wrong_fragment Number of error segments 9 urgent Number of emergency kits 10 hot Number of times sensitive files or directories were accessed 11 num_failed_logins Number of failed login attempts 12 logged_in Whether the login is successful 13 num_compromised Number of error-free connections 14 root_shell Whether to obtain root_shell 15 su_attempted Whether to execute super administrator commands 16 num_root Root user access 17 num_file_creations Number of files created during the connection 18 num_shells Number of normal user logins 19 num_access_files Number of files accessed during the connection 20 num_outbound_cmds Number of outbound ftp sessions 21 is_hot_login Whether to log in with root privileges 22 is_guest_login Whether to log in with guest permissions 23 count The number of connections with the same target host within 2 seconds 24 srv_count The number of connections with the same destination port within the past 2 seconds 25 serror_rate The historical SYN connection error ratio of the same target host within 2 seconds 26 srv_serror_rate The historical SYN error connection ratio of the same service within 2 seconds 27 rerror_rate The ratio of REJ error connections to the same target host within the past 2 seconds 28 srv_rerror_rate The ratio of REJ error connections in the same service within the past 2 seconds 29 sane_srv_rate Historical proportion of same service connections within 2 seconds 30 diff_srv_rate Historical connection ratios of different services within 2 seconds 31 srv_diff_host_rate The historical connection ratio of different target hosts within 2 seconds 32 dst_host_count Number of connections with the same destination IP within a 100-connection time window 33 dst_host_srv_count Number of connections to the same service within a 100-connection time window (the same below) 34 dst_host_same_srv_rate The proportion of connections to the same target host with the same service 35 dst_host_diff_srv_rate The proportion of different services in connections to the same target host 36 dst_host_same_src_port_rate The percentage of connections with the same destination host and the same source port 37 dst_host_srv_diff_host_rate The percentage of connections with the same destination host but different source hosts 38 dst_host_serror_rate The ratio of SYN connection errors to the same target host 39 dst_host_srv_serror_rate The ratio of SYN connection errors to the same target host and the same service 40 dst_host_rerror_rate Ratio of REJ error connections to the same target host 41 dst_host_srv_rerror_rate The proportion of REJ error connections to the same target host and the same service

[0056] It should be noted that the d feature dimensions selected can be determined based on experience, and feature dimensions related to abnormal conditions should be selected while ensuring that the direct correlation between the feature dimensions is weak. Preferably, the d feature dimensions are obtained by screening all features contained in the network traffic based on correlation analysis. The specific process is as follows:

[0057] Since the proportion of abnormal samples in real-world network traffic is low and the sample distribution is extremely unbalanced, this embodiment selects time series features related to network traffic anomalies based on eliminating redundant samples for extremely unbalanced data sets. This embodiment selects 36 continuous features and further analyzes their correlation. After preprocessing the network traffic data according to the above preprocessing method, the records of the same feature are spliced ​​into a continuous random variable X in time series. i (x1,x2,…,x t ), i=1, 2, 3, ..., 36. Specifically, in an optional implementation manner, the method for screening features based on correlation analysis includes:

[0058] 1) The Pearson coefficient is used to measure the correlation between two network traffic features. When the correlation between two features is greater than a first preset value, only one of the features is retained. Specifically, the Pearson coefficient is shown below, where the value range of r is [-1, 1]. When the value of the Pearson coefficient is close to 1, it indicates that the two random variables are almost positively linearly correlated.

[0059]

[0060] To avoid selecting redundant features, we can select one from a pair of features with strong correlation based on the subsequent feature selection results. The correlation matrix is ​​as follows: Figure 2 As shown in the figure, the depth of the heat map color block reflects the strength of the correlation between features. In addition, for two features, their correlation has nothing to do with the combination order, so in Figure 2The symmetrical upper half is hidden in the figure. From the heat map, we can see that the feature pairs with strong correlation are: serror_rate and srv_serror_rate, rerror_rate and srv_error_rate, dst_host_serror_rate and dst_host_srv_serror_rate.

[0061] 2) Spearman is used to measure the correlation between each feature dimension and the degree of abnormality, and the obtained correlation degrees are sorted from large to small, and the first d features with higher correlation degrees are retained.

[0062] Specifically, the time series sample label used to describe the degree of anomaly within a time window can also be regarded as a discrete random variable Y. The Spearman coefficient is used to evaluate the correlation between a continuous random variable and a discrete random variable. The specific formula is as follows:

[0063]

[0064] In this embodiment, under the premise of maintaining a confidence level of 95%, the Spearman coefficients of the features are sorted from large to small, and finally the first d features with a higher degree of correlation are retained.

[0065] Furthermore, since anomalies closely related to time series features may exist in the traffic data, this embodiment adopts the aforementioned filtered features and uses the tsfresh mathematical toolkit to further enhance the time series statistical features.

[0066] S2. Construct a test window T composed of the historical window W and the test window w in chronological order. Initially, the historical window W contains a normal historical sequence sample with d feature dimensions. The test window w is placed at the beginning of the sequence sample.

[0067] S3. Use the Vine Copula function to calculate the multivariate joint distribution between all feature dimensions of the samples in the inspection window T, and then obtain the marginal distribution of each feature dimension; based on the marginal distribution of each feature dimension, calculate the anomaly score matrix of the test window w in the inspection window T, and determine whether there is an abnormal point in the test window w based on the anomaly score matrix. If so, the samples in the historical window W remain unchanged, slide the test window w on the time axis of the sequence sample, and update the inspection window T; otherwise: move the samples in the test window w into the historical window W, and when the total length of the samples in the historical window W after the shift exceeds the window size of the historical window W, move out the samples with a longer time in the historical window W, and the length of the moved samples is the excess length; slide the test window w on the time axis of the sequence sample, and update the inspection window T;

[0068] It should be noted that most of the current unsupervised network traffic anomaly detection methods do not take the entire time series as the research object, but select fixed time intervals to detect it. First of all, this is because detecting the entire time series does not conform to the situation in the real scene. If the sampling is fixed at a time interval of 5 seconds, the length of the traffic time series data monitored for 3 days can reach more than 50,000. Modeling a time series of this magnitude requires a lot of computing resources, which is difficult to meet the needs of online real-time detection. Secondly, the time series data within the neighborhood of the observation point has a certain long-term or short-term dependency with it, and the influence of the sample points beyond the neighborhood range on the observation sample gradually weakens with the increase of distance. If the time series data is modeled only for the whole, subtle time series features are often ignored, which in turn affects the accuracy of anomaly detection. Therefore, the present invention selects a historical window of appropriate size and a window to be tested as the research object to detect abnormal time series fragments.

[0069] Preferably, in an optional embodiment, the time window size and sliding step size of the window to be measured w are both p; the time window size k of the historical window W is an integer multiple of p; in this embodiment, k is 90 and p is 5. Specifically, a historical window W with an upper limit of a time window size of k is set to retain the historical observation values ​​of normal samples, and a window to be measured w with a window size equal to the moving step size is set; the historical window W is used to eliminate the impact of a large number of abnormal samples that may exist in the sliding window to be measured on the accuracy of abnormality detection; the window size of the window to be measured w is p, which is divisible by k, and always contains the samples to be observed during the sliding period of the window to be measured w. The operating mechanism of the sliding window is: in the initial state, first save the normal samples that are smaller than the upper limit of the historical sliding window size to the historical sliding window W, and then detect whether there are abnormal samples in the sliding window to be measured w of size p based on the distribution of normal samples in the historical sliding window W. Specifically, in the process of determining abnormal samples, the data in the historical sliding window W and the data in the test sliding window w are spliced ​​in chronological order to form a temporary inspection window T. To reduce computational overhead, the intermediate results of the historical sliding window can be retained, and the sample distribution within the temporary inspection window can be calculated incrementally. If there are abnormal points in the generated anomaly score matrix, the test sliding window moves to continue to delineate the next section of network traffic samples to be detected, and the historical sliding window is not updated. If there are no abnormal samples, the historical sliding window incorporates the samples in the test sliding window into the historical observations. If the inclusion exceeds the upper limit of the window size, the long-standing historical observation samples in the window are removed, and the test window is then moved again, repeating this process until the test window exceeds the length of the sequence sample.

[0070] Specifically, if Figure 3 As shown in FIG, the method of using the Vine Copula function to calculate the multivariate joint distribution of all feature dimensions of sequence samples in the inspection window T includes:

[0071] After calculating the marginal probability density of each feature dimension of the sequence samples in the inspection window T respectively, the binary Copula function is further used to fit the binary joint distribution between each feature dimension. Based on the Cantor tree structure of the Vine Copula function, starting from the leaf nodes of the Cantor tree structure, the binary joint distribution is constructed and merged layer by layer until the root node, thereby obtaining the multivariate joint distribution of all feature dimensions of the sequence samples in the inspection window T.

[0072] Specifically, we first calculate the marginal probability density of each characteristic dimension of network traffic using histogram statistics or kernel density estimation. Then, we select the optimal binary Copula function to fit the binary joint distribution between each characteristic dimension in network traffic. Finally, starting from the leaf node of the Cantor tree, we construct and merge multiple binary joint distributions layer by layer until the root node, thereby constructing the joint distribution of all characteristic dimensions of network traffic.

[0073] It should be noted that the reason why the Cantor tree structure of the Vine Copula function can be used to fit the multivariate joint distribution is that the feature dimensions in the network traffic that are not related to anomalies have been eliminated through the above preprocessing and time series feature extraction process, thereby reducing noise interference in the fitting process. In addition, the feature dimensions with strong correlation have been reduced based on the Pearson coefficient, so that the probability density of each feature dimension of the network traffic can be approximated as the marginal probability density of the joint distribution.

[0074] In addition, since contextual features such as convolution, differentiation, and accumulation of the original network traffic data are extracted, the characteristics of the network traffic are closely coupled with the original data. Therefore, when constructing the joint distribution, the original data and features should be modeled separately. When identifying anomalies, if the sample to be tested is in an extremely low probability density space in any distribution, it is judged as an anomaly.

[0075] In an optional implementation, a kernel density estimation algorithm of a non-parametric test is used to calculate the probability density function of each feature of the network traffic within the inspection time window. Let X be a random variable composed of samples of a certain feature space dimension within the inspection window T, and m be the sample size. This embodiment uses a radial basis kernel function to perform kernel density estimation. In the kernel density estimation algorithm, the smoothing coefficient h is used to control the smoothness of the probability density curve. Let μ∈R be the average value of the samples in the historical observation window W, and σ∈R be the standard deviation, then the smoothing coefficient h can be calculated by an empirical formula.

[0076]

[0077] Skewness reflects the degree of asymmetry of data distribution and plays an important role in the characterization of non-normal distribution data. j is a sample in the historical window W, then the skewness b is:

[0078]

[0079] If the skewness b is greater than 0, the data is more dense on the left side of the axis; if the skewness b is less than 0, the data is more dense on the right side of the axis; if the skewness b is equal to 0, it is a normal distribution. The marginal probability density function under the i-th feature dimension is:

[0080]

[0081] Perform the above calculations for each feature space dimension to obtain the marginal probability density function f under each feature dimension i (X)(i=1,2,3…,d)。

[0082] Furthermore, selecting an appropriate binary Copula connection function based on the Vine Copula function model is the key to accurately constructing a multivariate joint distribution model. The following describes a method for selecting the optimal binary Copula function. This embodiment uses a semi-parametric maximum likelihood estimation (MLE) method to obtain the relevant parameters of the target binary Copula function, and uses the Akaike information and Bayesian information criterion to evaluate the fitting effect of the binary Copula function. Specifically, taking binary random variables x1 and x2 as an example, a natural logarithm likelihood function can be established based on the sample values:

[0083]

[0084] Where c(·) is the binary Copula probability density function, and is a set of sample points in the random variables x1 and x2, and n is the network traffic sample size. The problem of maximizing the natural log-likelihood function can be reduced to finding a point θ in the parameter space. * , so that the following formula holds:

[0085]

[0086] Many parameter estimation problems use the likelihood function as an indicator. This can lead to better fit when using large sample sizes, but this comes at the cost of increased model complexity, leading to overfitting, a common problem in machine learning. Therefore, it's important to strike a balance between model complexity and the model's representational power. The Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) are two common metrics for measuring model fit.

[0087] The above measurements are performed on the binary Copula functions in the binary Copula function candidate set to obtain each parameter set. The variable sample pairs and each candidate Copula parameter set are used as input and substituted into the following formula to obtain the BIC and AIC values ​​of the candidate function set:

[0088]

[0089]

[0090] Where k is the number of parameters in c(·). The BIC penalty term is greater than the AIC. When the sample size is too large, it can effectively avoid selecting a model with overly high precision and reduce computational costs. Ultimately, based on the calculation results, the binary copula function with the minimum AIC and BIC values ​​is the optimal choice. When AIC and BIC conflict, the AIC value should be prioritized.

[0091] Based on this, the binary Copula function used in this embodiment is the binary Copula function with the minimum Akaike information and Bayesian information in the binary Copula function candidate set; in this embodiment, the binary Copula functions in the binary Copula function candidate set include: Gaussian function, t-Copula function and Frank function.

[0092] Furthermore, the value of the i-th row and j-th column in the abnormal score matrix is ​​the sample point x at the j-th moment in the i-th feature dimension in the test window w within the inspection window T. ij The anomaly score is:

[0093]

[0094] M i (x ij ) is the sample point x under the i-th feature dimension ij The marginal distribution value of b i is the sample skewness of the i-th feature dimension in the historical window W within the inspection window T.

[0095] Furthermore, the method for determining whether there is an abnormal point in the test window w based on the abnormality score matrix includes:

[0096] Compare the abnormality score corresponding to each sample point in the test window w with the preset threshold. If it is greater than, the sample point is determined to be abnormal; otherwise, the sample point is determined to be normal.

[0097] The present invention can determine the health status of the current network based on the anomaly score matrix, which helps network operation and maintenance engineers estimate the severity of anomalies based on alarms, take corresponding security defense measures for different levels of network security incidents, and repair network failures in a timely manner.

[0098] It should be noted that the above-mentioned preset threshold value can be set according to experience.

[0099] In an optional implementation manner, a method for setting the preset threshold τ includes:

[0100] Get the minimum value min, maximum value max and average value Avg of the anomaly score in the anomaly score matrix;

[0101] The value of the preset threshold τ is increased from min to max. In this process, the probability P of the abnormal score in the abnormal score matrix being less than τ is calculated respectively. lower The probability P that the sum is greater than τ upper , and the mean of the anomaly scores less than τ Avg lower and the mean of the anomaly scores greater than τAvg upper , and calculate the corresponding result = P lower ×(Avg-Avg lower ) 2 +P upper ×(Avg-Avg upper ) 2 ;

[0102] The maximum value of the result is used as the final preset threshold τ.

[0103] S4. Repeat step S3 until the window w to be tested slides out from the end of the sequence sample.

[0104] Based on the above method, the present invention conducted experiments on two datasets, KDD Cup 99 and NSL-KDD. The experimental results showed that the average anomaly detection precision of the present invention in these two datasets was 0.924, the average recall was 0.908, and the average F1 score was 0.916. The average training time was 16 seconds, and it supported online anomaly detection. Compared with anomaly detection algorithms based on statistical learning methods, the average precision was improved by 14.2%, the average recall was improved by 22.2%, and the average F1 score increased by 18.3%. Compared with anomaly detection algorithms based on deep neural network models, the average precision was improved by 3.4%, the average recall was improved by 9.2%, and the average F1 score increased by 6.4%. The experiments confirmed the effectiveness of the above method provided by the present invention.

[0105] Furthermore, based on the above experimental results, it can be seen that the above method has the problem of false alarms. In order to reduce the impact of false alarms on the network traffic detection effect and assist experts in quickly taking effective network defense measures or carrying out network fault repair work according to different abnormal network traffic types, in an optional embodiment, the above network traffic anomaly detection method further includes step S5, which specifically includes: after the abnormal point is detected, obtaining an abnormal time series segment with d feature dimensions within the neighborhood of the abnormal point, and inputting it into the classification model to obtain the state category of the abnormal point;

[0106] The classification model includes a cascaded feature extraction network and a classification network; the feature extraction network is the encoding network in the pre-trained SimSiam model;

[0107] When an outlier is detected, abnormal time series segments with d feature dimensions in the neighborhood of the outlier are obtained and saved in a buffer pool. When the number of time series segments in the buffer pool exceeds a preset threshold, the SimSiam model is trained. The training method of the SimSiam model includes:

[0108] For each abnormal time series segment in the buffer pool, under the condition of retaining the abnormal time series segment, sample forward a constant time step on the sequence sample time axis to obtain an abnormal sample V, and sample backward the same constant time step to obtain an abnormal sample V', and obtain the corresponding positive sample pair<V,V'> ; Each positive sample pair<V,V'> Input into SimSiam module for comparative learning;

[0109] The training method of the above classification model includes: inputting the pre-collected training sample set into the classification model for training; the training sample set includes: time series segments with different category labels and corresponding state category labels; the state categories include normal categories and different abnormal categories.

[0110] It should be noted that the key to the SimSiam module's lack of negative sample pairs lies in predicting the perturbations of the perceptron on the projected representation, making them more evenly distributed on the manifold. Simultaneously, the other branch of the twin network stops updating gradients and slowly updates parameters through shared parameters, similar to the target network of the reinforcement learning DQN algorithm. The training process of the SimSiam module can be approximated as solving the EM problem, and the process of constructing positive sample pairs can be viewed as randomly masking similar data. Through contrastive learning, key homogeneous information can be extracted from similar positive sample pairs. Compared to methods using autoencoders or generative adversarial networks, the twin network contrastive learning framework has a simpler structure, lower training overhead, and is less prone to overfitting.

[0111] Specifically, after the method of steps S1-S4 above identifies an abnormal network traffic sample, it is saved in a cache pool. After the sample capacity in the cache pool exceeds the set threshold, the SimSiam module is trained in an unsupervised manner by default based on the prior knowledge of anomaly categories to correct the false alarms and abnormal traffic classification of the method of steps S1-S4 above. The purpose is to utilize the low time overhead of the anomaly detection algorithm to accelerate the anomaly classification model, and at the same time correct the false alarms of the anomaly detection algorithm based on the high accuracy of the anomaly classification model. When network experts receive anomaly alarms and classification prompts, they will promptly carry out existing network repair work, and at the same time quickly analyze anomalies of known categories according to business characteristics to ensure the availability and stability of network services.

[0112] Preferably, the encoding network in the SimSiam module includes: a cascaded encoding module and a projection module; the encoding module includes a cascaded three-layer encoder, and the projection module includes a cascaded two-layer projection perceptron. Preferably, the above-mentioned encoder is a GRU network. It should be noted that compared with RNN, LSTM and GRU can well overcome the problems of gradient vanishing and gradient explosion caused by too many hidden layers, while being able to capture the long-term dependencies of historical time series. The performance of the two is comparable in many tasks. Compared with LSTM, GRU has reduced a "gate" structure, has relatively fewer parameters and is easy to train and converge. With a limited number of network layers, the model uses GRU as the encoder component to achieve classification accuracy close to LSTM with lower training time overhead. Therefore, in this embodiment, GRU is preferably selected as the encoder of the classification model.

[0113] Specifically, in an optional embodiment, as Figure 4 Figure 2 shows the structural diagram of the SimSiam module. The model consists of a buffer pool component and a twin GRU network anomaly classification component. The buffer pool component is designed to store a certain capacity of abnormal network traffic samples to form an offline training dataset. During actual training, the sample pool is divided into a test set and a validation set in a ratio of 7:3, and cross-validation is used to select the classification threshold.

[0114] Due to the lack of abnormal category labels, the present invention supports the "pre-training-fine-tuning" paradigm of self-supervised learning to improve classification accuracy. First, data enhancement is performed on abnormal samples without category labels through multiple transformations, and multiple positive sample pairs are generated with the abnormal samples themselves as references; secondly, similar positive sample pairs are input into the SimSiam module, and the parameters in the SimSiam module are updated through continuous backpropagation, so that after the positive sample pairs are embedded by the encoder, the distance between their representation vectors generated by the hypersphere projection is close; finally, after data enhancement is performed on a small number of abnormal samples with category labels, positive and negative sample pairs are generated, and the model parameters are fine-tuned according to the requirements of downstream classification tasks.

[0115] Specifically, remember that there is an abnormal sample V in the cache pool. First, multiple enhanced sample pairs are generated.<V,V'> , this model uses two GRU encoders to receive the two samples respectively, embed them into the high-dimensional hypersphere space, and then pass them through a two-layer fully connected network through projection transformation f(·) to obtain two easily comparable vectors. It is worth noting that the feature vector is batch normalized after passing through the fully connected hidden layer, and then activated by the ReLU function. Finally, one of the vectors V is connected to the prediction perceptron to predict the embedding vector error. The setting between each hidden layer of the prediction perceptron is the same as the aforementioned fully connected network, and its operator is p(·). After passing through the prediction perceptron, the vector V will be compared with the other vector V' for cosine similarity. The formula is as follows.

[0116]

[0117] The total loss function is:

[0118]

[0119] Here, m is the number of positive sample logarithms. To enhance the generalization ability of the model, a regularization term μR(Ω) is often added to the formula. Ω is the neural network weight parameter, and μ is a hyperparameter. It is worth noting that when backpropagating to update the gradient, one branch of the twin network does not receive gradient information. This is done to prevent the model from collapsing and learning a trivial solution. When the training is complete, the prediction perceptron is removed, and the vector after encoding and projection is the representation of the abnormal time series segment, which can be used for downstream classification tasks. If small sample data labeled by experts is available in a specific business scenario, the softmax function can be used to replace the multi-layer perceptron. A similar method is used to perform data enhancement on the labeled positive and negative samples. The model is initialized using the network parameters learned from the previous pre-training. The positive and negative samples are then input into the model to fine-tune the network parameters. After training, the multi-dimensional abnormal time series segment within the time window is input, and the state category of the abnormal sample to be tested is output.

[0120] Furthermore, in the above process, in order to obtain sufficient abnormal samples for training, the sample capacity threshold of the buffer pool is determined to be 128 in this embodiment. k There is an abnormal point t within the timestamp range a , by analyzing t a The time series attributes within the neighborhood can capture the changing pattern of abnormal time series more comprehensively. i A single sample at can be represented as a vector with a feature space dimension of d Some self-supervised contrastive learning models treat samples from different batches as negative samples, which may lead to the following problems: Figure 5As shown in the error, the samples in batch 1 and batch 2 belong to the same type of network traffic anomaly, but are mistakenly regarded as negative sample pairs, which may cause the model to learn the wrong pattern. Figure 6 The data enhancement method shown in the figure retains the abnormal samples V in the time window and samples a constant time step forward or backward as positive samples.<V,V'> On this basis, various transformations such as random masking are performed on the common parts of the sample pairs to enhance the generalization ability and robustness of the model, forming multiple positive sample pairs for subsequent classification tasks. Although the contrastive learning framework that does not rely on negative sample pairs can avoid the risk of incorrectly dividing sample pairs mentioned above, and positive sample pairs are easy to obtain, this method of constructing positive sample pairs inevitably reduces the training sample capacity. Therefore, data transformation methods should be increased as much as possible to avoid the model from falling into the dilemma of overfitting due to lack of samples. Assuming that the dataset contains only two types of samples, A and B, if m data augmentation methods are used, the training sample capacity is m(2m-1) times that of a single training method. Since m>1, the growth multiple increases monotonically within this range, and the second-order derivative function is greater than 0, so the transformation methods should be expanded as much as possible to maximize the training set capacity, thereby better avoiding the phenomenon of overfitting of the model.

[0121] In summary, in order to solve the problem that existing anomaly classification models are difficult to learn high-dimensional abstract representations of various types of abnormal network traffic and have high computational overhead, the present invention applies the twin network contrast learning framework that does not require negative sample pairs to the classification of network traffic anomalies, and proposes a new model. The model uses the twin GRU network as the skeleton and adopts enhanced sampling to automatically construct multiple groups of positive sample pairs, adding random masks to enhance the robustness of the model. The model is then input into the twin GRU network to achieve spatiotemporal feature mapping, and finally clustering is completed based on contrast loss. The model supports the "pre-training-fine-tuning" method to improve accuracy. Specifically, according to the time series characteristics of abnormal network traffic, the present invention designs a reasonable data enhancement method to avoid the wrong negative sample pair division method, and then inputs the perturbed abnormal network traffic segments into the GRU twin neural network pair by pair. Through the contrast loss function, the representations generated after the GRU encoder mapping and projection of similar abnormal samples are closer on the hypersphere. The key to this classification model that does not require negative sample pairs is to predict the perturbation of the projected representation by the perceptron to make it more evenly distributed on the manifold. At the same time, the other branch of the twin network stops updating the gradient. Therefore, the training process of the entire model can be approximated as the process of solving the EM problem. The process of constructing positive sample pairs can be regarded as a reasonable perturbation of the data. Through contrastive learning, the key homogeneous information of similar positive sample pairs can be extracted. Compared with the method of using autoencoders or generative adversarial networks, the training overhead of the twin network contrast learning framework is smaller and it is not easy to fall into the dilemma of overfitting or mode collapse.

[0122] Specifically, the overall process of this embodiment is as follows: Figure 7 shown.

[0123] Example 2

[0124] A network traffic anomaly detection system includes: a memory and a processor, wherein the memory stores a computer program, and the processor executes the network traffic anomaly detection method provided in embodiment 1 of the present invention when executing the computer program.

[0125] The relevant technical solutions are the same as those in Example 1 and will not be described in detail here.

[0126] Example 3

[0127] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the network traffic anomaly detection method provided in Example 1 of the present invention.

[0128] The relevant technical solutions are the same as those in Example 1 and will not be described in detail here.

[0129] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting anomaly in network traffic, characterized in that: The following steps are involved: S1. After preprocessing the network traffic to be detected for anomalies, a sequence sample with d feature dimensions is obtained; d is a positive integer; The sequence under each feature dimension is obtained by splicing the records under the same feature of network traffic in time sequence; S2. Construct a test window T composed of the historical window W and the test window w in chronological order; in the initial state, the historical window W stores a normal sequence sample with d feature dimensions; the test window w is placed at the beginning of the sequence sample; S3. Use the Vine Copula function to calculate the multivariate joint distribution between all feature dimensions of the samples in the inspection window T, and then obtain the marginal distribution of each feature dimension; based on the marginal distribution of each feature dimension, calculate the anomaly score matrix of the test window w in the inspection window T, and determine whether there is an outlier in the test window w based on the anomaly score matrix. If so, the samples in the historical window W remain unchanged, the test window w is slid on the time axis of the sequence samples, and the inspection window T is updated; Otherwise: move the samples in the test window w into the history window W. When the total length of the samples in the history window W exceeds the window size of the history window W, remove the samples with a longer time in the history window W. The length of the removed samples is the excess length. Slide the test window w on the time axis of the sequence samples and update the inspection window T. S4, repeat step S3 until the window w to be tested slides out of the sequence sample; The value of the i-th row and j-th column in the abnormal score matrix is ​​the sample point x at the j-th moment under the i-th feature dimension in the test window w. ij The anomaly score is: M i (x ij ) is the sample point x under the i-th feature dimension ij The marginal distribution value of b i is the sample skewness of the i-th feature dimension in the historical window W within the inspection window T.

2. The network traffic anomaly detection method according to claim 1, characterized in that: Methods for using the VineCopula function to calculate the multivariate joint distribution between all feature dimensions of sequence samples in the inspection window T include: The marginal probability density of each feature dimension of the sequence samples in the inspection window T is calculated respectively, and the binary Copula function is further used to fit the binary joint distribution between each feature dimension. Based on the Cantor tree structure of the VineCopula function, starting from the leaf nodes of the Cantor tree structure, the binary joint distribution is constructed and merged layer by layer until the root node, thereby obtaining the multivariate joint distribution between all feature dimensions of the sequence samples in the inspection window T.

3. The network traffic anomaly detection method according to claim 1, characterized in that: The method for determining whether there is an abnormal point in the test window w based on the abnormal score matrix includes: Compare the anomaly score corresponding to each sample point in the test window w with the preset threshold. If it is greater than the threshold, the sample point is judged to be abnormal; otherwise, the sample point is judged to be normal. The method for setting the preset threshold τ includes: Obtain the minimum value min, maximum value max and average value Avg of the abnormality scores in the abnormality score matrix; The value of the preset threshold τ is increased from min to max. In this process, the probability P of the abnormal score matrix being less than τ is calculated. lower The probability P that the sum is greater than τ upper , and the mean of the anomaly scores less than τ Avg lower and the mean of the anomaly scores greater than τAvg upper , and calculate the corresponding result = P lower ×(Avg-Avg lower ) 2 +P upper ×(Avg-Avg upper ) 2 ; The maximum value of the result is used as the final preset threshold τ.

4. The network traffic anomaly detection method according to claim 1, wherein: The d features are obtained by screening all features contained in the network traffic based on correlation analysis; Specific screening methods include: The Pearson coefficient is used to measure the correlation between two network traffic features. When the correlation between two features is greater than a first preset value, only one feature is retained. Spearman is used to measure the correlation between each feature dimension and the degree of abnormality, and the obtained correlation degrees are sorted from large to small, and the first d features with higher correlation degrees are retained.

5. The network traffic anomaly detection method according to claim 1, wherein: The time window size and sliding step size of the test window w are both p; the time window size k of the history window W is an integer multiple of p.

6. The method for detecting network traffic anomaly according to any one of claims 1 to 5, characterized in that: The method further includes step S5, which specifically includes: after the abnormal point is detected, obtaining abnormal time series segments with d feature dimensions within the neighborhood of the abnormal point, and inputting them into the classification model to obtain the state category of the abnormal point; The classification model includes a cascaded feature extraction network and a classification network; the feature extraction network is an encoding network in a pre-trained SimSiam model; When an outlier is detected, abnormal time series segments with d feature dimensions within the neighborhood of the outlier are saved in a buffer pool. When the number of time series segments in the buffer pool exceeds a preset threshold, the SimSiam model is trained unsupervised. The training method of the SimSiam model includes: For each abnormal time series segment in the buffer pool, under the condition of retaining the abnormal time series segment, sample forward a constant time step on the sequence sample time axis to obtain an abnormal sample V, and sample backward the same constant time step to obtain an abnormal sample V', and obtain the corresponding positive sample pair<V,V'> ; Each positive sample pair<V,V'> Input into the SimSiam module for comparative learning; The training method of the classification model includes: inputting a pre-collected training sample set into the classification model for supervised training; the training sample set includes: time series segments with different category labels and corresponding state category labels; the state categories include normal categories and different abnormal categories.

7. The network traffic anomaly detection method according to claim 6, characterized in that: The encoding network in the SimSiam module includes: a cascaded encoding module and a projection module; the encoding module includes a cascaded three-layer encoder; the projection module includes a cascaded two-layer projection perception machine.

8. The network traffic anomaly detection method according to claim 6, characterized in that: When getting the positive sample pair<V,V'> Afterwards, the common parts of V and V' are transformed in various ways by random masking, and the data is expanded with positive samples.

9. A network traffic anomaly detection system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the network traffic anomaly detection method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the network traffic anomaly detection method according to any one of claims 1 to 8.

Images