A network traffic concept drift detection method based on count-min sketch data structure

Abstract

The application relates to a network traffic concept drift detection method based on a Count-Min sketch data structure and belongs to the network traffic analysis field. The application records the multidimensional statistical information of network traffic through a CM sketch data structure, starts from the multidimensional probability distribution of network flow, monitors the multidimensional Hellinger distance change condition every certain period, performs network traffic concept drift detection, and detects the type of network traffic concept drift based on the Euclidean distance. The application records the multidimensional statistical information of network traffic through a CM sketch data structure, saves the storage space, each dimension is relatively independent, can be processed in parallel, and saves the detection time; starts from the multidimensional probability distribution of network flow, monitors the multidimensional Hellinger distance change condition, performs network traffic concept drift detection, reduces the concept drift false detection rate and the missed detection rate, makes the detection result more accurate; can correctly identify the network traffic concept drift type, discovers new applications and distributed drift applications, and has important significance in network intrusion detection and the like.

Description

A Network Traffic Concept Drift Detection Method Based on Count-Min Summary Data Structure Technical Field

[0001] This invention belongs to the field of network traffic analysis, specifically relating to a method for detecting network traffic concept drift based on a Count-Min summary data structure. Background Technology

[0002] Unlike traditional static data, internet traffic is characterized by its massive volume, dynamism, and real-time nature, posing significant challenges to traffic classification. When network traffic experiences concept drift, it leads to a decline in classifier performance and classification accuracy. Therefore, it is necessary to detect whether network traffic has experienced concept drift, and if so, update the classification model to maintain classifier stability.

[0003] Currently, many classification methods for concept drift in data streams have been proposed. The first type of method constructs a classifier that can automatically adapt to changes in the data stream. This type of method updates the classifier incrementally and cannot detect concept shifts occurring within the data stream. The second type of method separates the data stream concept drift detector and the classification model. When the concept drift detector detects changes in the data stream, the classification model is then updated. The second type of method can identify the timing of concept drift in the data stream and data change points, providing supporting data for subsequent work.

[0004] The second category of concept drift detection methods also includes two different drift detection approaches. The first method determines whether drift has occurred based on the classifier's classification results, including classification metrics such as error rate, accuracy, recall, and precision. This type of method is prone to over-reliance on classification results, potentially leading to misjudgments and missed detections of concept drift. The second method starts with the probability distribution of the data stream, detecting changes in the data stream's probability distribution to determine whether concept drift has occurred.

[0005] Concept drift detection methods based on data flow distribution are well-suited for detecting concept drift in network traffic. They determine whether concept drift has occurred by detecting changes in the distribution of different dimensional attributes of network traffic. However, existing methods cannot detect the type of concept drift in network traffic. Therefore, to meet the requirements of real-time, high-dimensional traffic data concept drift detection under high-speed network conditions, this invention uses a summary data structure to store statistical information on the attribute characteristics of network traffic. On this data structure, changes in the Hellinger distance of each dimension of network traffic are monitored to determine whether concept drift has occurred. After detecting network traffic concept drift, the type of concept drift is further investigated. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] The technical problem to be solved by this invention is how to provide a network traffic concept drift detection method based on the Count-Min summary data structure, so as to solve the problem that existing methods cannot detect the type of network traffic concept drift.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention proposes a network traffic concept drift detection method based on a Count-Min summary data structure, which includes the following steps:

[0010] Step 1: Use the Count-Min (CM) summary data structure to store the multidimensional feature information of network traffic;

[0011] Step 2: Calculate the Hellinger distance for each dimension of network traffic on the two data summary structures to determine whether concept drift has occurred;

[0012] Step 3: Calculate the network traffic concept drift threshold;

[0013] Step 4: Identify network traffic concept drift types, including network application category drift detection and network flow distribution drift detection.

[0014] (III) Beneficial Effects

[0015] This invention proposes a network traffic concept drift detection method based on the Count-Min summary data structure. The network traffic concept drift detection method based on the Count-Min summary data structure of this invention has the following advantages:

[0016] 1) By recording multidimensional statistical information of network traffic through the CM summary data structure, not only is storage space saved, but each dimension is also relatively independent and can be processed in parallel, saving detection time.

[0017] 2) Starting from the multidimensional probability distribution of network flow, we monitor the changes in multidimensional Hellinger distance and perform network traffic concept drift detection, which reduces the false detection rate and false negative rate of concept drift, making the detection results more accurate.

[0018] 3) It can correctly identify network traffic concept drift types, discover new applications and applications with distributed drift, which is of great significance in network intrusion detection and other aspects. Attached Figure Description

[0019] Figure 1 is a flowchart of the network traffic concept drift detection process of the present invention;

[0020] Figure 2 shows the summary data structure of the multidimensional Count-Min. Detailed Implementation

[0021] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0022] The main objective of this invention is to detect concept drift in network traffic, enabling timely updates to the network traffic classification model and improving classification accuracy. This invention records multidimensional statistical information of network traffic using a Count-Min (CM) summary data structure. Then, starting with the multidimensional probability distribution of network flows, it monitors changes in multidimensional Hellinger distance at regular intervals to detect network traffic concept drift. Furthermore, it detects the type of network traffic concept drift based on Euclidean distance, providing data support for network management and maintenance.

[0023] This invention presents a network traffic concept drift detection method based on the CM summary data structure. First, it detects whether concept drift occurs in the data stream. If it does, it further identifies the type of concept drift. The detection process is shown in Figure 1. Network traffic concept drift detection includes the following steps:

[0024] Step 1: Use the CM summary data structure to store multi-dimensional feature information of network traffic;

[0025] Step 2: Calculate the Hellinger distance for each dimension of network traffic on the two data summary structures to determine whether concept drift has occurred;

[0026] Step 3: Calculate the network traffic concept drift threshold;

[0027] Step 4: Identify network traffic concept drift types, including network application category drift detection and network flow distribution drift detection.

[0028] Furthermore, the specific process of step one is as follows:

[0029] CM consists of multiple hash functions (h1, h2, ... h... d The system consists of a hash function and a two-dimensional table, where each hash function corresponds to a row in the table. When a network flow arrives, it needs to be processed by each hash function, and the calculated hash value is stored in a counter corresponding to the hash value in each row. When querying the feature statistics of the network flow, the minimum value in the counter is taken as the estimated value of that feature statistic.

[0030] Furthermore, the specific process of step two is as follows:

[0031] Calculate the Hellinger distance for each dimension of network traffic on two summary data structures, denoted as H. kk represents the number of distinct dimensional elements in network traffic. A network flow is composed of different features, such as port number, source IP, destination IP, etc., and k refers to the number of feature dimensions. Hellinger distance is a metric that measures the difference between two probability distributions. The larger the Hellinger distance, the greater the difference in data distribution. k If the threshold is exceeded, it is considered that concept drift has occurred. Let P = (p1, p2, ..., p... n ) and Q = (q1, q2, ..., q n Let x and y represent the probability distributions of network traffic in the k-dimensional dimension, respectively. The Hellinger distance between x and y in the k-dimensional dimension can be defined by formula (1):

[0032]

[0033] Among them, H k p represents the Hellinger distance in the k-dimensional dimension. i p represents the probability of taking the value i in the k-th dimension of network traffic x. i =v i / m,v i =min d CM[d,h(i)], where m is the total number of network traffic x. i Let q be the probability of taking the value i in the k-th dimension of network traffic y. i =v i / m', v i =min d CM[d,h(i)], where m' is the total number of network traffic y. Here, d is the number of hash functions, and h(i) is the hash function value.

[0034] If H k >σ k Then it is considered that network traffic has undergone conceptual drift in k-dimensional features, σ k The threshold set for the conceptual drift of network traffic.

[0035] Furthermore, the specific process of step three is as follows:

[0036] This invention employs the Bootstrap method to estimate the threshold in the network traffic concept drift detection process. It constructs bootstrap samples through repeated sampling, calculates a given statistic based on the extracted sample data, and obtains the variance of the overall data statistic from the sample variance of the statistic. The specific steps are as follows:

[0037] S31: Establish the concept drift null hypothesis for network traffic x and sampled dataset x', H0: P(x) = P'(x'), that is, x and x' have the same probability distribution;

[0038] S32: Sample s datasets from network traffic x using the Bootstrap method, resulting in (x1, x2, ..., x...) s Each dataset contains t network flow sample data;

[0039] S33: Calculate the Hellinger distance between the original dataset and each sampled dataset to obtain the corresponding Hellinger distance set H = (H1, H2, ..., H...). s );

[0040] S34: Sort the data in H from largest to smallest, and find the critical interval [H]. λ ,∞), 1≤λ≤s, when the threshold is greater than H λ At that time, the null hypothesis H0 does not hold, and network traffic undergoes conceptual drift.

[0041] Furthermore, step four is further divided into network application category drift detection and network flow distribution drift detection. The specific process of network application category drift detection is as follows:

[0042] S41: First, detect the attribute dimension that caused the concept drift according to the method in step S2, and then trace back the original data y' that caused the concept drift; y' is the data of the attribute dimension of y that caused the concept drift.

[0043] S42: Define the λ of y' c,q , represents the q nearest neighbors of y' in category C (network traffic includes different categories, such as email, audio, video, etc.), i.e. the q network traffic with the smallest Euclidean distance to y', where n' is the number of network traffic in y'. Representing y' and λ c,q The average distance is described by formula (2):

[0044]

[0045] Where, D(λ', y) j ) represents λ' and y j The Euclidean distance between them.

[0046] S43: Calculate y' in all categories This represents the minimum value, where the network application category is C. min If y' is a new application category C new Then the Euclidean distance between network flows in this new application category is smaller than the distance between network flows in any other existing category, i.e. Mark y' as class C new .

[0047] The specific process of network flow distribution drift detection is as follows:

[0048] S44: This invention considers network traffic concept drift to be network flow distribution drift if it is not application category drift, and simultaneously detects the network application experiencing flow distribution drift. If y' is not a new application, it is considered network application C. min Let P y＇，k and Q Cmin,k Representing y' and C respectively min Given a probability distribution in k dimensions, calculate the Hellinger distance H'.

[0049] S45: If H' > σ k Then it is believed that C min The attribute features in the k-dimensional dimension have undergone distribution drift.

[0050] In summary, the network traffic concept drift detection method based on the CM summary data structure of this invention has the following advantages: 1) Recording multidimensional statistical information of network traffic through the CM summary data structure not only saves storage space, but also allows for parallel processing of each dimension, saving detection time. 2) By monitoring the changes in multidimensional Hellinger distance from the multidimensional probability distribution of network flows, network traffic concept drift detection is performed, reducing the false detection rate and false negative rate of concept drift, resulting in more accurate detection results. 3) It can correctly identify the type of network traffic concept drift and discover new applications and applications of distribution drift, which is of great significance in network intrusion detection and other fields.

[0051] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting network traffic concept drift based on a Count-Min summary data structure, characterized in that, The method includes the following steps: Step 1: Storing multidimensional feature information of network traffic using a Count-Min (CM) summary data structure; Step 2: Calculating the Hellinger distance of each dimension of network traffic on two summary data structures to determine whether concept drift has occurred; Step 3: Calculating the concept drift threshold of network traffic; Step 4: Identifying the type of concept drift in network traffic, including network application category drift detection and network flow distribution drift detection; wherein, in Step 4, the specific process of network application category drift detection is as follows: S41: First, detect the attribute dimension that has caused concept drift according to the method in Step 2, and then trace back the original data that caused the concept drift. ； It refers to the data of the attribute dimension where the concept of network traffic y is shifted; S42: Definition of This indicates that in category C, The q nearest neighbors, for Number of network flows in China express and The average distance is described by formula (2): (2) Among them, express and S43: Calculate the Euclidean distance between them; In all categories ， This represents the minimum value, where the network application category is C. min ;if For a new application category C new Then the Euclidean distance between network flows in this new application category is smaller than the distance between network flows in any other existing category, i.e. ,Will Marked as class C new In step four, the specific process of network flow distribution drift detection is as follows: S44: If the network traffic concept drift is not an application category drift, then it is a network flow distribution drift, and the network application that has experienced the flow distribution drift is detected; if it is not a new application, it is considered to be network application C. min Let P y＇，k and Q Cmin,k They represent and C min Given a probability distribution in k dimensions, calculate the Hellinger distance H'; S45: If Then it is believed that C min The attribute features in the k-dimensional dimension exhibited a distribution shift, where, The threshold set for the conceptual drift of network traffic.

2. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 1, characterized in that, Step one specifically includes: CM consists of multiple hash functions (h1, h2, ... h... d It consists of a hash function and a two-dimensional table, where each hash function corresponds to each row in the table. When a network flow arrives, it needs to be calculated by each hash function, and the hash value obtained is stored in the counter corresponding to the hash value in each row. When querying the feature statistics of the network flow, the minimum value in the counter is taken as the estimated value of the feature statistics.

3. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 2, characterized in that, Step two specifically includes: calculating the Hellinger distance for each dimension of network traffic on two summary data structures, denoted as... k represents the number of distinct dimensional elements in network traffic; Hellinger distance is a metric for measuring the difference between two probability distributions. The larger the Hellinger distance, the greater the difference in data distribution. If the threshold is exceeded, it is considered that concept drift has occurred; let P = (p1, p2, ..., p...). n ) and Q = (q1, q2, ..., q n Let x and y represent the probability distributions of network traffic in the k-dimensional space, respectively. The Hellinger distance between x and y in the k-dimensional space is defined by formula (1): (1) Among them, Represents the Hellinger distance in the k-dimensional dimension. Let x be the probability of taking the value i in the k-th dimension of network traffic x. ， , where m is the total amount of network traffic x; Let y be the probability of taking the value i in the k-th dimension of network traffic y. ， ， Let be the total network traffic y; where, It is the number of hash functions. It is the hash function value.

4. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 3, characterized in that, if If so, it is believed that network traffic has undergone conceptual drift in k-dimensional features.

5. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 3, characterized in that, In step three, the Bootstrap method is used to estimate the threshold in the network traffic concept drift detection process. A bootstrap sample is constructed by repeated sampling. The given statistic is calculated based on the sample data. The variance of the overall data statistic is obtained based on the sample variance of the statistic.

6. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 3, characterized in that, Step three specifically includes the following steps: S31: For network traffic x and sampled datasets... Establish the null hypothesis of concept drift. That is, x and The probability distributions are the same; S32: Sample s datasets from network traffic x using the Bootstrap method, which are (x1, x2, ..., x...). s Each dataset contains t network flow samples; S33: Calculate the Hellinger distance between the original dataset and each sampled dataset to obtain the corresponding Hellinger distance set H = (H1, H2, ..., H...). s S34: Sort the data in H from largest to smallest and find the critical interval. ,∞)，1≤ ≤s, when the threshold is greater than H λ At that time, the null hypothesis H0 does not hold, and network traffic undergoes conceptual drift.

7. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 1, characterized in that, Categories of network traffic include: email, audio, and video.

8. The network traffic concept drift detection method based on the Count-Min summary data structure as described in claim 1, characterized in that, In step two, the Hellinger distance changes are monitored at regular intervals to detect network traffic concept drift.

Images