Two-level Hash-Sketch network traffic measurement method based on time window

Abstract

The present invention discloses a two-level hash-Sketch network flow measurement method based on a time window. First, through matching filtering, a data packet that needs to be measured for flow measurement is obtained. Then, the flow characteristics of the first-level and second-level hash tables are recorded and the flow is measured, that is, the large flow separated by the two-level hash table is measured. The flow ID data packet expelled from the first-level and second-level hash tables, that is, the small flow, is measured using the existing CM-Sketch algorithm. The flow measurement mechanism of the present invention can more accurately distinguish between large flows and small flows, allowing large flows to be measured in the first-level and second-level hash tables, and small flows to be measured in CM-Sketch, thereby reducing the measurement error caused by the conflict between large and small flows, thereby realizing fine-grained measurement of network flow and improving the accuracy of measurement.

Description

Technical Field

[0001] The present invention belongs to the technical field of network management, and more specifically, relates to a two-level hash-Sketch network flow measurement method based on a time window. Background Art

[0002] With the rapid development of network technology, today's network is a complex system composed of various network devices. On top of the complex network infrastructure, various large-scale distributed services with strict QoS requirements are also built. At the same time, in order to meet the needs of different applications, the network configuration is also very complex. The traffic generated by each application is bursty and fluctuates greatly over time, which makes the traffic difficult to predict and manage. As an important part of network management, network measurement can provide network managers with a large amount of basic information and is an important reference for network management decision-making. Therefore, it is particularly important for network operators to achieve refined network measurement.

[0003] Network measurement can be divided into topology measurement, traffic measurement and performance measurement according to different measurement objects. Network topology measurement is to measure the topological relationship between routers and routers, and between routers and subnets in the network; network performance measurement refers to measuring performance indicators in the network, such as delay, packet loss rate, network bandwidth and network throughput; network traffic measurement is a network technology that counts and analyzes the number, size and distribution of network traffic on network nodes such as switches. It is an important prerequisite for network management behaviors such as analyzing network performance, understanding user behavior and detecting network security incidents. Therefore, the relevant research on network traffic measurement (especially the measurement of high-speed links) technology has received widespread attention at home and abroad. However, in high-speed networks, the massive network traffic that continues to arrive on high-speed links has brought great difficulties to the measurement and analysis of network traffic. The large number of flows and data packets on high-speed links makes traditional traffic measurement methods face the dual challenges of computing resources and memory resources. On the one hand, the high-speed memory of network nodes such as switches and routers is limited and cannot accommodate huge network data; on the other hand, the processing speed of data packets cannot match the ever-increasing port rate. This inevitably reduces the accuracy of traditional measurement methods for high-speed link measurement.

[0004] In the current research on network traffic measurement technology, there are mainly three types of network traffic measurement methods: 1) network traffic measurement method based on switch flow table; 2) network traffic measurement method based on Counter; 3) network traffic measurement method based on Sketch.

[0005] The main principle of the network traffic measurement method based on the switch flow table is to directly use the counter in the switch flow table on the data plane to count the size of each forwarding flow, and the control plane calculates the size of each flow by periodically requesting the counter value of the data plane flow table. However, the main problem faced by this method is that the number of flow tables of the switch (for example, ordinary switches usually have only a few thousand flow tables) is very limited, while the number of flows with measurement is very large, so it is impossible to measure fine-grained flows directly using the flow table.

[0006] The basic principle of the Counter-based network traffic measurement method is to use an efficient data structure (such as a heap) in the switch to maintain K counters, each of which is responsible for counting the size of a flow. Because the number of flows is usually much larger than the number of counters, the K counters can only record the information of K flows at the same time. In actual networks, there are fewer large flows, but a few large flows generate most of the network traffic. Therefore, the Counter-based traffic measurement method usually only uses limited counters to store the information of large flows. In the specific implementation process, a preemption mechanism can be used to ensure that large flows can preempt counters occupied by small flows. The Counter-based traffic measurement method can basically meet the traffic planning algorithm's demand for traffic information, but it requires a large computational overhead and is difficult to apply on high-speed data planes.

[0007] The basic principle of the Sketch-based network traffic measurement method is to use various Sketches (such as CM Sketch) to store flow statistics. Sketch is essentially a measurement method based on hash mapping. Its principle is to use a hash function to map a range of the data domain to another range. For example, CM Sketch can be regarded as a two-dimensional counter array, each row of which corresponds to a hash function, and each flow can be hashed to a counter in each row according to the ID. Based on this, a large amount of data to be measured can be mapped to a limited number of counters. However, this hash mapping will inevitably have hash conflicts, which will pollute the data stored in the counter and cause measurement errors. This is particularly obvious when hash conflicts occur between small and large flows. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a two-level hash-Sketch network traffic measurement method based on time window, which not only solves the problem of limited computing and storage resources of network nodes, but also overcomes the problem of measurement accuracy caused by the limitations of Sketch structure.

[0009] To achieve the above-mentioned object of the invention, the present invention provides a two-stage hash-Sketch network traffic measurement method based on a time window, characterized in that it includes the following steps:

[0010] (1) Match filtering

[0011] For the arriving data packets, the network node will send them to the matching filter, and the matching filter will filter the measurement data packets according to the matching filtering rules sent by the control plane;

[0012] (2) The first-level hash table records the flow measurement characteristics of the current time window and measures the flow

[0013] The filtered data packet enters the first-level hash table, and the flow ID is set as the key value of the hash table. When the data packet reaches the first-level hash table, the node position corresponding to its insertion into the hash table is calculated according to its flow ID hash;

[0014] In the first-level hash table, if the node position calculated by the hash of the packet flow ID is empty, it means that the packet is the first packet in the flow to reach the first-level hash table. The packet flow ID is directly inserted, and the timestamp and measurement characteristics in the packet are recorded;

[0015] If the node position calculated by hashing the flow ID of the data packet is not empty, the flow ID of the data packet and the flow ID of the node position calculated by hashing are matched. If they are the same, it indicates that no hash collision has occurred, and the measurement features are directly counted. If they are not the same, it indicates that a hash collision has occurred. Two sub-situations are considered according to the time window: if the difference between the timestamp cur of the data packet and the node position calculated by hashing, that is, the time t recorded at the conflicting node position, is greater than the threshold θ, that is, cur-t≥θ, it means that the next time window has been entered at this time, and the data packet flow ID and measurement features recorded at the conflicting node position are driven out to the secondary hash table for record. The data packet flow ID and timestamp cur are recorded as time t and measurement features at the conflicting node position. Conversely, if cur-t<θ, the data packet is driven out to the secondary hash table for record.

[0016] According to the measurement characteristics recorded in the first-level hash table, the flow rate of the data packet corresponding to the flow ID is obtained;

[0017] (3) The secondary hash table records all time window flow measurement features and measures the flow

[0018] For the data packets in the secondary hash table under the condition of cur-t<θ, the flow ID is hashed again to obtain the corresponding node position in the secondary hash table; if the node position calculated by the hash of the flow ID of the data packet is empty, the flow ID of the data packet is directly inserted, and the measurement characteristics in the data packet are recorded; if the node position calculated by the hash of the flow ID of the data packet is not empty, the flow ID of the data packet and the flow ID of the node position calculated by the hash are matched. If they are the same, it indicates that no hash collision occurs, and the measurement characteristics are directly counted; if they are not the same, it indicates that a hash collision occurs, and the data packet belongs to a small flow. The data packet is expelled to the CM-Sketch (Count-Min Sketch) module for flow measurement to obtain the flow of the data packet corresponding to the flow ID;

[0019] For the data packet flow ID and measurement features expelled to the secondary hash table under the condition of cur-t≥θ, the data packet flow ID is hashed again to obtain the corresponding node position in the secondary hash table; if the node position calculated by hashing the data packet flow ID is empty, the data packet flow ID and measurement features are directly inserted; if the node position calculated by hashing the data packet flow ID is not empty, the data packet flow ID and the flow ID of the node position calculated by hashing are matched; if they are the same, it indicates that no hash collision occurs; the measurement features expelled to the secondary hash table are directly added to the measurement features of the node position calculated by hashing, and the addition result is used to update the measurement features of the node position calculated by hashing; if they are not the same, it indicates that a hash collision occurs; the measurement features expelled to the secondary hash table are compared with the measurement features of the node position calculated by hashing, and the flow ID with a small measurement feature value and its measurement features are expelled to the CM-Sketch (Count-MinSketch) module for traffic measurement to obtain the traffic of the corresponding flow ID data packet; the flow ID with a large measurement feature value and its measurement features are recorded at the conflicting node position;

[0020] According to the measurement characteristics recorded in the secondary hash table, the flow rate of the data packet corresponding to the flow ID is obtained.

[0021] The object of the invention of the present invention is achieved in this way:

[0022] In order to solve the problem of limited computing and storage resources of network nodes and the problem that the measurement accuracy is difficult to meet the requirements due to the limitations of the Sketch structure, the present invention proposes a two-level hash-Sketch network traffic measurement method based on a time window. First, through matching filtering, the data packets that need to be measured are obtained, and then the first-level and second-level hash table flow characteristics are recorded and the flow is measured, that is, the large flow separated by measuring the two-level hash table is measured, and the flow ID data packets expelled from the first-level and second-level hash tables, that is, the small flow, are measured using the existing CM-Sketch algorithm. The flow measurement mechanism of the present invention can more accurately distinguish between large and small flows, allowing large flows to be measured in the first-level and second-level hash tables, and small flows to be measured in CM-Sketch, thereby reducing the measurement error caused by the conflict between large and small flows, thereby realizing fine-grained measurement of network traffic and improving measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the principle of the two-stage hash-Sketch network traffic measurement method based on time window of the present invention;

[0024] Figure 2 It is a flowchart of a specific implementation method of the two-stage hash-Sketch network traffic measurement method based on time window of the present invention;

[0025] Figure 3 It is a first-level hash representation that records the measurement characteristics of the current time window stream;

[0026] Figure 4 It is a secondary hash representation that records the cumulative values ​​of all time window flow characteristics. DETAILED DESCRIPTION

[0027] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0028] Existing network traffic measurement methods have certain limitations, and the limited computing and storage resources of network nodes restrict the selection and improvement of network traffic measurement methods, which makes it difficult for network nodes to accurately measure the flows in the network.

[0029] By analyzing the traffic trace captured in the research network, it is found that the size distribution of the flow is very close to the Zipf distribution, that is, very few large flows contribute to the vast majority of the traffic in the network. Therefore, the present invention proposes to adopt a time window mechanism to insert the flows in the same time window into the first-level hash table. When the data packet of the next time window arrives, the flow in the original first-level hash table is driven into the second-level hash table, and the flow is replaced with the packet of the new time window. Since large flows often arrive continuously in the network, and small flows are interspersed among the large flows, the time window mechanism is used to periodically divide the network data packets, which can effectively reduce the probability of large flows being expelled into Sketch by small flows, thereby reducing the measurement error caused by hash conflicts between large and small flows.

[0030] In addition, in order to measure large flows more accurately, the present invention designs a two-level hash table mechanism. Data packets that collide in the first-level hash table will not be immediately expelled to Sketch, but will be expelled to the second-level hash table. If a data packet collides in both the first and second-level hash tables, and its flow measurement characteristics (such as the number of packets) are less than the flow measurement characteristics of the original node in the second-level hash table, it will be identified as a small flow and expelled to Sketch for measurement.

[0031] Compared with the traditional network flow measurement method, the present invention has the following characteristics:

[0032] 1. Propose a time window-based mechanism to periodically count network traffic;

[0033] 2. A two-level hash-Sketch mechanism is proposed to accurately measure large flows and small flows respectively.

[0034] To achieve the above-mentioned purpose of the invention, the two-level hash-Sketch network traffic measurement method based on time window of the present invention mainly includes the following contents:

[0035] (1) A first-level hash table that records the current time window flow characteristics

[0036] (1.1) Design of the first-level hash table

[0037] (1.2) Design of time window

[0038] (2) A secondary hash table that records the cumulative values ​​of all time window flow measurement features

[0039] (2.1) Design of the secondary hash table

[0040] (2.2) Small flow expulsion design

[0041] Figure 1 It is a schematic diagram of the principle of the two-stage hash-Sketch network traffic measurement method based on time window of the present invention.

[0042] like Figure 1 The innovative part of the present invention is mainly divided into three parts, and its functions are divided as follows:

[0043] 1. Matching Filters

[0044] The data packet will first pass through the matching filter, and the filter will filter the measurement packet according to the rules sent by the control plane.

[0045] 2. Level 1 hash table

[0046] It mainly records the flow characteristics of the current time window. Its hash table mainly records the flow ID, the arrival time of the first data packet of the flow, and the characteristics of the flow, such as the number of packets.

[0047] 3. Secondary hash table

[0048] The cumulative value of the stream characteristic for all time windows to be recorded.

[0049] The specific implementation steps are as follows:

[0050] In the network traffic measurement solution, the first and second level hash tables are responsible for large flow measurement, while CM-Sketch is responsible for small flow measurement. The data packet will first pass through the matching filter, which will filter the measurement packet according to the rules issued by the control plane, for example, only measuring the flow with the source of network segment 10 or the destination of network segment 20.

[0051] Next, the data packet will enter the first-level hash table, which mainly records the flow characteristics of the current time window. The hash table mainly records the flow ID, the arrival time of the first data packet of the flow, and the measurement characteristics of the flow, such as the number of packets. Data packets that collide in the first-level hash table will be expelled to the second-level hash table, which is responsible for recording the cumulative value of the flow characteristics of all time windows. Flows that are expelled from both the first and second-level hash tables are identified as a small flow and measured using CM-Sketch.

[0052] Figure 2 It is a flowchart of a specific implementation of the two-stage hash-Sketch network traffic measurement method based on time window of the present invention.

[0053] In this embodiment, if Figure 2 As shown, the two-stage hash-Sketch network traffic measurement method based on time window of the present invention includes the following steps:

[0054] Step S1: Match filtering

[0055] For the arriving data packets, the network node will send them to the matching filter, and the matching filter will filter the measurement data packets according to the matching filtering rules sent by the control plane. Figure 1 As shown, for example, only the flow whose source is the network segment 10 (Src: 10.0.0.0 / 8) or the flow whose destination is the network segment 20 (Dst: 20.0.0.0 / 16) is measured.

[0056] Step S2: The first-level hash table records the flow measurement characteristics of the current time window and measures the flow

[0057] The filtered data packet enters the first-level hash table, and the flow ID is set as the key value of the hash table. When the data packet reaches the first-level hash table, the node position corresponding to its insertion into the hash table is calculated based on its flow ID hash. The flow ID can be a five-tuple of the flow, source and destination IP, etc.

[0058] In the first-level hash table, if the node position calculated by hashing the data packet flow ID is empty, it means that the data packet is the first data packet in the flow to reach the first-level hash table. The data packet flow ID is directly inserted, and the timestamp and measurement characteristics in the data packet are recorded. In this embodiment, the measurement characteristics recorded in the data packet are the number of packets Count value set to 1, such as Figure 3 As shown in ①.

[0059] If the node position calculated by hashing the flow ID of the data packet is not empty, the flow ID of the data packet and the flow ID of the node position calculated by hashing are matched. If they are the same, it indicates that no hash collision occurs, and the feature is directly measured by statistics. In this embodiment, the direct statistical measurement feature is the packet count Count value plus 1, such as Figure 3 As shown in ② in . If they are not the same, it indicates that a hash collision occurs. Two sub-situations are considered according to the time window: if the difference between the timestamp cur of the data packet and the node position calculated by the hash, that is, the time t recorded at the conflicting node position, is greater than the threshold θ, that is, cur-t≥θ, it means that the next time window has been entered at this time, and the data packet flow ID and measurement features recorded at the conflicting node position are driven out to the secondary hash table record, and the data packet flow ID and timestamp cur are recorded as time t and measurement features at the conflicting node position, as shown in Figure 3 As shown in ③; on the contrary, if cur-t<θ, the data packet is expelled to the secondary hash table for recording, such as Figure 3 As shown in ④.

[0060] According to the measurement characteristics recorded in the first-level hash table, the traffic of the data packet corresponding to the flow ID is obtained.

[0061] Step S3: The secondary hash table records all time window flow measurement features and measures the flow

[0062] According to the working mechanism of the first-level hash table, there are a total of two situations in which the data packet or the recorded data packet flow ID and measurement characteristics will be expelled to the second-level hash table. If a data packet collides in the first-level hash table, and the time difference between the timestamp of the data packet and the time in the original node is less than the threshold, that is, cur-t<θ, the data packet is expelled to the second-level hash table for recording. For the data packets that are in the second-level hash table under the situation of cur-t<θ, its flow ID is hashed again to obtain its corresponding node position in the second-level hash table. If the node position calculated by the hash of the data packet flow ID is empty, the data packet flow ID is directly inserted, and the measurement characteristics in the data packet are recorded. If the node position calculated by the hash of the data packet flow ID is not empty, the flow ID of the data packet and the flow ID of the node position calculated by the hash are matched. If they are the same, it indicates that no hash collision has occurred, and the measurement characteristics are directly counted. In this embodiment, the directly counted measurement characteristics are the number of packets Count value plus 1, such as Figure 4 As shown in ④, the packet count Count value is cnt_s1+1; if they are different, it indicates that a hash collision occurs and the data packet belongs to a small flow. The data packet is expelled to the CM-Sketch (Count-Min Sketch) module for flow measurement to obtain the flow of the corresponding flow ID data packet.

[0063] If in the first-level hash table, the difference between the timestamp cur of the data packet and the node position calculated by the hash, that is, the time t recorded at the conflicting node position, is greater than the threshold θ, that is, cur-t≥θ, the data packet flow ID and measurement characteristics of the conflicting node position will be expelled to the second-level hash table. For the data packet flow ID and measurement features expelled to the secondary hash table in the case of cur-t≥θ, the data packet flow ID is hashed again to obtain the corresponding node position in the secondary hash table; if the node position calculated by hashing the data packet flow ID is empty, the data packet flow ID and measurement features are directly inserted; if the node position calculated by hashing the data packet flow ID is not empty, the data packet flow ID and the flow ID of the node position calculated by hashing are matched; if they are the same, it indicates that no hash collision has occurred; the measurement features expelled to the secondary hash table are directly added to the measurement features of the node position calculated by hashing (such as the number of packets Count of the data packet plus 1), and the measurement features of the node position calculated by hashing are updated with the addition result; if they are not the same, it indicates that a hash collision has occurred; the measurement features expelled to the secondary hash table are compared with the measurement features of the node position calculated by hashing, and the flow ID with a small measurement feature value and its measurement features are expelled to the CM-Sketch (Count-Min Sketch) module for flow measurement to obtain the flow of the corresponding flow ID data packet; the flow ID with a large measurement feature value and its measurement features are recorded at the conflicting node position. In this embodiment, Figure 4As shown in ⑤, the flow ID Fk with a small measurement feature value and its measurement feature cnt_s3 are sent to the CM-Sketch module one by one, and the flow ID F1 with a large measurement feature value and its measurement feature cnt1 are recorded at the conflict node position.

[0064] According to the measurement characteristics recorded in the secondary hash table, the flow rate of the data packet corresponding to the flow ID is obtained.

[0065] like Figure 1 As shown, the CM-Sketch module performs five hash calculations on the flow ID, and the measurement features of the five node positions are 1545, 2134, 1297, 2341, and 1441. The smallest measurement feature 1297 is selected as the flow measurement. The flow measurement of the CM-Sketch module belongs to the prior art and will not be repeated here.

[0066] The flow measurement mechanism of the present invention can more accurately distinguish large flows from small flows, allowing large flows to be measured in the hash table and small flows to be measured in the CM-Sketch, thereby reducing measurement errors caused by conflicts between large and small flows and realizing a fine-grained flow measurement mechanism.

[0067] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A two-level hash-Sketch network traffic measurement method based on time window, characterized in that: The following steps are involved: (1) Match filtering For the arriving data packets, the network node will send them to the matching filter, and the matching filter will filter the measurement data packets according to the matching filtering rules sent by the control plane; (2) The first-level hash table records the flow measurement characteristics of the current time window and measures the flow The filtered data packet enters the first-level hash table, and the flow ID is set as the key value of the hash table. When the data packet reaches the first-level hash table, the node position corresponding to its insertion into the hash table is calculated according to its flow ID hash; In the first-level hash table, if the node position calculated by the hash of the packet flow ID is empty, it means that the packet is the first packet in the flow to reach the first-level hash table. The packet flow ID is directly inserted, and the timestamp and measurement characteristics in the packet are recorded; If the node position calculated by hashing the flow ID of the data packet is not empty, the flow ID of the data packet and the flow ID of the node position calculated by hashing are matched. If they are the same, it indicates that no hash collision has occurred, and the measurement features are directly counted. If they are not the same, it indicates that a hash collision has occurred. Two sub-situations are considered according to the time window: if the difference between the timestamp cur of the data packet and the node position calculated by hashing, that is, the time t recorded at the conflicting node position, is greater than the threshold θ, that is, cur-t≥θ, it means that the next time window has been entered at this time, and the data packet flow ID and measurement features recorded at the conflicting node position are driven out to the secondary hash table for record. The data packet flow ID and timestamp cur are recorded as time t and measurement features at the conflicting node position. Conversely, if cur-t<θ, the data packet is driven out to the secondary hash table for record. According to the measurement characteristics recorded in the first-level hash table, the flow rate of the data packet corresponding to the flow ID is obtained; (3) The secondary hash table records all time window flow measurement features and measures the flow For each data packet in the secondary hash table under the condition of cur-t<θ, its flow ID is hashed again to obtain its corresponding node position in the secondary hash table; if the node position calculated by hashing the flow ID of the data packet is empty, the flow ID of the data packet is directly inserted and the measurement features in the data packet are recorded; If the node position calculated by hashing the flow ID of the data packet is not empty, the flow ID of the data packet and the flow ID of the node position calculated by hashing are matched. If they are the same, it indicates that no hash collision has occurred, and the characteristics are directly measured statistically. If they are not the same, it indicates that a hash collision has occurred and the data packet belongs to a small flow. The data packet is expelled to the CM-Sketch module for flow measurement to obtain the flow of the data packet with the corresponding flow ID; For the data packet flow ID and measurement features expelled to the secondary hash table under the condition of cur-t≥θ, the data packet flow ID is hashed again to obtain the corresponding node position in the secondary hash table; if the node position calculated by hashing the data packet flow ID is empty, the data packet flow ID and measurement features are directly inserted; if the node position calculated by hashing the data packet flow ID is not empty, the data packet flow ID and the flow ID of the node position calculated by hashing are matched; if they are the same, it indicates that no hash collision has occurred; the measurement features expelled to the secondary hash table are directly added to the measurement features of the node position calculated by hashing, and the addition result is used to update the measurement features of the node position calculated by hashing; if they are not the same, it indicates that a hash collision has occurred; the measurement features expelled to the secondary hash table are compared with the measurement features of the node position calculated by hashing, the flow ID with a small measurement feature value and its measurement features are expelled to the CM-Sketch module for traffic measurement, the traffic of the corresponding flow ID data packet is obtained, and the flow ID with a large measurement feature value and its measurement features are recorded at the conflicting node position; According to the measurement characteristics recorded in the secondary hash table, the flow rate of the data packet corresponding to the flow ID is obtained.

2. The two-stage hash-Sketch network traffic measurement method based on time window according to claim 1 is characterized in that: The measurement feature is the number of packets in the data packet.

Images