Fine-grained network measurement structure based on funnel type Sketch and measurement algorithm thereof

Abstract

The invention discloses a fine-grained network measurement structure based on funnel type Sketch and a measurement algorithm thereof. The fine-grained network measurement structure comprises a top layer structure, a middle layer structure and a bottom layer structure. The top layer structure is composed of a two-dimensional counter array, small counters are adopted, each row corresponds to an independent hash function, and the top layer structure is used for recording and filtering mouse flow. The middle layer structure is composed of a two-dimensional array, and each barrel comprises two counter arrays and a key value pair. Wherein the two counter arrays adopt counters with different sizes and are used for dividing the network flow more finely, and the key value pair is used for recording the flow identifier of the elephant flow and the flow size corresponding to the flow identifier of the elephant flow. The bottom layer structure is composed of a hash table and used for recording the flow overflowing from the middle layer structure. According to the method, aiming at the network flow with deflected distribution, the measurement precision of the mouse flow can be improved on the premise of ensuring the detection accuracy of the elephant flow, and rapid network flow insertion and query are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of network measurement, and particularly relates to a network measurement structure based on a funnel-shaped Sketch and its measurement algorithm. Background Art

[0002] With the continuous expansion of the scale of the Internet and the continuous emergence of new services and new demands such as holographic communication, sensory interconnection, intelligent interaction, and communication perception, the network traffic in today's Internet has increased sharply. The massive network traffic has brought huge challenges to network management and network monitoring, and many research institutions at home and abroad are actively exploring high-performance network traffic measurement methods.

[0003] In order to ensure the line-speed processing of network packets, the network measurement structure must be deployed in on-chip memory to improve the measurement speed. However, the storage space of on-chip memory is very limited, so the network measurement structure needs to achieve low storage consumption. As a probabilistic data structure, Sketch has been widely applied to the research in the field of network measurement due to its characteristics of compact storage and efficient measurement.

[0004] Regarding the network measurement problem based on Sketch, the current research ideas mainly include traditional Sketch-based methods, large and small flow separation methods, and hierarchical counter methods. Among them, the traditional Sketch-based method will cause serious memory waste when dealing with skewed network traffic; the large and small flow separation method can achieve accurate elephant flow detection, but it is difficult to accurately measure mouse flows; the hierarchical counter method requires multiple hash mappings and memory accesses, so its throughput is difficult to support the rapid processing of large-scale network traffic.

[0005] In summary, when facing skewed network traffic, the current main research results are difficult to accurately estimate mouse flows and achieve fast traffic measurement at the same time. Therefore, it is urgent to design a more efficient and fine-grained network measurement structure and measurement algorithm to achieve fast measurement and accurate estimation of network traffic. Summary of the Invention

[0006] In view of the above-mentioned prior art, the present invention designs a fine-grained network measurement structure based on a funnel-shaped Sketch and its measurement algorithm. This structure can improve the measurement accuracy of mouse flows while ensuring the detection accuracy of elephant flows, and its measurement algorithm can achieve fast traffic measurement for skewed network traffic.

[0007] To solve the above technical problems, a fine-grained network measurement structure based on a funnel-shaped Sketch proposed by the present invention includes a top layer structure, a middle layer structure, and a bottom layer structure. Among them:

[0008] The top - layer structure is composed of a two - dimensional counter array. It uses smaller counters and each row corresponds to an independent hash function, which is used to record and filter the mouse flow.

[0009] The middle - layer structure is composed of a two - dimensional array. Each bucket in the array contains two one - dimensional counter arrays, denoted as A1 and A2, which are used to divide the traffic more precisely. It also contains a key - value pair, denoted as (k, v), which is used to store the flow identifier of the elephant flow and its corresponding traffic size.

[0010] The bottom - layer structure is composed of a hash table, which is used to record the elephant flows that overflow from the middle - layer structure.

[0011] In the present invention, a measurement algorithm for the above - mentioned funnel - type Sketch - based network measurement structure is also proposed, which mainly includes inserting data packets into the measurement structure and querying the traffic size after inserting the data packets. The specific insertion steps are as follows:

[0012] Step 1: Insert the data packet into the top - layer structure, including the following steps:

[0013] Step 1 - 1: All arriving data packets will be mapped to the corresponding counters in the top - layer structure through a hash function according to their flow identifiers (source IP address and destination IP address);

[0014] Step 1 - 2: Determine whether the minimum value of the counter overflows. If there is no overflow, the minimum counter performs an increment operation and the insertion ends. Otherwise, execute Step 2;

[0015] Step 2: Insert the data packet into the A1 counter array using the hash value. Determine whether the minimum value of the counter overflows. If there is no overflow, the minimum counter performs an increment operation and the insertion ends. Otherwise, execute Step 3;

[0016] Step 3: Insert the data packet into the key - value pair in the middle - layer structure, and sequentially access the keys in the corresponding bucket of the middle - layer structure, including the following steps:

[0017] Step 3 - 1: If the key is empty, insert the flow identifier of the data packet into the key and perform an increment operation on the value, and the insertion ends.

[0018] Step 3 - 2: If the key is not empty and the flow identifier stored in the key matches the flow identifier of the data packet. At this time, determine whether the value overflows. If there is no overflow, perform an increment operation on the value in the key - value pair and the insertion ends. Otherwise, execute Step 6;

[0019] Step 3 - 3: If the key is not empty and the key does not match the flow identifier of the data packet. Then access the key in the corresponding bucket of the next row and re - execute Steps 3 - 1, 3 - 2, and 3 - 3;

[0020] Step 3-4: If all the keys in the bucket corresponding to the middle layer structure are not empty and the key does not match the flow identifier of the data packet, then execute Step Four;

[0021] Step Four: Insert the data packet into the corresponding counter in bucket A2. Determine whether the minimum value of all counters overflows. If no overflow occurs, increment the minimum counter by one and execute Step Five. Otherwise, execute Step Six;

[0022] Step Five: Perform a flow replacement in the middle layer structure. Determine the ratio of the minimum value of the counters in all A2s to the minimum value of the key-value pairs in the corresponding bucket. If the ratio is greater than the preset threshold, replace the flow identifier of the data packet with the key-value pair with the smallest replacement value among the minimum values of the counters in A2, and re-insert the original key-value pair into the corresponding A2 counter array. After the insertion is completed. Otherwise, end the insertion.

[0023] Step Six: Insert the data packet into the hash table in the bottom layer structure, map it to the corresponding counter using the hash value corresponding to the first row in the top layer structure, and increment the counter by one. After the insertion is completed.

[0024] The specific query steps are as follows:

[0025] Step One: Query the flow in the top layer structure, including the following steps:

[0026] Step 1-1: Map it to the corresponding counter in the top layer structure through a hash function according to the flow identifier;

[0027] Step 1-2: Determine whether the minimum value of all mapped counters overflows. If no overflow occurs, directly return the minimum value of the counter and end the query. Otherwise, execute Step Two;

[0028] Step Two: Map the flow to the corresponding counter in the A1 counter array in the bucket using the hash value. Determine whether the minimum value of all counters overflows. If no overflow occurs, directly return the sum of the minimum value of the A1 counter and the maximum value that the counter in the top layer structure can record, and end the query. Otherwise, execute Step Three.

[0029] Step Three: Query the traffic in the key-value pairs in the middle layer structure, and sequentially access the keys in the bucket corresponding to the middle layer structure, including the following steps:

[0030] Step 3-1: If the key matches the flow identifier of the data packet. At this time, determine whether the value overflows. If there is no overflow, directly return the sum of the value in the key-value pair, the maximum value that the A1 counter can record, and the maximum value that the top-level structure counter can record, and the query ends. Otherwise, execute Step Five, return the sum of the bottom-level structure hash table counter value, the maximum value that the key-value pair can record, the maximum value that the A1 counter can record, and the maximum value that the top-level structure counter can record, and the query ends;

[0031] Step 3-2: If the key does not match the flow identifier of the data packet. Then access the key in the corresponding bucket of the next line and re-execute Steps 3-1 and 3-2;

[0032] Step Four: Query this traffic in the A2 counter array in the middle-level structure. Determine whether the minimum value of all counters reaches the maximum value that the counter can record. If it does not reach the maximum value, directly return the sum of the minimum value of the A2 counter, the maximum value that the A2 counter can record, and the maximum value that the counter in the top-level structure can record, and the query ends. If it has reached the maximum value, execute Step Five, return the sum of the bottom-level structure hash table counter value, the maximum value that the A1 counter can record, the maximum value that the A2 counter can record, and the maximum value that the top-level structure counter can record, and the query ends;

[0033] Step Five: Query this traffic in the bottom-level structure, map it to the corresponding counter using the hash value corresponding to the first line of the data packet in the top-level structure, and return the counter value.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] The fine-grained network measurement structure based on the funnel-shaped Sketch and its measurement algorithm of the present invention are deployed and tested on a small workstation configured with an Intel(R) Core(TM) i9-10920X CPU of 3.50GHz and 64GB. In the experiment, an insertion and query test is carried out using a data set containing about 1.7 million network flows. The experimental results show that in a 250KB memory space, compared with the traditional CM Sketch, the average absolute error of the measurement of this structure is reduced by 90.93%. In a 500KB memory space, compared with the existing better-performing Elastic Sketch, the average relative error of the measurement is reduced by 85.67%. For the lookup speed, compared with the Diamond Sketch based on hierarchical counters, the insertion throughput of this structure is increased by 30%. Therefore, it is practical in actual applications. This shows that the fine-grained network measurement structure based on the funnel-shaped Sketch and its measurement algorithm designed in the present invention can improve the storage efficiency and measurement accuracy while ensuring the measurement speed, and has good comprehensive performance. Brief Description of the Drawings

[0036] Figure 1 It is a design diagram of a fine-grained network measurement structure based on a funnel-shaped Sketch in the present invention;

[0037] Figure 2 It is a flow chart of the insertion algorithm of the network measurement algorithm of the present invention;

[0038] Figure 3 It is a flow chart of the query algorithm of the network measurement algorithm of the present invention;

[0039] Figure 4 It is an example diagram of the network measurement insertion and query algorithms of the present invention; Detailed Embodiment

[0040] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means any limitation to the present invention.

[0041] In the present invention, a fine-grained network measurement structure based on a funnel-shaped Sketch is designed, as Figure 1 shown. The measurement structure includes a top layer structure, a middle layer structure, and a bottom layer structure. The top layer structure is composed of a two-dimensional counter array, which uses smaller counters and each row corresponds to an independent hash function, and is used to record and filter mouse flows. The middle layer structure is composed of a two-dimensional array, and each bucket in the array contains two one-dimensional counter arrays, denoted as A1 and A2, which are used to more finely divide the traffic. At the same time, it contains a key-value pair, which is used to store the flow identifier of the elephant flow and its corresponding traffic size. The bottom layer structure is composed of a hash table, which is used to record the elephant flows overflowed from the middle layer structure.

[0042] The insertion flow chart of this measurement structure is as Figure 2 shown. The specific steps are as follows:

[0043] Step 1: Insert the data packet into the top layer structure, including the following steps:

[0044] Step 1-1: All arriving data packets will be mapped to the corresponding counters in the top layer structure through a hash function according to their flow identifiers (source IP address and destination IP address);

[0045] Step 1-2: Determine whether the minimum value of the counter overflows. If there is no overflow, the minimum counter performs an increment operation, and the insertion ends. Otherwise, go to Step 2;

[0046] Step 2: Insert the data packet into the A1 counter array using the hash value. Determine whether the minimum value of the counter overflows. If there is no overflow, the minimum counter performs an increment operation, and the insertion ends. Otherwise, go to Step 3;

[0047] Step 3: Insert the data packet into the key-value pair in the middle layer structure, and sequentially access the keys in the bucket corresponding to the middle layer structure, including the following steps:

[0048] Step 3-1: If the key is empty, insert the flow identifier of the data packet into the key, increment the value by one, and the insertion ends.

[0049] Step 3-2: If the key is not empty and the flow identifier stored in the key matches the flow identifier of the data packet. At this time, determine whether the value overflows. If no overflow occurs, increment the value in the key-value pair by one, and the insertion ends. Otherwise, execute Step 6;

[0050] Step 3-3: If the key is not empty and the key does not match the flow identifier of the data packet. Then access the key in the corresponding bucket in the next row, and re-execute Steps 3-1, 3-2, and 3-3;

[0051] Step 3-4: If all the keys in the bucket corresponding to the middle layer structure are not empty and the keys do not match the flow identifier of the data packet, then execute Step 4;

[0052] Step 4: Insert the data packet into the corresponding counter in bucket A2. Determine whether the minimum value of all counters overflows. If no overflow occurs, increment the minimum counter by one, and execute Step 5. Otherwise, execute Step 6;

[0053] Step 5: Perform flow replacement in the middle layer structure. Determine the ratio of the minimum value of the counters in all A2s to the minimum value of the key-value pairs in the corresponding bucket. If the ratio is greater than the preset threshold, replace the flow identifier of the data packet with the key-value pair with the minimum replacement value in the minimum value of the counters in A2, and re-insert the original key-value pair into the corresponding A2 counter array, and the insertion ends. Otherwise, end the insertion.

[0054] Step 6: Insert the data packet into the hash table in the bottom layer structure, map it to the corresponding counter using the hash value corresponding to the first row of the data packet in the top layer structure, and increment the counter by one, and the insertion ends.

[0055] The query flow block diagram of this measurement structure is as Figure 3 shown. The specific steps are as follows:

[0056] Step 1: Query the flow in the top layer structure, including the following steps:

[0057] Step 1-1: Map it to the corresponding counter in the top layer structure through the hash function according to the flow identifier;

[0058] Step 1-2: Determine whether the minimum value of all mapped counters overflows. If no overflow occurs, directly return the minimum value of the counter, and the query ends. Otherwise, execute Step 2;

[0059] Step 2: Map the flow to the corresponding counter in the A1 counter array in the bucket using the hash value. Determine whether the minimum value of all counters overflows. If there is no overflow, directly return the sum of the minimum value of the A1 counter and the maximum value that the counter in the top-level structure can record, and the query ends. Otherwise, execute Step 3.

[0060] Step 3: Query this traffic in the key-value pairs in the middle-level structure, and sequentially access the keys in the corresponding bucket of the middle-level structure, including the following steps:

[0061] Step 3-1: If the key matches the flow identifier of this data packet. At this time, determine whether the value overflows. If there is no overflow, directly return the value in this key-value pair, the sum of the maximum value that the A1 counter can record and the maximum value that the counter in the top-level structure can record, and the query ends. Otherwise, execute Step 5, return the sum of the counter value in the bottom-level structure hash table, the maximum value that the key-value pair can record, the maximum value that the A1 counter can record and the maximum value that the counter in the top-level structure can record, and the query ends;

[0062] Step 3-2: If the key does not match the flow identifier of this data packet. Then access the key in the next line of the corresponding bucket, and re-execute Steps 3-1 and 3-2;

[0063] Step 4: Query this traffic in the A2 counter array in the middle-level structure. Determine whether the minimum value of all counters reaches the maximum value that the counter can record. If it does not reach the maximum value, directly return the sum of the minimum value of the A2 counter, the maximum value that the A2 counter can record and the maximum value that the counter in the top-level structure can record, and the query ends. If it has reached the maximum value, execute Step 5, return the sum of the counter value in the bottom-level structure hash table, the maximum value that the A1 counter can record, the maximum value that the A2 counter can record and the maximum value that the counter in the top-level structure can record, and the query ends;

[0064] Step 5: Query this traffic in the bottom-level structure, map it to the corresponding counter using the hash value corresponding to the first line of the data packet in the top-level structure, and return the counter value.

[0065] Embodiment:

[0066] In the present invention, the designed network measurement insertion and query algorithm examples are as Figure 4As shown, among them, the maximum values that the top - level structure counter, A1, A2, and the key - value pair can record are 15, 255, 255, and 255 respectively; the replacement threshold is 2. For data packet f1, it is first inserted into the top - level structure through a hash function. The minimum counter is 5 and there is no overflow. At this time, an increment operation is performed on it. For data packets f2, f3, f4, and f5, the minimum counter in the top - level structure overflows, so they are inserted into A1 using the hash value. For f2, the minimum counter is 50 and there is no overflow. At this time, an increment operation is performed on it. For f3, f4, and f5, the minimum counter is 255 and there is an overflow, so they are inserted into the key - value pair. For f3, the key is empty, so it is inserted into the key - value pair and the value is incremented by 1. For f4, the key matches and the value is 255 with an overflow, so it is inserted into A2. The minimum counter is 59 and there is no overflow, and an increment operation is performed. At this time, the ratio of the minimum counter 60 to the minimum value 30 of the key - value pair is 2, which meets the replacement condition. Therefore, the key - value pair is replaced, and the original key - value pair is re - inserted into A2. For the query algorithm, the return value of flow f1 is 6; the return value of flow f2 is 15 + 51; the return value of flow f3 is 15 + 255 + 30; the return value of flow f4 is 33 + 255 + 255 + 15; the return value of flow f5 is 60 + 255 + 15.

[0067] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above - mentioned specific embodiments. The above - mentioned specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many variations without departing from the purpose of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A fine-grained network measurement structure based on a funnel-shaped sketch, characterized in that It includes a top - layer structure, a middle - layer structure, and a bottom - layer structure: The top - layer structure is composed of a two - dimensional counter array. It uses smaller counters and each row corresponds to an independent hash function, which is used to record and filter the mouse flow. The middle - layer structure is composed of a two - dimensional array. Each bucket in the array contains two one - dimensional counter arrays, denoted as A1 and A2, which are used to divide the traffic more precisely. It also includes a key - value pair, which is used to store the flow identifier of the elephant flow and its corresponding traffic size. The bottom - layer structure is composed of a hash table, which is used to record the elephant flows overflowed from the middle - layer structure.

2. An insertion algorithm for the fine - grained network measurement structure based on the funnel - type Sketch according to claim 1, and the specific steps are as follows: Step 1: Insert the data packet into the top - layer structure, including the following steps: Step 1 - 1: All arriving data packets will be mapped to the two - dimensional counter array of the top - layer structure through the hash function corresponding to each row according to their flow identifiers. Step 1 - 2: Determine whether the minimum value of all mapped counters reaches the maximum value that the counter can record. If it does not reach the maximum value, increment all counters corresponding to the minimum value by one, and the insertion ends. If it has reached the maximum value, execute Step 2. Step 2: Insert the data packet into the A1 counter array in the middle - layer structure, including the following steps: Step 2 - 1: Use the hash value obtained from the top - layer structure to map the data packet to the buckets corresponding to each row in the middle - layer structure. Step 2 - 2: Use the hash value to map the data packet to the corresponding counter in A1 in the bucket. Step 2 - 3: Determine whether the minimum value of all counters reaches the maximum value that the counter can record. If it does not reach the maximum value, increment all counters corresponding to the minimum value by one, and the insertion ends. If it has reached the maximum value, then execute Step 3. Step 3: Insert the data packet into the key - value pair in the middle - layer structure, and access the key in the bucket corresponding to the first row of the middle - layer structure, including the following steps: Step 3 - 1: If the key is empty, insert the flow identifier of the data packet into the key and increment the value by one, and the insertion ends. Step 3 - 2: If the key is not empty and the flow identifier stored in the key matches the flow identifier of the data packet. At this time, determine whether the value reaches the maximum value that the count can record. If it does not reach the maximum value, increment the value in the key - value pair by one, and the insertion ends. If it has reached the maximum value, execute Step 6. Step 3 - 3: If the key is not empty and the flow identifier stored in the key does not match the flow identifier of the data packet. Then access the key in the bucket corresponding to the next row and re - execute Steps 3 - 1, 3 - 2, and 3 - 3. Step 3 - 4: If all keys in the corresponding bucket of the middle - layer structure are not empty and the flow identifiers stored in the keys do not match the flow identifier of the data packet, then execute Step 4. Step 4: Insert the data packet into the A2 counter array in the middle - layer structure, including the following steps: Step 4 - 1: Use the hash value to map the data packet to the corresponding counter in the A2 counter array in the bucket. Step 4-2: Determine whether the minimum value of all counters reaches the maximum value that the counter can record. If it does not reach the maximum value, increment by one all the counters corresponding to the minimum values, and execute Step Five. If it has reached the maximum value, then execute Step Six; Step Five: Perform flow replacement in the middle layer structure. It includes the following steps: Step 5-1: Determine the ratio of the minimum value of the counters in all A2s to the minimum value of the key-value pairs in the corresponding buckets. If this ratio is greater than the preset threshold, then replace the flow identifier of this data packet with the key-value pair with the smallest replacement value among the minimum values of the counters in A2. Execute Step 5-2. Otherwise, end the insertion. Step 5-2: The key in the original key-value pair will be hashed again and inserted into the A2 counter array corresponding to it, and all counters will be incremented by the value in the original key-value pair, along with the key-value pair with the smallest replacement value among the minimum values of the counters in A2. Execute Step 5-2. Otherwise, end the insertion. Step 5-2: The key in the original key-value pair will be hashed again and inserted into the A2 counter array corresponding to it, and all counters will be incremented by the value in the original key-value pair, and the insertion ends. Step Six: Insert the data packet into the hash table in the bottom layer structure, map it to the corresponding counter using the hash value corresponding to the first row in the top layer structure, perform an increment operation on this counter, and the insertion ends.

3. A query algorithm for the fine-grained network measurement structure based on the funnel-shaped Sketch according to Claim 1, and the specific query steps are as follows: Step One: Query the flow in the top layer structure, including the following steps: Step 1-1: Map to the corresponding counter in the top layer structure through the hash function according to the flow identifier; Step 1-2: Determine whether the minimum value of all mapped counters overflows. If no overflow occurs, directly return the minimum value of the counter, and the query ends. Otherwise, execute Step Two; Step Two: Map the flow to the corresponding counter in the A1 counter array in the bucket using the hash value. Determine whether the minimum value of all counters overflows. If no overflow occurs, directly return the sum of the minimum value of the A1 counter and the maximum value that the counter in the top layer structure can record, and the query ends. Otherwise, execute Step Three. Step Three: Query this traffic in the key-value pairs in the middle layer structure, and sequentially access the keys in the corresponding bucket in the middle layer structure, including the following steps: Step 3-1: If the key matches the flow identifier of this data packet. At this time, determine whether the value overflows. If no overflow occurs, then directly return the value in this key-value pair, the sum of the maximum value that the A1 counter can record and the maximum value that the counter in the top layer structure can record, and the query ends. Otherwise, execute Step Five, return the counter value in the bottom layer structure hash table, the maximum value that the key-value pair can record, the sum of the maximum value that the A1 counter can record and the maximum value that the counter in the top layer structure can record, and the query ends; Step 3-2: If the key does not match the flow identifier of this data packet. Then access the key in the corresponding bucket in the next row, and re-execute Steps 3-1 and 3-2; Step 4: Query this traffic in the A2 counter array in the middle layer structure. Determine whether the minimum value of all counters reaches the maximum value that the counter can record. If it does not reach the maximum value, directly return the sum of the minimum value of the A2 counter, the maximum value that the A2 counter can record, and the maximum value that the counter in the top layer structure can record, and the query ends. If it has reached the maximum value, execute Step 5, return the hash table counter value of the bottom layer structure, the maximum value that the A1 counter can record, the maximum value that the A2 counter can record, and the sum of the maximum value that the top layer structure counter can record, and the query ends; Step 5: Query this traffic in the bottom layer structure, map it to the corresponding counter by using the hash value corresponding to the first row in the top layer structure of this data packet, and return the counter value.

Images