A low-overhead top-k network stream high-precision extraction method

Abstract

The application discloses a low-overhead Top-k network flow high-precision extraction method, comprising: a small flow filter, which is used for filtering most small flows in a network, reducing resource overhead, and reducing a hash collision rate; and a large flow extractor, which is used for extracting Top-k flows in the network and improving the accuracy of Top-k flow identification. The method proposes an adaptive updating strategy based on a counter over-value proportion. Whenever the proportion of the number of counters exceeding a threshold value in the small flow filter is too high, each counter records whether it exceeds the threshold value in the current period through a flag bit, and then is reset to enter the next period, so that the continuous effectiveness of the small flow filter is maintained. Meanwhile, the large flow extractor is designed by using a segmented hash algorithm, multiple candidate positions are provided for incoming flows, and when all the candidate hash buckets are full, a replacement strategy is carried out through a voting mechanism, so that small flows can be kicked out as much as possible each time, and large flows can be saved in the hash buckets, and therefore the accuracy of Top-k flow extraction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network measurement, in particular to a low-overhead high-precision Top-k network flow extraction architecture and method. BACKGROUND

[0002] Top-k flow extraction is mainly used to find out the network flow with the top-k number of packets in network traffic, which can provide support for congestion control, network operation, network billing and anomaly detection, and is a basic task of network measurement. The current mainstream Top-k flow extraction scheme is to use Sketch data structure to record the number of packets of all flows, and then extract Top-k flow through Min-heap. However, since the number of network flows is much larger than the number of counters in each row of Sketch, many flows may be mapped to the same counter of Sketch, which is prone to the problem that small flows are misjudged as large flows. An effective improvement scheme is to use Sketch to filter small flows, and then use a hash structure to record the filtered Top-k flow, so as to improve the accuracy of Top-k flow extraction. However, as network packet flows continue to arrive, all counters in Sketch will continue to increase until overflow, eventually causing the small flow filter to be completely ineffective, thereby seriously affecting the accuracy of subsequent large flow extraction. At the same time, when multiple flows are mapped to the large flow extractor, hash collision may occur, which is prone to the situation that large flows are kicked out, thereby reducing the accuracy of Top-k flow extraction. In view of this, based on the idea of filtering small flows first and then extracting large flows, the present application proposes a low-overhead high-precision Top-k network flow extraction method, which solves the problem of invalidity of the small flow filter, reduces the hash collision rate of the large flow extractor, and thereby improves the accuracy of Top-k flow extraction.

[0003] Comparative document CN111262756A discloses a high-speed network elephant flow accurate measurement method and architecture, which is based on a Sketch filter capable of filtering mouse flows in data packets to reduce subsequent calculation and space overhead and improve the accuracy of subsequent elephant flow measurement; and a Cuckoo hash-based extractor capable of reducing the probability of elephant flow being kicked out, thereby saving extraction resources and improving the accuracy of elephant flow measurement. The design of the filter based on the Sketch filter in the comparative document scheme has the problem of filtering invalidity, and the extractor based on the Cuckoo hash has the problem of low accuracy. In view of this, the present application designs a compact small flow filter supporting adaptive update, which ensures the continuous effectiveness of the small flow filter when facing large traffic, reduces the resource overhead of subsequent small flow processing, and filters out most small flows through the small flow filter, so that the flows entering the large flow extractor are basically large flows, thereby improving the extraction accuracy of Top-k flow. SUMMARY

[0004] The patent is based on the idea of filtering small streams to extract large streams, and then designs a Top-k stream accurate extraction scheme of filtering first and then extracting. In the scheme, the small stream filter adopts a compact sketch data structure, when the number of large value counters in the filter reaches a preset proportion, each counter is configured with a flag bit to record whether it exceeds the threshold, which will be used as the basis for determining whether the incoming stream is released in the next period, and then the small stream is always accurately filtered, thereby improving the accuracy of Top-k stream extraction. On this basis, a segmented hash algorithm is used to design a large stream extractor to provide multiple candidate positions for each stream, and when all candidate positions are full, the smallest stream is selected based on the voting idea, and then it is judged whether to replace the incoming stream to preserve the large stream, thereby realizing the accurate extraction of Top-k stream.

[0005] To solve the above technical problems, the application adopts the following technical solutions:

[0006] The application provides a low-overhead Top-k network stream high-precision extraction architecture and method, comprising:

[0007] The small stream filter proposes an adaptive update strategy based on the proportion of counter overvalue, each element in which contains a counter and a flag bit, the counter is used to record the number of packets mapped to this position in the current period, since the counter only needs to record the number of packets in a period, and the period is usually set short, it can be set to a few bits, the flag bit occupies one bit, which is used to record the state of the corresponding counter before reset, that is, whether it exceeds the threshold at the end of the last period, as the basis for determining whether the incoming stream is released in the current period, only when all the flag bits in the sketch are 1, or all the counters reach the threshold, the stream is likely to be a large stream, and it will be released, otherwise discarded, thereby realizing the function of filtering small streams. In short, each element only occupies a few bits, so that the sketch data structure occupies a small space and has high space utilization;

[0008] The large stream extractor is designed based on a segmented hash algorithm, each hash bucket contains multiple slots for recording streams, each slot contains a stream signature, a positive vote counter and a negative vote counter, the stream signature is used to identify the stream, the positive vote counter is used to record the number of packets of the stream, and the negative vote counter is used to record the number of packets mapped to the corresponding hash bucket but not belonging to any stream. For each incoming stream, the large stream identifier provides multiple candidate positions using the segmented hash method, and selects an empty position for storage, when all candidate positions are full, the smallest stream is selected based on the voting idea, and then it is judged whether to replace the incoming stream, thereby achieving the effect of accurately identifying Top-k stream;

[0009] The small flow filter, a filter for small flow filtering, filters the small flow by setting a threshold, wherein the counter represents the number of packets mapped to the position in the current period, and the flag represents the state before the counter is reset, i.e., at the end of the last period;

[0010] The large flow extractor, a large flow extractor based on segmented hashing, extracts the large flow by using a segmented hashing algorithm, wherein the extracted content field is a signature value sig used for identifying the flow, a counter count used for recording the number of packets of the flow, and a counter count n The counter count is used for recording the number of packets mapped to the corresponding hash bucket but not belonging to any flow.

[0011] The method also provides a method based on the above architecture, comprising:

[0012] The small flow filter first extracts the flow identifier fid in the insertion process, then maps an element in each array of the sketch through d different hash functions, reads the corresponding flag and counter, adds 1 to the minimum counter value, and if all the flag values are not all 1 and the minimum counter does not reach the threshold, it indicates that it is a small flow, and the data packet is directly discarded, otherwise, it is passed; if the number of counters reaching the threshold in the current sketch exceeds a certain proportion, the corresponding flag value is updated according to the counter value; if the counter value reaches the threshold, the flag is updated to 1, otherwise, it is 0. Finally, all the counter values are emptied.

[0013] The large flow extractor maps the flow identifier fid to multiple candidate positions through a segmented hash function, and then searches in the corresponding hash bucket in parallel; if a flow is successfully found, the positive vote count of the flow is added by 1; if the search fails and there is a vacancy in the candidate position, a vacancy is randomly selected to store the flow, and the positive vote count is set to 1; otherwise, the negative vote count of all candidate positions is added by 1, and the flow with the smallest ratio of positive vote count to negative vote count is selected from all candidate positions; if the ratio is greater than a predetermined threshold, the data packet is directly discarded, otherwise, the incoming flow replaces the smallest flow, and the positive vote count is added by 1 and the negative vote count is reset to 0.

[0014] Further, the efficient method comprises the following operations:

[0015] 1. Flow extraction process;

[0016] When a packet is received, the protocol header is first parsed, the five-tuple field is extracted, the flow identifier fid is calculated, and then the small flow determination process is entered to determine whether the flow is a small flow; if it is a small flow, the packet is directly discarded, otherwise, the Top-k flow extraction process is entered, and the flow fingerprints and packet numbers of all Top-k flows are recorded for query.

[0017] 2. A flow filtering method;

[0018] Each incoming flow is filtered by a small flow filter, and if a threshold is reached, the flow is released to a large flow extractor, and if the threshold is not reached, the flow is discarded directly, wherein only the number of packets is extracted for each flow;

[0019] 3. A flow identification method;

[0020] An incoming flow is extracted by a large flow extractor, and it is determined whether the flow has been extracted by the large flow extractor according to the flow identifier of the extracted flow, wherein the flow identifier, the number of packets, and the number of hash collisions are extracted for each flow;

[0021] 4. A flow replacement method;

[0022] When the mapping position of an incoming flow is full and replacement conditions are met, the incoming flow replaces an extracted hash collision flow, wherein the flow identifier, the number of packets, and the number of hash collisions are extracted for each flow.

[0023] 5. A flow output method;

[0024] The large flow extractor then iterates through all slots in the hash bucket in parallel, extracts flows with a flow number greater than a threshold, and then outputs the flow id and the flow number in descending order.

[0025] The present application has the following beneficial effects:

[0026] 1. In view of the failure problem of a conventional small flow filter under continuous network packet arrival, the present application proposes an adaptive update strategy based on a counter over-threshold proportion for the small flow filter to ensure its continuous effectiveness. When the number of counters exceeding the threshold in the small flow filter accounts for a preset proportion, each counter records its state, i.e., whether it exceeds the threshold, with a flag bit, and is then reset to zero and starts counting again in the next period. In the new period, the flag bit of a certain counter is used as a basis for determining whether to release a data packet after the data packet is mapped to the counter, thereby continuously filtering small flows;

[0027] 2. In view of the problem that a conventional large flow extractor usually provides only one hash candidate position for an incoming flow, which is prone to hash collisions, the present application adopts a segmented hash method to provide multiple candidate positions for each incoming flow in the large flow extractor to reduce the hash collision rate and thus accommodate as many large flows as possible. If all candidate positions are full, the present application uses a voting idea to select the smallest flow from all candidate positions, and then determines whether to replace the smallest flow with the incoming flow, so that the large flow is retained and the discarded flow is as small as possible, thereby improving the precision of Top-k flow extraction. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only aim to some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0029] Figure 1 is a low-overhead Top-k network stream high-precision extraction architecture of the method of the present application.

[0030] Figure 2 is a small stream filter structure in the method of the present application.

[0031] Figure 3 is a large stream extractor structure in the method of the present application.

[0032] Figure 4 is a stream extraction flow structure in the method of the present application.

[0033] Figure 5 is a stream filtering method flow chart in the method of the present application.

[0034] Figure 6 is a stream identification method flow chart in the method of the present application.

[0035] Figure 7 is a stream replacement method flow chart in the method of the present application.

[0036] Figure 8 is a stream output method flow chart in the method of the present application. DETAILED DESCRIPTION

[0037] In order to better illustrate the content of the present application, the present application will be further verified by specific embodiments. It is particularly stated that the embodiments are only for more directly describing the present application, and they are only a part of the present application, and cannot constitute any limitation on the present application.

[0038] As shown in Figure 1 , the embodiment of the present application provides a low-overhead Top-k network stream high-precision extraction architecture and method, which comprises:

[0039] The small flow filter first extracts its flow identifier fid during the insertion process, then maps an element in each array of the sketch through d different hash functions, and then reads the corresponding flag bit and counter, adds 1 to the minimum counter value, and if all flag bits are not all 1 and the minimum counter does not reach the threshold value, it indicates that it is a small flow, and the data packet is directly discarded, otherwise it is passed. If the number of counters reaching the threshold value in the current sketch exceeds a certain proportion, the corresponding flag bit value is updated according to the counter value, and if the counter value reaches the threshold value, the flag bit is updated to 1, otherwise it is 0. Finally, all counter values are emptied, as shown in Figure 2 .

[0040] The large flow extractor maps its flow identifier fid to multiple candidate positions through a segmented hash function, and then performs parallel search in the corresponding hash bucket. If a flow is successfully found, the positive vote number of the flow is increased by 1. If the search fails and there is a vacancy in the candidate position, a vacancy is randomly selected to store the flow, and the positive vote number is set to 1. Otherwise, the negative vote number of all candidate positions is increased by 1, and the flow with the smallest ratio of positive vote number to negative vote number is selected from all candidate positions. If the ratio is greater than a predetermined threshold, the data packet is directly discarded. Otherwise, the incoming flow replaces the smallest flow, and the positive vote number is increased by 1 and the negative vote number is reset to 0, as shown in Figure 3 .

[0041] When the switch receives a certain data packet p, it first parses important fields in the header, such as source / destination IP address, source / destination MAC address, source / destination port number, IP protocol type, etc., and then extracts the flow key fid. Then, the data packet enters the first layer, i.e. the small flow filter filters the small flow. Finally, the data packet passing through the filter enters the second layer, i.e. the large flow extractor completes the flow information statistics;

[0042] The embodiment also provides a method based on the above architecture, including the following steps:

[0043] 1. Flow extraction process;

[0044] As shown in Figure 4 , when a packet is received, the protocol header is first parsed, the five-tuple field is extracted, the flow identifier fid is calculated, and then the small flow determination process is entered to determine whether the flow is a small flow. If it is a small flow, the packet is directly discarded. Otherwise, the Top-k flow extraction process is entered, and the flow fingerprint and packet number of all Top-k flows are recorded for query use;

[0045] 2. Flow filtering method;

[0046] As shown in Figure 5As shown, when a certain data packet p is received, the data packet is first processed by the filter, and the number of counters (k) in the filter is taken as a threshold value;

[0047] Judging flag i If all are 1, the counter min is incremented by 1, and the flow enters the flow identification process; if not, it is further judged whether the counter min reaches the threshold value, if yes, the flow enters the flow identification process; if not, the counter min is incremented by 1.

[0048] It is judged whether the counter min reaches the threshold value, if not, the operation ends; if yes, the number of full counters k is incremented by 1.

[0049] It is judged whether the value of k reaches the threshold value, if not, the operation ends; if yes, the corresponding flag value is updated according to the counter value, for example, if the counter value does not reach the threshold value, the flag value is updated to 0; otherwise, to 1, and then all counter values counter i are reset to 0, and the operation ends.

[0050] 3. A flow identification method;

[0051] As shown, Figure 6 first, important fields in the data packet header are parsed, and a flow key fid is extracted; then, a hash function is used to generate a connection signature value sig from the flow key fid, and a hash value of m bits is calculated by a hash function Hash, and n bits are randomly selected from each segment, and each segment corresponds to a position on w hash tables;

[0052] Parallel search is performed on the w candidate hash buckets, and when the hash bucket contains the flow (B j [index j ].sig==sig), the counter count of the flow is incremented by 1, and the flow identification process ends.

[0053] When the hash bucket does not contain the flow and the corresponding position is empty (B j [index j ][i].sig==0), the connection signature value, the counter value, and the reverse counter value (sig, 1, 0) are written into the empty slot, and the flow identification process ends.

[0054] When the hash bucket contains flows and is not the flow, a new to-be-replaced cache item item is created, and the flow replacement process is entered.

[0055] 4. A flow replacement method;

[0056] As shown,Figure 7 As shown, first, input the stream keyword fid and connection signature value sig of the cache item to be replaced;

[0057] Calculate the w candidate hash bucket positions B of the item to be cached. j [index j Then, retrieve the counter values ​​from all candidate buckets, compare them, and find the slot position (B) with the largest ratio of the counter value to the total counter value. j [index j ]), and retrieve its counter value C. min Calculate the ratio μ between the counter value and the inverse counter value;

[0058] Determine if the ratio μ reaches a predetermined threshold. If it does, clear the flow information of that slot and write the flow information of the item to be cached into the hash bucket (sig, C). min +1,0), operation complete.

[0059] If the desired result is not achieved, delete the item to be cached and replace, and the operation ends.

[0060] 5. Stream output method;

[0061] like Figure 8 As shown, first, we input a threshold to define a large flow (for example, if we define a flow with more than 500 as a large flow, then we input 500). The large flow extractor then traverses all the slots in the hash buckets in parallel, extracts the flows with more than the threshold, and then outputs the flow ID and the number of flows in descending order, and the operation ends.

[0062] This patent adopts the idea of ​​filtering small flows and then extracting large flows, and proposes a low-overhead, high-precision extraction method for Top-k network flows. The invention has the following advantages: (1) This patent stores as many large flows as possible while ensuring continuous filtering of small flows, thereby reducing the resource overhead of inserting and replacing a large number of small flows, and thus achieving accurate extraction of Top-k flows with low overhead. (2) The small flow filter records the state of the corresponding counter before the reset, i.e., at the end of the previous cycle, through the flag bit, ensuring its continuous effectiveness in filtering small flows, while making the filter occupy a small space and have high space utilization. (3) The large flow extractor reduces the hash collision rate by providing multiple candidate positions for each flow, thereby accommodating as many large flows as possible, and thus improving the extraction accuracy of Top-k flows.

[0063] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art can make various modifications or additions to the described embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A low-overhead Top-k network flow high-precision extraction method, characterized in that, The small flow filter comprises an adaptive updating strategy based on a counter exceeding a threshold ratio, each element in the small flow filter comprising a counter and a flag bit, the counter being used for recording the number of packets mapped to the element in a current period, and the flag bit occupying one bit and being used for recording the state of the corresponding counter before being reset, i.e., whether the counter exceeds the threshold at the end of a previous period, as a basis for judging whether an incoming flow is released in the current period, and only when all the flag bits of the incoming flow mapped to the sketch are 1 or all the counters reach the threshold, the flow is a potential large flow and is released, otherwise the flow is discarded; The large flow extractor comprises a large flow extraction structure based on a segmental hash algorithm, each hash bucket comprising a plurality of slots for recording flows, each slot comprising a flow signature, a positive vote counter and a negative vote counter, the flow signature being used for identifying the flow, the positive vote counter being used for recording the number of packets of the flow, and the negative vote counter being used for recording the number of packets mapped to the corresponding hash bucket but not belonging to any flow in the hash bucket, for each incoming flow, the large flow identifier provides a plurality of candidate positions by using a segmental hash method, and randomly selects one empty position for storage, and when all the candidate positions are full, the smallest flow is selected by using a voting idea, and then it is judged whether the smallest flow is replaced by the incoming flow; The small flow filter is used for filtering small flows by setting a threshold, wherein the counter represents the number of packets mapped to the position in the current period, and the flag bit represents the state before the corresponding counter is reset, i.e., the state at the end of the previous period; the small flow filter first extracts the flow identifier fid in the insertion process, then maps an element in each array of the sketch by using d different hash functions, reads the corresponding flag bit and counter, adds 1 to the minimum counter value, and if all the flag bits are not all 1 and the minimum counter does not reach the threshold, it indicates that the corresponding flow is a small flow, and the data packet is directly discarded, otherwise the data packet is released, if the number of counters reaching the threshold in the current sketch exceeds a certain proportion, the corresponding flag bit value is updated according to the counter value, if the counter value reaches the threshold, the flag bit is updated to 1, otherwise it is 0, and finally all the counter values are cleared; The large flow extractor maps the flow identifier fid to a plurality of candidate positions by using a segmental hash function, and then searches in the corresponding hash buckets in parallel, if a flow is successfully found, the positive vote number of the flow is added by 1, if the search fails and there is an empty position in the candidate positions, a random empty position is selected to store the flow, and the positive vote number is set to 1, otherwise the negative vote number of all the candidate positions is added by 1, and the flow with the minimum ratio of the positive vote number to the negative vote number is selected from all the candidate positions, if the ratio is greater than a preset threshold, the data packet is directly discarded, otherwise the incoming flow replaces the minimum flow, and the positive vote number is added by 1 and the negative vote number is reset to 0. The large flow extractor, the large flow extractor based on segment hash, extracts large flow by using multi-hash algorithm, wherein the extracted content field is a signature value sig used for identifying the flow, a counter count used for recording the number of packets of the flow, and a reverse counter count n is used for recording the number of packets mapped to the corresponding hash bucket but not belonging to any one of the flows.

2. A method based on claim 1, characterized in that, The flow extraction process comprises the following operations: a, flow extraction process; 3. The method of claim 2, wherein, ​ ​ When a packet is received, its protocol header is first parsed, the five-tuple fields are extracted, the flow identifier fid is calculated, and then the small flow judgment process is entered to determine whether the flow is a small flow. If it is a small flow, the packet is discarded directly. Otherwise, the Top-k flow extraction process is entered, and the flow fingerprints and packet numbers of all Top-k flows are recorded for query use. b. A flow filtering method Each incoming flow is filtered by the small flow filter. If the threshold is reached, the flow is passed to the large flow extractor. If the threshold is not reached, the flow is discarded directly. Each flow only extracts the packet number. c. A flow identification method The incoming flow is extracted by the large flow extractor. Whether the incoming flow has been extracted by the large flow extractor is determined according to the flow identification of the extracted flow. Each flow extracts the flow identification, the packet number, and the hash collision number. d. A flow replacement method When the mapping position of the incoming flow is full and the replacement condition is met, the incoming flow replaces the extracted hash collision flow. Each flow extracts the flow identification, the packet number, and the hash collision number. e. A flow output method The large flow extractor parallelly traverses the slots in all hash buckets, extracts the flows with a flow number greater than the threshold, and then outputs the flow id and the flow number in descending order.

Images