Data processing method, system and storage medium for large flow detection

Abstract

The present invention provides a data processing method, system, and storage medium for large-scale flow detection. The method includes establishing a data storage structure, comprising: constructing a data storage structure composed of W hash tables, each hash table containing N hash buckets, for a total of W*N hash buckets; each hash bucket containing multiple counters and entries; and storing fingerprint information and counter information for each flow in an entry in a hash bucket of a hash table. The present invention overcomes some of the shortcomings of the prior art and achieves significant improvements and optimizations in storage space utilization, element deletion accuracy, large-scale flow detection capability, throughput, and adaptability.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, system and storage medium for large flow detection. Background Art

[0002] Data flow measurement uses data structures to record statistical information about the arrival of multiple data flows in order to answer related queries. The measurement module records the current frequency of elements in each flow, updates the corresponding statistics with each new element arriving, and uses these statistics to answer heavy hitter detection questions (heavy hitters: finding all flows with a frequency exceeding a certain threshold).

[0003] Cuckoo Hash is a hash collision resolution strategy designed to improve hash table space efficiency and ensure O(1) query time complexity. Its core mechanism relies on two hash functions, assigning two potential storage locations to each key-value pair to handle possible collisions.

[0004] Cuckoo Hashing is unique in its paired hash function design, where each element is assigned two positions—a primary position and a backup position. When a conflict occurs, Cuckoo Hashing resolves the issue by "kicking" the element already in the primary position to its backup position. If the backup position is also occupied, the process continues, kicking the new element to its backup position, and so on, until an element has been kicked out a set number of times, triggering a rehash operation to reallocate hash table space. This method of continuously "kicking" to make room for new elements is the core of Cuckoo Hashing.

[0005] Cuckoo Filter, based on Cuckoo Hash, further optimizes the efficiency of handling hash conflicts and element deletions. Cuckoo Filter significantly improves space utilization by expanding the storage structure into a two-dimensional array and allowing multiple elements to be stored at each index, achieving a utilization rate of up to 98%. In addition, Cuckoo Filter introduces hash fingerprint technology to ensure that even when an element is "kicked out", it can be effectively relocated to other parts of the hash table through fingerprint information, reducing conflicts and improving space efficiency. Cuckoo Filter relies on Cuckoo Hash's dynamic element addition mechanism to reduce conflicts, while using hash fingerprints to ensure the uniqueness of data. However, the storage capacity of Cuckoo Filter is limited by the size of the hash table and the size of the fingerprint.

[0006] Although Cuckoo Hash and Cuckoo Filter have made progress in resolving hash conflicts and improving space utilization, they still face problems with high misjudgment rates and operational complexity when deleting elements. In addition, their performance is limited by the size of the hash table and fingerprint, especially when processing large amounts of network traffic, it is difficult to effectively identify large flows. Existing network measurement solutions require a trade-off between accuracy, speed, and memory usage, and it is difficult to meet the accurate measurement requirements under different traffic characteristics. In addition, storage overhead, computational overhead, and deviations caused by hash conflicts are also problems that need to be solved by existing solutions. In the task implementation of large flow detection, all Sketches used for frequency estimation can perform large flow detection simultaneously by adding a minimum heap, but the effect is not good. The insertion complexity is too high, and an additional heap is required, which destroys the simplicity of the sketch. Summary of the Invention

[0007] The present invention provides a data processing method for large flow detection, including a step of establishing a data storage structure. The step of establishing the data storage structure includes: constructing a data storage structure, the data storage structure consisting of W hash tables, each hash table containing N hash buckets, a total of W*N hash buckets, each hash bucket containing multiple counters and entries, and fingerprint information and counter information of each flow are stored in an entry in a hash bucket of a hash table.

[0008] As a further improvement of the present invention, the data storage structure includes a cuckoo hash, and the cuckoo hash calculates the position of the flow in another array through the current position of the flow and the fingerprint.

[0009] As a further improvement of the present invention, a designated entry is provided in the hash bucket, and the large flow information is stored through the designated entry.

[0010] As a further improvement of the present invention, the ID and the hash fingerprint are stored in the hash bucket.

[0011] The present invention also provides a data processing system for large flow detection, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the data processing method of the present invention when called by the processor.

[0012] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the data processing method of the present invention when called by a processor.

[0013] The beneficial effects of the present invention are as follows: the present invention overcomes some defects of the prior art and achieves significant improvements and optimizations in storage space utilization, accuracy of element deletion, large flow detection capability, throughput and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the data structure diagram of HeavyFinder;

[0015] Figure 2 It is a processing flow chart of the present invention. DETAILED DESCRIPTION

[0016] This paper discloses a data processing method for high-volume flow detection and proposes a highly efficient storage structure, called HeavyFinder. This approach aims to address a series of challenges encountered in network traffic measurement and high-volume flow detection, specifically how to reduce memory requirements, lower false positive rates, and improve detection accuracy while maintaining high throughput. HeavyFinder achieves these goals through the following key innovations:

[0017] Removing the Min-Heap Structure: Traditional large flow detection algorithms rely on a min-heap to dynamically adjust the flow ranking to keep track of the largest flow. However, the use of a min-heap increases the complexity of the algorithm and requires additional memory and time overhead for each insertion and deletion operation. HeavyFinder avoids this overhead by setting a special entry in each bucket (the special entry is a designated entry) to directly store large flow information.

[0018] Simultaneous storage of IDs and hash fingerprints (FRPs): Unlike traditional counters that can only store either IDs or FRPs, HeavyFinder's design allows for simultaneous storage of both IDs and FRPs within each bucket. The ID uniquely identifies the flow, while the FRP acts as a hash fingerprint, enhancing data uniqueness and retrieval efficiency. This design not only increases the amount of information stored but also provides higher accuracy for large flow detection.

[0019] Real-time switching and ordered sorting within buckets: HeavyFinder employs a clever design that allows real-time switching of the order of IDs and FRPs during insertion, enabling sorting of data within buckets. This approach simplifies the query process for heavy flow detection traffic, as queries only require a partial traversal of the top of the bucket, rather than a complete traversal, significantly reducing query complexity.

[0020] Load Mechanism: To further improve accuracy, HeavyFinder introduces a load balancing mechanism that allows for load balancing of large flow detection between primary and backup buckets. This ensures even distribution of data across buckets, avoiding issues such as excessive congestion or uneven load on a single bucket.

[0021] In summary, through these innovative designs, HeavyFinder not only significantly improves throughput and insertion speed, but also achieves a lower error rate and higher accuracy under limited memory resources, meeting the needs of efficient network traffic measurement and large-scale flow detection.

[0022] The data structure of HeavyFinder is as follows Figure 1 As shown, the system primarily consists of a Cuckoo Hash-based data structure consisting of two arrays, each of which consists of multiple buckets. Two candidate buckets are set for each element. Multiple entries are also set in each bucket to resolve hash collisions. In HeavyFinder, an additional top entry is set in each bucket to store possible heavy flow detection candidates. When heavy flows are inserted, they are added to the top entry through the accumulation of counters and become heavy flow detection candidates. Finally, the top heavy flow is identified through sorting.

[0023] HeavyFinder's data structure consists of W hash tables, each containing N hash buckets, for a total of W*N hash buckets. Each hash bucket contains multiple counters and entries. The fingerprint and counter information of each flow are stored in an entry in a hash bucket of a hash table, namely Array[x][y].Entry[z]. This method allows HeavyFinder to efficiently store and manage large amounts of traffic data and accurately detect large flows.

[0024] This paper introduces partial-key cuckoo hashing, which derives the alternative location of a flow based on its fingerprint. For a network packet p belonging to flow e, we calculate the index of its two candidate buckets as follows: h1(e) = hash(e), In the formula (XOR) ensures that h1(e) can also be calculated from h2(e) and the fingerprint of e, that is, Therefore, no matter which array a stream appears in, we can calculate its position in the other array using its current position and fingerprint:

[0025]

[0026] Table 1 is the overall data structure of the present invention

[0027]

[0028] like Figure 2 As shown, the insertion process of the present invention includes the following steps:

[0029] Step S01: traverse the hash buckets in each hash table, use different hash functions to calculate M hash buckets Array[i][hi(f)], and obtain the fingerprint record in each hash bucket.

[0030] Step S02: Determine whether the fingerprint records in all elements are equal to the fingerprint inserted into the stream. If there are equal fingerprint records (i.e., Array[x][hx(f)].Entry[y].fpt=f.fpt), then execute step S03; otherwise, it is considered that there is no such stream record in the current data structure, and execute step a.

[0031] Step S03: Increase the value of the counter cnt in the found element, that is, increase Array[x][hx(f)].Entry[y].cnt.

[0032] Step S04: Determine whether the counter overflows. If so, execute step S05; if the counter cnt does not overflow, that is, the current bit length is sufficient to accommodate the new counter value, the insertion process ends and goes directly to step a.

[0033] Step S05: After the element is inserted, it is compared with the large flow entries in its two primary and backup buckets to determine whether the inserted element is larger than the top element of the candidate bucket. If so, then execute step 1, otherwise execute step S06.

[0034] Step S06: Try to exchange the elements in the current hash bucket. Compare the counter value of the inserted element with the counter value of the large flow entry, and insert the element to replace the hot flow element and enter the large flow entry.

[0035] Step 1: Kick the element out to the alternative bucket.

[0036] Step 2: After an element is inserted into a common entry in the bottom layer, it is compared with the high-flow entries in its two candidate buckets. If there is an empty space in either high-flow entry, the element is inserted into the empty space. Otherwise, the counter value of the inserted element is compared with the counter value of the high-flow entry to determine whether the inserted element can replace the hot-flow element in the high-flow entry.

[0037] Step 3: Replace the top element of the candidate bucket, push the other elements down in turn, and then execute step e.

[0038] Step a: Determine whether there is a vacancy in the current M hash buckets Array[i][hi(f)]. If so, for the kicked element P, use the cuckoo hash design to try to find the position and complete the reinsertion; otherwise, execute step b.

[0039] Step b: Complete the process of kicking out elements. When an element is kicked out, the design of Cuckoo Hash allows its position in other hash tables to be calculated. If the original Array[x][hx(f)].Entry[k] is kicked out, based on the element's fingerprint and other information, its position in the yth hash table can be calculated as hy(fk). This means attempting to operate on the hash bucket Array[y][hy(fk)] to determine whether there is a free element position and whether the bit length of the counter in that free position can accommodate the counter size of the kicked element. If there is a vacant position, the process of kicking out the element is complete. If there is no vacant position, repeat the above steps and try to kick an element in the current hash bucket to another hash table (this element may be different from the previously kicked element). If repeated several times and still unsuccessful, this method discards the current kicked element. End the operation.

[0040] Step c: If all hash buckets are full and no empty space can be found to place the current flow, the element P with the smallest counter value is replaced according to the pre-configured probability replacement strategy, and the new flow replaces P's position.

[0041] Step d: Kick out the minimum counter.

[0042] Step e: If there is an empty position in the two large flow entries, insert the empty position; store the information of the current inserted flow f in the empty position we found (update the pointer information and counter information of the corresponding entry).

[0043] The present invention improves upon the prior art, overcomes some of the shortcomings of Cuckoo Hash and Cuckoo Filter, and achieves the following advantages:

[0044] Reduced false positive rate for element deletion: Compared to the existing Cuckoo hash, this invention uses a more effective deletion strategy, which can reduce the false positive rate for element deletion. This can improve the accuracy and reliability of data storage.

[0045] Improved space utilization: Based on Cuckoo Filter, this paper further optimizes the data structure by expanding the array into a two-dimensional form, allowing each index to store multiple elements. This allows for more efficient use of table space, improves space utilization, and reduces storage overhead.

[0046] Improved high-volume flow detection: Compared to traditional Cuckoo Filter, this invention improves high-volume flow detection. By designing an orderly sorting of traffic, it can better identify and process high-volume flows, improving the accuracy and efficiency of high-volume flow detection.

[0047] High throughput: This paper proposes a storage structure called HeavyFinder. By removing the minimum heap and setting up special entries in each bucket to store large flows, this achieves high throughput without taking up additional memory. This improves the speed and efficiency of network measurement.

[0048] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A data processing method for large flow detection, characterized in that: Steps to create a data storage structure: Steps for establishing a data storage structure: Build a data storage structure consisting of W hash tables. Each hash table contains N hash buckets, for a total of W*N hash buckets. Each hash bucket contains multiple counters and entries. The fingerprint information and counter information of each flow are stored in an entry in a hash bucket of a hash table. The data processing method comprises the following steps: Step S01: traverse the hash buckets in each hash table, use different hash functions to calculate M hash buckets, and obtain the fingerprint record in each hash bucket; Step S02: Determine whether the fingerprint records in all elements are equal to the fingerprint of the inserted stream. If so, execute step S03; otherwise, execute step a. Step S03: increase the value of the counter cnt in the found element; Step S04: Determine whether the calculator overflows, if so, execute step S05, otherwise execute step a; Step S05: After the element is inserted, it is compared with the large flow entries in the two primary and backup buckets to determine whether the inserted element is larger than the top element of the candidate bucket. If so, step 1 is executed; otherwise, step S06 is executed. Step S06: Try to exchange the elements of the current hash bucket, and then insert it to end; Step 1: Kick the element out to the alternative bucket; Step 2: Compare the counter value of the inserted element with the counter value of the large flow entry to see whether the inserted element can replace the hot flow element and enter the large flow entry; Step 3: Replace the top element of the candidate bucket, push the other elements down in turn, and then execute step e; Step a: Determine whether there is a vacancy in the current M hash buckets. If so, insert the element into the location using the cuckoo hashing design, and then end; Otherwise, proceed to step b; Step b: Complete the process of kicking out elements; Step c: According to the pre-configured probability replacement strategy, replace the element P with the smallest counter value, and the new flow takes the place of P; Step d: Kick out the minimum counter; Step e: If there are free positions in the two large flow entries, insert the free position; store the information of the current inserted flow in the free position.

2. The data processing method according to claim 1, wherein: The data storage structure includes a cuckoo hash, which calculates the position of a stream in another array using its current position and fingerprint.

3. The data processing method according to claim 1, wherein: A designated entry is set in the hash bucket, and the large flow information is stored through the designated entry.

4. The data processing method according to claim 1, wherein: The hash bucket stores the ID and hash fingerprint.

5. The data processing method according to claim 1, wherein: During the insertion process, the order of ID and hash fingerprint is switched in real time to achieve sorting of data in the hash bucket.

6. The data processing method according to claim 1, wherein: In step S06 , the counter value of the inserted element is compared with the counter value of the large flow entry, and the inserted element replaces the hot flow element and enters the large flow entry.

7. The data processing method according to claim 1, wherein: In step e, the pointer information and counter information of the corresponding entry are updated.

8. A data processing system for large flow detection, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 7 when called by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 7 when called by a processor.

Images