A network flow data storage method and system based on cold and hot flow separation, an insertion method and a query method

Abstract

The application discloses a network flow data storage method and system based on cold and hot flow separation, an insertion method and a query method, and the key idea is to adopt different strategies for frequent item and non-frequent item flow keys. In view of the diversity of real-time distribution application requirements of each flow key, the workflow of Hermetis includes two parts: a lean part and a rough part. The lean part is used for storing frequent items within a time window. The lean part removes non-frequent items through a competitive selection process, and for real-time frequent item query tasks, the lean part greatly improves query efficiency and reduces the probability of false positives caused by hash collision. The rough part is a probabilistic data structure with a self-adjusting counter size according to the change of the time window, and is used for recording the distribution of 'fuzzy flow keys'. Experiments show that the data structure of the application is significantly improved in accuracy, throughput and memory overhead compared with other data structures and algorithms.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and more specifically, to a storage method and system, an insertion method, and a query method for network flow data based on cold and hot flow separation. Background Art

[0002] In recent years, network measurement and monitoring has become a research hotspot in the network field. Using network measurements to map network traffic conditions and using these measurements as security situational awareness data is an important development direction for threat situational awareness, including traffic behavior classification, performance diagnosis, and anomaly detection.

[0003] In various types of traffic characterization, measuring flow key distribution can provide more fine-grained information. Flow-level granularity, such as packet size and latency, has great potential for improving anomaly detection, link load adjustment, and user quality of service. While considerable work has been devoted to achieving accurate traffic distribution queries, existing stream processing methods have not considered the aging mechanism of data distribution. However, when traffic feature distributions change rapidly, flow key distributions can vary significantly. Summary of the Invention

[0004] This paper proposes a storage method and system, an insertion method, and a query method for network flow data based on hot and cold flow separation. It designs a compact data structure for real-time measurement of per-flow key distribution, aiming to achieve fine-grained per-flow key distribution monitoring. This method can accurately measure and adapt to the distribution status of each flow, and report the frequent items in the current time window.

[0005] To achieve the above objectives, the present invention provides a first aspect of a method for storing network flow data based on cold and hot flow separation, comprising:

[0006] Design the Hermetis data structure, which consists of a lean part and a coarse part. The lean part is modeled as a hash table with several buckets. Each bucket includes a flow key ID for recording the maintained flow key and a negative counter for recording the flow count that is mapped to the bucket but is different from the flow key ID in the bucket. The coarse part contains several slots, each slot contains several entries, and each entry contains a fingerprint, a time tag, and a count value recording the number of times the data item has been inserted.

[0007] The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

[0008] In one embodiment, each bucket of the lean part also includes multiple histogram interval arrays, wherein each histogram interval includes a histogram interval ID part, a start time, a last time and multiple counters, the start time and the last time respectively mark the timestamps of the earliest arrival flow and the latest arrival flow of the flow key maintained in bucket B, and the hash function H maps the flow key ID to an index of l buckets.

[0009] In one embodiment, the coarse part includes a hash function and an array A[k]. The hash function is used to map an input element to an index position of the array A[k], and then approximately count the frequency through multiple hash functions.

[0010] Based on the same inventive concept, the third aspect of the present invention provides a network flow data insertion method based on the network flow data storage method based on the cold and hot flow separation described in the first aspect, comprising:

[0011] For an item arriving at time t, E =<e,bid,t> , calculate the insertion index pos=H(e) of item E in the lean part; locate the bucket at the corresponding position according to the insertion index

[0012] If the bucket The maintained stream key ID is the same as the stream key ID of project E or the bucket If the bucket is empty, then the item E=<e,bid,t> Insert into the lean section;

[0013] If the bucket If the maintained stream key ID is different from the stream key ID of project E, the bucket will be Replace the stream key maintained in with the stream key ID of project E and The originally maintained flow keys are expelled into the coarse part.

[0014] In one embodiment, if the stream key ID of item E cannot be saved to the bucket , then the stream key ID of item E is evicted to the coarse part.

[0015] In one embodiment, it includes calculating the slice index of the current time and performing interval update operations on the counters, specifically: when the time interval exceeds the set threshold, all counters are cleared and the timestamps are reset; when the difference between the earliest arrival time of the data and the stream key maintained in the bucket exceeds the time difference threshold, the oldest counter is removed and the time range is updated; otherwise, the current counter is directly updated and the last timestamp is adjusted.

[0016] In one embodiment, evicting the stream key ID of item E into the coarse portion includes:

[0017] According to the hash function hfp Determine the target slot for each row and check whether the slot's timestamp has expired. If so, update it to the current timestamp.

[0018] If the slot fingerprint is consistent with the data fingerprint, the count value is directly updated; if not, the entry content of the slot is replaced with the second probability.

[0019] Based on the same inventive concept, the third aspect of the present invention provides a network flow data query method based on the network flow data storage method based on the cold and hot flow separation described in the first aspect, comprising:

[0020] Check whether the target flow exists in the lean part. If so, accumulate the count value in the corresponding time slice;

[0021] Access the rough part, calculate the fingerprint and time tag of the traffic data, and search for the corresponding slot row by row. If the slot fingerprint matches, the statistical result is updated to the minimum count value;

[0022] Add the results of the lean part and the rough part and output the final statistical value.

[0023] Based on the same inventive concept, a fourth aspect of the present invention provides a network flow data storage system based on cold and hot flow separation, comprising:

[0024] The data structure design module is used to design the Hermetis data structure. The Hermetis data structure consists of a lean part and a coarse part. The lean part is modeled as a hash table with several buckets. Each bucket includes a flow key ID for recording the maintained flow key and a negative counter for recording the flow count that is mapped to the bucket but is different from the flow key ID in the bucket. The coarse part contains several slots, each slot contains several entries, and each entry contains a fingerprint, a time tag, and a count value recording the number of times the data item has been inserted.

[0025] The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

[0026] Based on the same inventive concept, the fifth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the network flow data storage method based on cold and hot flow separation described in the first aspect is implemented.

[0027] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0028] The present invention proposes a storage method for network flow data based on the separation of hot and cold flows, which adopts different strategies to store frequent items and infrequent item flow keys. First, a Hermetis data structure is designed, and the lean part is used to store frequent items within a time window. The lean part eliminates infrequent items through a competitive selection process. For real-time frequent item query tasks, the lean part greatly improves the query efficiency and reduces the probability of false positives due to hash collisions. The rough part is a probabilistic data structure that automatically adjusts the counter size according to the change of the time window, and is used to record the distribution status of the cold flow. Taking into account the differences and high skewness of the distribution of different flow keys, the rough part adopts a method of dynamically adjusting the counter size to match the distribution of the data flow. Experiments show that the data structure of the present invention has significantly improved the performance in terms of accuracy, throughput and memory overhead compared with other data structures and algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is an overview of the overall data structure of Hermetis in an embodiment of the present invention;

[0031] Figure 2 This is a detailed diagram of the data structure of the Hermetis lean part of an embodiment of the present invention;

[0032] Figure 3 This is a detailed diagram of the data structure of the Hermetis coarse part of an embodiment of the present invention;

[0033] Figure 4 This is a flowchart of inserting a data stream into Hermetis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In this paper, the real-time data distribution of a flow key is defined as the distribution of each flow key at the current moment within a fixed-size sliding window. Existing solutions cannot promptly remove outdated traffic information, resulting in a significant gap between the stored data distribution and the actual flow key distribution. Therefore, to meet the demand for real-time measurement of traffic distribution, improving the data aging process for outdated data to more accurately reflect immediate distribution changes is crucial for maintaining measurement accuracy in dynamic data environments.

[0035] The main inventive concept of this paper is to propose Hermetis, a compact data structure for real-time measurement of per-stream key distribution. The key idea is to use different strategies for hot and cold stream keys. Given the diverse application requirements for real-time distribution of per-stream keys, it is desirable to propose a fine-grained per-stream key distribution monitoring method that can accurately measure and adapt to the distribution of each stream and report the frequent items within the current time window.

[0036] Specifically, the present invention designs a Hermetis data structure. The workflow of Hermetis includes two parts: a lean part and a rough part. The lean part is used to store frequent items within a time window. The lean part eliminates infrequent items through a competitive selection process. For real-time frequent item query tasks, the lean part greatly improves query efficiency and reduces the probability of false positives due to hash collisions. The rough part is a probabilistic data structure that automatically adjusts the counter size according to changes in the time window, and is used to record the distribution status of cold streams (the distribution of infrequent items). Taking into account the differences and high skewness of the key distributions of different streams, the rough part adopts a method of dynamically adjusting the counter size to match the distribution of the data stream.

[0037] Compared with the prior art, the present invention has the following innovations and technical advantages:

[0038] 1. Innovations of the new measurement data structure:

[0039] Data Structure Innovation: Hermetis introduces a dual structure design consisting of lean and coarse components, effectively separating frequent and infrequent items and recording frequent item information in real time. This enables highly accurate estimation of stream key distribution and frequent item queries. This design improves response speed to changes in traffic distribution and enhances real-time monitoring capabilities.

[0040] Technical Difficulties: Due to the rapid changes in traffic characteristics, existing stream processing methods have difficulty updating the stream key distribution in a timely manner, resulting in a significant gap between stored data and the actual distribution. The technical difficulty of Hermetis lies in how to utilize the arrival patterns of cold and hot streams to design data structures and timestamp markers with low storage overhead to implement an efficient and accurate aging data removal mechanism. Secondly, Hermetis considers how to couple data structures and timestamp markers to maintain the compactness of the data structure. Therefore, designing a data structure that can both efficiently remove outdated data and accurately reflect the current stream key distribution is a challenge.

[0041] 2. Hybrid aging strategy:

[0042] The hybrid aging data purge strategy proposed in this paper, under a global clock, combines memory expansion with window slicing to effectively support data aging threads. By deeply analyzing the traffic characteristics of frequent and infrequent items, this strategy achieves high-precision data purge with low memory overhead, significantly improving the real-time performance and accuracy of traffic monitoring.

[0043] Technical Advantages: Hermetis's technical strengths lie in its efficient memory usage and accurate estimation capabilities. In experiments, Hermetis demonstrated outstanding performance, particularly when processing small flows, achieving a 96.6% error-free maximum value estimation rate. It also demonstrated remarkable memory efficiency in high-volume scenarios, requiring only 3MB of memory to process a 27MB project. This makes Hermetis highly scalable and practically applicable in network traffic monitoring.

[0044] 3. Asynchronous timeline data clearing:

[0045] In order to reduce the data aging operation error caused by using time tags in the rough part, the present invention uses different time axis offsets for multiple arrays, and each array adopts a completely independent hash map. When querying, the present invention adopts a query strategy similar to CM Sketch (Counted Minimum Sketch) to improve the accuracy of aging data removal.

[0046] 4. Histogram Competition:

[0047] We observe that most of the frequent items' flow keys are distributed in a few histogram intervals. The lean part always attempts to maintain the latest flow histogram intervals. Hermetis proposes a histogram competition mechanism that evicts histogram intervals with the oldest time labels and smallest counter values ​​to the coarse part, thereby reducing the number of counters in the lean part.

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] Example 1

[0050] The present invention discloses a method for storing network flow data based on cold and hot flow separation, comprising:

[0051] Design the Hermetis data structure, which consists of a lean part and a coarse part. The lean part is modeled as a hash table with several buckets. Each bucket includes a flow key ID for recording the maintained flow key and a negative counter for recording the flow count that is mapped to the bucket but is different from the flow key ID in the bucket. The coarse part contains several slots, each slot contains several entries, and each entry contains a fingerprint, a time tag, and a count value recording the number of times the data item has been inserted.

[0052] The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

[0053] Specifically, this paper proposes Hermetis, a compact data structure for real-time measurement of per-stream key distribution. The key concept of this paper is to adopt different strategies for frequent and infrequent item stream keys. Given the diverse application requirements for real-time distribution of per-stream keys, we propose a fine-grained per-stream key distribution monitoring method that can accurately measure and adapt to the distribution status of each stream, while reporting the frequent items within the current time window.

[0054] The present invention designs the Hermetis data structure. Figure 1 As shown, the data structure consists of two parts: a “lean part” that records frequent traffic and a “coarse part” that records infrequent traffic.

[0055] The lean part is modeled as a hash table with l buckets. Each bucket Midstream key K record bucket The negative counter NC records the flow count that is mapped to the bucket but is different from the flow key K in the bucket. Figure 2 As shown, the histogram array (H) contains m histogram intervals. For each histogram interval I, it is split into the histogram interval ID part bid, the start time T first , final time T last And d+1 counters form Counter[0], Counter[1], ..., Counter[d]. Start time T first and the final time T last Mark the buckets separately The timestamps of the earliest and latest arrival streams of the stream key K maintained in . The hash function H maps the stream key ID to an index of l buckets: H(·)→[0,l-1].

[0056] like Figure 3As shown, the rough part contains r slots A[0], A[1], ..., A[r-1], each slot consists of c entries. Each entry (fp, time_tag, v) contains a fingerprintfp, which is generated by the hash function h fp Generated about<e,bid> The 8-bit hash value, time tag time_tag and a count value v that records the number of times the data item is inserted. Hash function h k Associated with an array A[k], where k = 0, 1, ..., r-1. A hash function maps an input element to an index in the array A[k] and then approximates the frequency count using multiple hash functions. When multiple hash functions are used, hash functions help reduce collisions and increase the accuracy and space efficiency of data structures.

[0057] Example 2

[0058] Based on the same inventive concept, this embodiment discloses a network stream data insertion method based on the storage method of embodiment 1, comprising:

[0059] For an item arriving at time t, E =<e,bid,t> , calculate the insertion index pos=H(e) of item E in the lean part; locate the bucket at the corresponding position according to the insertion index

[0060] If the bucket The maintained stream key ID is the same as the stream key ID of project E or the bucket If the bucket is empty, then the item E=<e,bid,t> Insert into the lean section;

[0061] If the bucket If the maintained stream key ID is different from the stream key ID of project E, the bucket will be Replace the stream key maintained in with the stream key ID of project E and The originally maintained flow keys are expelled into the coarse part.

[0062] Specifically, data insertion includes: when item E=<k,bid,t> Upon arrival, the insertion index h(k) of the item in the lean part is calculated. The status of bucket B is checked: if the stream key maintained in bucket B is k or bucket B is empty, item E is directly inserted. If the stream key maintained in bucket B is not k, the eviction policy is executed. The eviction policy determines whether item E is evicted to the coarse part. If so, the evicted stream key is inserted into the coarse part.

[0063] See Figure 4, insert the Hermetis flow chart for the data flow. Step 1: Check whether E is stored in the lean part. If it is stored (the corresponding decision result is "yes"), then execute step 2: determine whether histogram interval competition occurs. If it is not stored (the corresponding decision result is "no"), then execute step 4: whether to insert E into the coarse part.

[0064] If histogram interval competition occurs (the corresponding decision result is "yes"), step three is executed: whether to remove E to the coarse part.

[0065] A plurality of histogram intervals are maintained in the histogram array. Since an item may have multiple different intervals, when there is no empty histogram interval to maintain a new interval, the present invention decides whether to expel a certain interval to the coarse part through the form of histogram interval competition. Step three is specifically as follows: if histogram competition occurs, first determine whether the items E′ and E maintained in a specific histogram are interchanged according to a certain probability. If they are not interchanged (corresponding to the decision result "yes"), E is removed from the coarse part; otherwise (corresponding to the decision result "no"), the original item in this histogram, that is, e′, is inserted into the coarse part, and the new item E is inserted into this position.

[0066] Step 4 specifically involves checking several insertion slots for E in the lean part. If none of these slots contain item E, a probability-based decision is made as to whether to swap the stream k maintained in one of these slots with E. If not (a "yes" decision), E is inserted into the rough part. Otherwise (a "no" decision), the existing item in this slot, k, is inserted into the rough part, and the new item E is inserted into this slot.

[0067] In the specific implementation process, the algorithm for inserting data into the lean part is shown in Table 1 (Algorithm 1).

[0068] Table 1

[0069]

[0070] Specifically, network traffic data insertion includes the following scenarios:

[0071] Scenario 1.2.1: The stream key e is stored in the histogram array: traverse the histogram intervals in the histogram array and check whether there is an interval that stores the stream value ID bid of e. If there is an interval I0 in bucket B that maintains the histogram interval ID bid, perform the interval update operation shown in Algorithm 3 in Table 3 on I0. Otherwise, if there is an empty interval I1, set the stream value ID maintained by I1 to bid and set T first and T lastSet to t. If there is no empty interval, randomly select an interval I' with the smallest counter value. Assume that the histogram interval ID maintained in I' is bid'. Set E'= <e,bid′, t′=I′.T last > Evict to the coarse part and empty the interval I'. Then insert item E into the interval I'. E' represents an item, e is the stream key of item E', bid' is the histogram interval ID of E', v' is the count value of E', and t' is the timestamp of E'.

[0072] Scenario 1.2.2: The bucket is empty. At this point, the stream key e is inserted into the first interval I[0]. Interval I[0] is updated as shown in Algorithm 3.

[0073] Scenario 1.2.3: Stores another stream key e′: The count value of the middle flow is V, and the count value of the negative counter is NC. With probability b0 -(V-N-1) The bucket Replace the flow key maintained in with e, and The original flow key is expelled into the coarse part.

[0074] Case 1.2.4: If the above operation fails to save e to If , e is expelled to the coarse part, the coarse part data insertion operation shown in Algorithm 2 is performed, and the count value of the negative counter is updated to NC+v.

[0075] For cases 1.2.1, 1.2.2, and 1.2.3, if a data item e is inserted into an interval I in the lean part, the interval update operation shown in Algorithm 3 in Table 3 is performed on interval I. For detailed operations, see the data stream aging operation in the lean part.

[0076] For scenario 1.2.4, the algorithm for inserting data into the coarse part is shown in Table 2 (Algorithm 2). Data is inserted into the compact data structure of the coarse part by calculating its fingerprint and time stamp. First, the target slot for each row is determined using a hash function. The slot's time stamp is checked for expiration and updated to the current time stamp if necessary. Next, if the slot's fingerprint matches the data's fingerprint, the count value is directly updated. If not, the slot's contents are replaced with a certain probability. This ensures data validity while simultaneously achieving probabilistic conflict resolution and time-dependent aging management.

[0077] Table 2

[0078]

[0079] In the specific implementation process, the process of Algorithm 2 includes:

[0080] Step 1.4.1, calculate fingerprint fp = h fp (e, bid), time tag time_tag = t / (N / d) mod (d+1). Here N is the time window size, d is the number of sub-windows divided by the time window;

[0081] Step 1.4.2, execute the loop: Forkin[0,r-1]: (where r is the number of rows in the rough part);

[0082] Step 1.4.3, calculate the slot insertion index idx=h of the kth row of the rough part k (e, bid), candidate slot Slot = Sketch[k][idx]. Here h k It is the hash function in the data structure mentioned above. As mentioned above, the rough part is modeled as a Sketch, where Sketch refers to the rough part.

[0083] In step 1.4.4, if (time_tag - Slot.time_tag) mod (d + 1) == d, then set Slot.time_tag = time_tag. The calculation method of time_tag is described in step 1.4.1. Slot.time_tag represents the time tag maintained in the candidate slot.

[0084] In step 1.4.5, if Slot.fp = fp, then set Slot.v + = v. The calculation method of fp (fingerprint) is described in step 1.4.1. lot.fp represents the fingerprint maintained in the candidate slot;

[0085] Step 1.4.6, if Slot.fp≠fp, then use the probability Replace the stream recorded by this counter with e.

[0086] After inserting the item E = <e, bid, t> into the lean part, the lean part interval update operation (Algorithm 3) is executed. Specifically, the interval update operation is performed on the counters by calculating the slice index of the current time: when the interval exceeds the set threshold, all counters are cleared and the timestamp is reset; when the data is too old, the oldest counter is removed and the time range is updated; otherwise, the current counter is directly updated and the latest timestamp is adjusted. This process ensures that the counters can efficiently maintain statistical information that changes over time through dynamic clearing and updating.

[0087] In the specific implementation process, the lean part interval update operation can be achieved through the following steps:

[0088] Step 2.1, calculate the time slice index of e

[0089] Step 2.2, if t - I.T last > N, then clear all counters in interval I; counter I.C[stp] = 1, I.T first = I.T last = t;

[0090] Step 2.3, if then calculate clear the oldest clean_step counters of all distance counters I.C[stp] in interval I; counter I.C[stp] = 1, I.T first = t - N, I.T last = t, where clean_step represents the step of clearing the counter.

[0091] Step 2.4, except for the above two cases, let counter I.C[stp] = I.C[stp] + 1, I.T last = t.

[0092] The specific algorithm is shown in Table 3.

[0093] Table 3

[0094]

[0095]

[0096] Example Three

[0097] Based on the same inventive concept, the embodiment discloses a network flow data query method based on the network flow data storage method based on cold and hot flow separation of the embodiment one, comprising:

[0098] checking whether the target flow exists in the lean part, and if so, accumulating the count value in the corresponding time slice;

[0099] accessing the coarse part, finding the corresponding slot by calculating the fingerprint and time label of the flow data, and if the slot fingerprint matches, updating the statistical result to the minimum count value;

[0100] adding the results of the lean part and the coarse part, and outputting the final statistical value.

[0101] Specifically, the data query method calculates the target traffic statistics by simultaneously accessing the data structures of the lean and coarse parts. First, it checks whether the target traffic exists in the lean part. If so, it accumulates the count values ​​within the corresponding time slice. It then accesses the coarse part and calculates the traffic fingerprint and time tag, searching for the corresponding slot row by row. If the slot fingerprint matches, the statistical result is updated to the minimum count value. Finally, the results of the lean and coarse parts are added together to output the final statistical value. This algorithm combines the accuracy of the lean part with the compactness of the coarse part, improving query efficiency and storage utilization.

[0102] The data query method can be implemented through the following steps:

[0103] Step 3.1, set result = 0. result represents the query result of the project and is initialized to 0;

[0104] In step 3.2, if the stream key e is in the lean part, query the value of the stream key e in the lean part:

[0105] Step 3.2.1, calculate the time slice index of e

[0106] Step 3.2.2, add result to the d buckets closest to IC[step]: IC[(stp)mod(d+1)], IC[(stp-1)mod(d+1)], ..., IC[(stp-d+1)mod(d+1)]:

[0107] Step 3.3, query the value of stream key e in the rough part:

[0108] Step 3.3.1, calculate the fingerprint fp and time tag time_tag,

[0109] fp=h fp (e, bid), time_tag=t / (N / d)mod(d+1);

[0110] Step 3.3.2: Set coarse_result = ∞. coarse_result represents the query result of the coarse part and is initialized to ∞.

[0111] Step 3.3.3, execute the loop Forkin[0,r-1] where r represents the number of rows in the rough part;

[0112] Step 3.3.4, calculate Candidate slot Slot = Sketch[k][idx];

[0113] Step 3.3.5, if Slot.fp == fp, then coarse_result = min(coarse_result, Slot.v);

[0114] Step 3.4, if coarse_reslut≠∞, reslut=result+coarse_result;

[0115] Step 3.5, output result.

[0116] The specific algorithm implementation is shown in Table 4.

[0117] Table 4

[0118]

[0119]

[0120] In one embodiment, integrated data storage, insertion, and querying specifically include the following steps:

[0121] S1: Determine the data insertion location. For each piece of data that arrives, we first calculate its storage location in the lean part based on the data content (stream key).

[0122] S2: Determine the status of the storage location. If the location already stores information about the same stream, traverse all storage areas at this location to find out whether there is an ID space that already stores this stream, and apply different processing strategies to the results returned by the search.

[0123] S3: If space is found to store the ID of this stream, the time and count information of this interval are updated. If space is not found to store the ID of this stream, but there is an empty storage interval, this interval is allocated to the new data and the relevant time record is set. If there is no free storage interval, the interval with the least number of records is found and the data in this space is evicted. The evicted data is placed in the coarse part and the new data is inserted into this position if it is empty. The new data is inserted into the first storage interval and the time record is updated if the position stores information of another stream. Based on the probability, it is decided whether to replace the existing stream and put the new data into the lean part. If a replacement occurs, the original stream data is moved to the coarse part; otherwise, the new data is directly placed in the coarse part.

[0124] S4: After successfully inserting the data into a storage interval of the lean part, the count and time of the interval are updated to reflect the latest data situation.

[0125] S5: The data stored in the lean part is "aged" according to time. If the current time is more than a certain time away from the last update of the data, the system will clear all counters in this interval; if the earliest time of the data is significantly different from the current time, the oldest counters will be cleared; if the current data does not exceed the aging time threshold, the counter will be directly increased and the last access time will be updated.

[0126] S6: When a piece of data needs to be queried, the system prioritizes searching the lean part. If the corresponding stream data is found in the lean part, the system selects the count values ​​from the most recent time slices and adds them together as the query result. If no data is found in the lean part, the system attempts to find records of the stream in the coarse part, calculates the minimum count value, and adds it to the query result. Finally, the query results from the lean and coarse parts are combined to obtain the query value for the data.

[0127] Data insertion includes:

[0128] Data stream insertion: When item E arrives, the insertion index pos in the lean bucket is first calculated. If bucket B[pos] is empty or the stream key already exists, the item is inserted directly. If different stream keys exist, a replacement or eviction strategy is chosen based on probability.

[0129] Regularly clear outdated data: The lean part regularly clears outdated data through a combination of scanning and real-time cleaning to ensure the real-time and accuracy of the data.

[0130] Stream key frequent item query: Based on the storage of stream keys in the lean part, it efficiently returns frequent items in the current time window, and estimates the distribution of infrequent items through the coarse part.

[0131] The aging data removal strategy combines a global clock mechanism with a window slicing mechanism to effectively support data aging threads and improve the real-time and accuracy of traffic monitoring. Specifically, it includes:

[0132] Timestamp: Whenever a data item arrives, it is assigned a timestamp indicating the time of its arrival. This facilitates subsequent data cleanup and query operations.

[0133] Aging strategy: Through the global clock, the system can determine which data is outdated and decide when to clear the outdated data based on the timestamp to ensure the update and accuracy of the data structure.

[0134] Coordinate data streams: A global clock helps the system process data from different streams, ensuring that data insertion and query operations are based on the same time base.

[0135] The data structure consists of the following parts:

[0136] The lean part and the rough part include:

[0137] The lean part is used to store frequent items within the time window, and the coarse part is used to record the distribution of "fuzzy stream keys".

[0138] The histogram competition mechanism maintains the latest flow histogram interval and regularly evicts the oldest histogram with the smallest counter value to reduce the number of counters in the lean part and improve the overall performance.

[0139] The present invention realizes real-time and accurate measurement of flow-by-flow key distribution through the Hermetis data structure, providing a new solution for network traffic monitoring with efficient memory usage and accurate estimation capabilities.

[0140] The data structure provided by this invention has high accuracy under various types of queries, including:

[0141] Select the optimal model parameters through cross-validation, including learning rate, batch size, etc.;

[0142] Use evaluation indicators such as classification accuracy, recall rate, and F1 score to adjust the model weights to ensure that the model has good accuracy when facing data streams with different characteristics and robustness under low memory configuration.

[0143] The technical solution of the present invention will be further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0144] Example:

[0145] In network measurement, quantile estimation refers to estimating the value corresponding to a quantile in a large amount of traffic data. Quantile estimation is used to describe the distribution characteristics of data. Taking the quantile estimation task as an example, the input data of this embodiment is a real-time traffic data set. Before performing quantile estimation, the data stream is first processed by the Hermetis structure, such as Figure 1 As shown in the figure, the traffic data is decomposed into different flow keys and its frequent items (hot flow) and infrequent items (cold flow) are maintained separately. In this process, Hermetis uses the lean part to efficiently store the frequent flow keys. Figure 2 As shown, the infrequent flow keys are expelled to the coarse part for approximate management, as shown in Figure 3 As shown. Next, the input traffic data is inserted into the stream key by stream key. By calling the Hermetis data insertion algorithm, the stream key and the corresponding count value are inserted into the appropriate bucket in real time, as shown in Figure 4As shown, the system maintains real-time statistics for each stream key as traffic continues to grow. By invoking Hermetis's data insertion algorithm, the stream key and corresponding count value are inserted into the appropriate bucket in real time. This allows the system to maintain real-time statistics for each stream key as traffic continues to grow. During traffic monitoring, the stream key data for both the lean and coarse parts are updated in real time. In the lean part, interval updates are performed regularly to ensure that the stream key count value and timestamp information are updated promptly, preventing data aging from affecting quantile calculations.

[0146] When quantile estimation is required, the data stream query algorithm, as shown in Table 4, first queries the frequent items within the current time window. The lean query algorithm obtains stream key count information. Simultaneously, the approximate count values ​​of infrequent items are obtained from the coarse query algorithm. Combining these data, a weighted average method is used to calculate the required quantile.

[0147] Finally, the results are output. If the estimated quantile exceeds the preset threshold, the system issues an alarm and records the relevant traffic information for subsequent analysis and processing. Conversely, if the quantile is within the normal range, it is allowed to pass through the network normally.

[0148] Through the above steps, this embodiment realizes quantile estimation of real-time traffic data, which not only improves processing efficiency but also ensures accurate monitoring and management of frequent and infrequent flow keys.

[0149] Example 4

[0150] Based on the same inventive concept, this embodiment provides a network flow data storage system based on cold and hot flow separation, including:

[0151] The data structure design module is used to design the Hermetis data structure. The Hermetis data structure consists of a lean part and a coarse part. The lean part is modeled as a hash table with several buckets. Each bucket includes a flow key ID for recording the maintained flow key and a negative counter for recording the flow count that is mapped to the bucket but is different from the flow key ID in the bucket. The coarse part contains several slots, each slot contains several entries, and each entry contains a fingerprint, a time tag, and a count value recording the number of times the data item has been inserted.

[0152] The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

[0153] Since the system described in Example 4 of the present invention is the system used to implement the method for storing network flow data based on cold and hot flow separation in Example 1 of the present invention, those skilled in the art will be able to understand the specific structure and variations of this system based on the method described in Example 1 of the present invention, and therefore will not be described in detail here. All systems used in the method of Example 1 of the present invention are within the scope of protection of the present invention.

[0154] Example 5

[0155] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first embodiment when executing the program.

[0156] Since the computer device described in Example 5 of the present invention is the computer device used to implement the method for storing network flow data based on cold and hot flow separation in Example 1 of the present invention, those skilled in the art will be able to understand the specific structure and variations of the computer device based on the method described in Example 1 of the present invention, and therefore will not be described in detail here. All computer devices used in the method of Example 1 of the present invention fall within the scope of protection of the present invention.

[0157] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, the present invention is intended to include such changes and modifications to the embodiments of the present invention if they fall within the scope of the claims and their equivalents.

Claims

1. A method for storing network flow data based on cold and hot flow separation, characterized in that: include: Design the Hermetis data structure, which includes a lean part and a rough part. The lean part is modeled as a hash table with several buckets, each bucket including the flow key for recording the flow key of maintenance. , used to record the data mapped to the bucket but not the stream key in the bucket The negative counter of different traffic counts, the coarse part contains several slots, each slot contains several entries, each entry contains a fingerprint, a time tag and a count value of the number of times the data item is inserted, each bucket of the lean part also includes multiple histogram interval arrays, where each histogram interval includes a histogram interval ID part, a start time, a last time and multiple counters, and the start time and the last time mark the bucket respectively The timestamps of the earliest and latest arrival streams of the stream key maintained in the hash function Maps stream key IDs to At an index of a bucket; the rough part also includes hash functions and arrays , a hash function is used to map input elements to an array An index position of , and then the frequency is approximately counted through multiple hash functions; The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

2. A network flow data insertion method based on the network flow data storage method based on cold and hot flow separation according to claim 1, characterized in that: include: For a time Arrival items , calculation project Insert index in lean section ; Locate the bucket at the corresponding position according to the insertion index ; If the bucket Maintained stream keys With the project Stream key Same or Bucket If the bucket is empty, the item Insert into the lean section; If the bucket Maintained stream keys With the project Stream key If they are not the same, the bucket will be The flow keys maintained in are replaced by items Stream key , and The originally maintained flow keys are expelled into the coarse part.

3. The network stream data insertion method according to claim 2, wherein: If the project Stream key Unable to save to bucket , then the project Stream key Drive into the rough part.

4. The network stream data insertion method according to claim 2, wherein: The project Inserting into the lean part includes performing interval update operations on the counters by calculating the slice index of the current time. Specifically, when the time interval exceeds the set threshold, all counters are cleared and the timestamps are reset; when the difference between the earliest arrival time of the data and the stream key maintained in the bucket exceeds the time difference threshold, the oldest counter is removed and the time range is updated; otherwise, the current counter is directly updated and the last timestamp is adjusted.

5. The network stream data insertion method according to claim 3, wherein: The project Stream key Expelled into the rough parts, including: According to the hash function Determine the target slot for each row and check whether the slot's timestamp has expired. If so, update it to the current timestamp. If the slot fingerprint is consistent with the data fingerprint, the count value is directly updated; if not, the entry content of the slot is replaced with the second probability.

6. A network flow data query method based on the network flow data storage method based on cold and hot flow separation according to claim 1, characterized in that: include: Check whether the target flow exists in the lean part. If so, accumulate the count value in the corresponding time slice; Access the rough part, calculate the fingerprint and time tag of the traffic data, and search for the corresponding slot row by row. If the slot fingerprint matches, the statistical result is updated to the minimum count value; Add the results of the lean part and the rough part and output the final statistical value.

7. A network flow data storage system based on cold and hot flow separation, characterized in that: include: Data structure design module, used to design the Hermetis data structure, where the Hermetis data structure includes a lean part and a rough part. The lean part is modeled as a hash table with several buckets, each bucket including a flow key for recording maintenance flow keys. , used to record the data mapped to the bucket but not the stream key in the bucket The negative counter of different traffic counts, the coarse part contains several slots, each slot contains several entries, each entry contains a fingerprint, a time tag and a count value of the number of times the data item is inserted, each bucket of the lean part also includes multiple histogram interval arrays, where each histogram interval includes a histogram interval ID part, a start time, a last time and multiple counters, and the start time and the last time mark the bucket respectively The timestamps of the earliest and latest arrival streams of the stream key maintained in the hash function Map stream key IDs to At an index of a bucket; the rough part also includes hash functions and arrays , a hash function is used to map input elements to an array An index position of , and then the frequency is approximately counted through multiple hash functions; The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for storing network flow data based on cold and hot flow separation according to claim 1 is implemented.

Images