Storage method and system, insertion method and query method of network flow data based on cold and hot flow separation

Abstract

The invention discloses a storage method and system, an insertion method and a query method for network flow data based on cold and hot flow separation. The key idea is that different strategies are adopted for frequent project flow keys and non-frequent project flow keys. In view of diversity of real-time distribution application requirements of stream-by-stream keys, a Hermetis working process comprises two parts: a lean part and a rough part. The lean part is used for storing frequent items in the time window. In the lean part, non-frequent items are eliminated through a competitive selection process, and for a real-time frequent item query task, the lean part greatly improves the query efficiency and reduces the probability of misinformation caused by Hash collision. The rough part is a probabilistic data structure capable of automatically adjusting the size of a counter according to the change of a time window and is used for recording the distribution condition of fuzzy flow keys. Experiments show that compared with other data structures and algorithms, the effects of the data structure in the aspects of accuracy, throughput and memory overhead are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and more specifically, to a storage method and system, an insertion method, and a query method for network flow data based on cold and hot flow separation. Background Art

[0002] In recent years, network measurement and monitoring has become a research hotspot in the network field. Using network measurement to map network traffic conditions and using these measurements as security situation awareness data is an important development direction for threat situation awareness, traffic behavior classification, performance diagnosis and anomaly detection.

[0003] In various types of traffic feature mapping, the measurement of flow key distribution can provide more fine-grained information, such as packet size, latency, etc. The distribution information of flow-level granularity has great potential for improving the performance of anomaly detection, link load adjustment, and user service quality. Although there has been considerable work devoted to achieving accurate traffic distribution query, however, existing stream processing methods do not consider the aging mechanism of data distribution. However, when the traffic feature distribution changes rapidly, the flow key distribution may vary significantly. Summary of the invention

[0004] The present invention proposes a storage method and system, an insertion method, and a query method for network flow data based on cold and hot flow separation, and designs a compact data structure for real-time measurement of per-flow key distribution, aiming to achieve fine-grained per-flow key distribution monitoring. This method can accurately measure and adapt to the distribution status of each flow, and report frequent items in the current time window.

[0005] To achieve the above object, the present invention provides a first aspect of a method for storing network flow data based on cold and hot flow separation, comprising:

[0006] Design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part. The lean part is modeled as a hash table with several buckets, each bucket includes a flow key ID for recording the maintained flow key, and a negative counter for recording the flow count mapped to the bucket but different from the flow key ID in the bucket. The rough part includes several slots, each slot includes several entries, and each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted.

[0007] The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

[0008] In one embodiment, each bucket of the lean part also includes multiple histogram interval arrays, wherein each histogram interval includes a histogram interval ID part, a start time, a last time, and multiple counters, the start time and the last time respectively mark the timestamps of the earliest arrival flow and the latest arrival flow of the flow key maintained in bucket B, and the hash function H maps the flow key ID to an index of l buckets.

[0009] In one implementation, the coarse part includes a hash function and an array A[k]. The hash function is used to map an input element to an index position of the array A[k], and then the frequency is approximately counted through multiple hash functions.

[0010] Based on the same inventive concept, the third aspect of the present invention provides a network flow data insertion method based on the network flow data storage method based on the cold and hot flow separation described in the first aspect, comprising:

[0011] For an item arriving at time t, E =<e,bid,t> , calculate the insertion index pos=H(e) of item E in the lean part; locate the bucket at the corresponding position according to the insertion index

[0012] If the bucket The maintained stream key ID is the same as the stream key ID of project E or bucket If the bucket is empty, then add item E =<e,bid,t> Insert into the lean section;

[0013] If the bucket If the stream key ID maintained is different from the stream key ID of project E, the bucket is moved to the first probability. Replace the stream key maintained in with the stream key ID of project E and The stream keys originally maintained in the .hash are driven out to the coarse part.

[0014] In one embodiment, if the stream key ID of item E cannot be saved to the bucket , then the stream key ID of item E is evicted to the coarse part.

[0015] In one embodiment, it includes calculating the slice index of the current time and performing interval update operations on the counters, specifically: when the time interval exceeds the set threshold, all counters are cleared and the timestamp is reset; when the difference between the earliest arrival time of the data and the stream key maintained in the bucket exceeds the time difference threshold, the oldest counter is removed and the time range is updated; otherwise, the current counter is directly updated and the last timestamp is adjusted.

[0016] In one embodiment, evicting the stream key ID of item E into the coarse portion includes:

[0017] According to the hash function hfp Determine the target slot for each row and check whether the slot's time stamp has expired. If so, update it to the current time stamp.

[0018] If the slot fingerprint is consistent with the data fingerprint, the count value is directly updated; if not, the entry content of the slot is replaced with the second probability.

[0019] Based on the same inventive concept, the third aspect of the present invention provides a network flow data query method based on the network flow data storage method based on the cold and hot flow separation described in the first aspect, comprising:

[0020] Check whether the target flow exists in the lean part. If so, accumulate the count value in the corresponding time slice;

[0021] Access the rough part, by calculating the fingerprint and time tag of the traffic data, find the corresponding slot row by row, if the slot fingerprint matches, update the statistical result to the minimum count value;

[0022] Add the results of the lean part and the rough part and output the final statistical value.

[0023] Based on the same inventive concept, the fourth aspect of the present invention provides a network flow data storage system based on cold and hot flow separation, comprising:

[0024] A data structure design module, used to design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part, the lean part is modeled as a hash table with a number of buckets, each bucket includes a flow key ID for recording the maintained flow key, and a negative counter for recording the flow count mapped to the bucket but different from the flow key ID in the bucket, the rough part includes a number of slots, each slot includes a number of entries, each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted;

[0025] The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

[0026] Based on the same inventive concept, the fifth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for storing network flow data based on cold and hot flow separation as described in the first aspect is implemented.

[0027] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0028] The present invention proposes a storage method for network flow data based on the separation of cold and hot flows. Different strategies are used to store frequent items and infrequent item flow keys. First, the Hermetis data structure is designed, and the lean part is used to store frequent items within the time window. The lean part eliminates infrequent items through a competitive selection process. For real-time frequent item query tasks, the lean part greatly improves the query efficiency and reduces the probability of false positives due to hash collisions. The rough part is a probabilistic data structure that adjusts the counter size according to the change of the time window, which is used to record the distribution status of the cold flow. Taking into account the differences and high skewness of the distribution of different flow keys, the rough part adopts a method of dynamically adjusting the counter size to match the distribution of the data flow. Experiments show that the data structure of the present invention is significantly improved in terms of accuracy, throughput and memory overhead compared with other data structures and algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 is an overview of the overall data structure of Hermetis in an embodiment of the present invention;

[0031] Figure 2 It is a detailed diagram of the data structure of the Hermetis lean part of the embodiment of the present invention;

[0032] Figure 3 It is a detailed diagram of the data structure of the Hermetis coarse part of an embodiment of the present invention;

[0033] Figure 4 It is a flowchart of inserting data stream into Hermetis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In the present invention, the real-time data distribution of the flow key is defined as the distribution of each flow key at the current moment in a fixed-size time sliding window. Since the existing solution cannot clear outdated traffic information in a timely manner, there is a significant gap between the stored data distribution and the actual flow key distribution. Therefore, in view of the real-time measurement requirements of traffic distribution, improving the data aging process of outdated data to more accurately reflect the immediate distribution changes is crucial to maintaining the accuracy of measurement in a dynamic data environment.

[0035] The main inventive concepts of the present invention include: Proposing Hermetis, a compact data structure for real-time measurement of per-stream key distribution. The key idea is to adopt different strategies for hot and cold stream keys. In view of the diversity of application requirements for real-time distribution of per-stream keys, it is desirable to propose a fine-grained per-stream key distribution monitoring method that can accurately measure and adapt to the distribution status of each stream while reporting frequent items in the current time window.

[0036] Specifically, the present invention designs a Hermetis data structure. The workflow of Hermetis includes two parts: a lean part and a rough part. The lean part is used to store frequent items within a time window. The lean part eliminates non-frequent items through a competitive selection process. For real-time frequent item query tasks, the lean part greatly improves the query efficiency and reduces the probability of false positives due to hash collisions. The rough part is a probabilistic data structure that self-adjusts the counter size according to the change of the time window, and is used to record the distribution status of the cold flow (the distribution of non-frequent items). Taking into account the differences and high skewness of the key distribution of different streams, the rough part adopts a method of dynamically adjusting the counter size to match the distribution of the data stream.

[0037] Compared with the prior art, the present invention has innovations and technical advantages in the following aspects:

[0038] 1. Innovations of the new measurement data structure:

[0039] Data structure innovation: Hermetis introduces a dual structure design of lean part and rough part, which can effectively separate frequent items from infrequent items and record frequent item information in real time, thereby achieving high-accuracy stream key distribution estimation and frequent item query. This design improves the response speed to changes in traffic distribution and enhances real-time monitoring capabilities.

[0040] Technical difficulties: Due to the rapid changes in traffic characteristics, existing stream processing methods find it difficult to update stream key distribution in a timely manner, resulting in a significant gap between stored data and actual distribution. The technical difficulty of Hermetis lies in how to use the arrival rules of cold and hot streams to design data structures and timestamp tags with low storage overhead to achieve an efficient and accurate aging data removal mechanism. Secondly, Hermetis considers how to couple data structures and timestamp tags to maintain the compactness of the data structure. Therefore, it is a challenge to design a data structure that can both efficiently remove outdated data and accurately reflect the current stream key distribution.

[0041] 2. Hybrid aging strategy:

[0042] The hybrid aging data clearing strategy under global clock proposed in this paper combines memory expansion mechanism and window slicing mechanism to effectively support data aging thread. Through in-depth analysis of the traffic characteristics of frequent and infrequent items, this strategy achieves high-precision and low-memory overhead data clearing, significantly improving the real-time and accuracy of traffic monitoring.

[0043] Technical advantages: The technical advantages of Hermetis lie in its efficient memory usage and accurate estimation capabilities. In the experiment, Hermetis showed excellent performance, especially when processing small traffic, the error-free rate of its maximum value estimation reached 96.6%. At the same time, it also showed significant memory efficiency in large traffic scenarios, only 3MB was needed to process 27M projects. This makes Hermetis have good scalability and practical application value in network traffic monitoring.

[0044] 3. Asynchronous timeline data clearing:

[0045] In order to reduce the data aging operation error caused by using time tags in the rough part, the present invention uses different time axis offsets for multiple arrays, and each array adopts a completely independent hash map. When querying, the present invention adopts a query strategy similar to CM Sketch (Counted Minimum Sketch) to improve the accuracy of aging data removal.

[0046] 4. Histogram Competition:

[0047] It is observed that most of the frequent items' stream keys are distributed in a few histogram intervals. The lean part always tries to maintain the latest stream histogram interval. Hermetis proposed a histogram competition mechanism to evict the histogram interval with the oldest time label and the smallest counter value to the coarse part to reduce the number of counters in the lean part.

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1

[0050] The present invention discloses a method for storing network flow data based on cold and hot flow separation, comprising:

[0051] Design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part. The lean part is modeled as a hash table with several buckets, each bucket includes a flow key ID for recording the maintained flow key, and a negative counter for recording the flow count mapped to the bucket but different from the flow key ID in the bucket. The rough part includes several slots, each slot includes several entries, and each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted.

[0052] The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

[0053] Specifically, the present invention proposes Hermetis, a compact data structure for real-time measurement of stream-by-stream key distribution. The key idea of ​​the present invention is to adopt different strategies for frequent and infrequent item stream keys. In view of the diversity of real-time distribution application requirements of stream-by-stream keys, a fine-grained stream-by-stream key distribution monitoring method is proposed, which can accurately measure and adapt to the distribution status of each stream, and report the frequent items in the current time window.

[0054] The present invention designs the Hermetis data structure. Figure 1 As shown, the data structure consists of two parts: a “lean part” that records frequent traffic and a “coarse part” that records infrequent traffic.

[0055] The lean part is modeled as a hash table with l buckets. Midstream key K record bucket The negative counter NC records the flow count that is mapped to the bucket but is different from the flow key K in the bucket. Figure 2 As shown, the histogram array (H) contains m histogram intervals. For each histogram interval I, it is split into the histogram interval ID part bid, the start time T first , the last time T last And d+1 counters form Counter[0], Counter[1], ..., Counter[d]. Start time T first and the final time T last Label the buckets separately The timestamps of the earliest and latest arrivals of the stream key K maintained in . The hash function H maps the stream key ID to an index in l buckets: H(·)→[0,l-1].

[0056] like Figure 3As shown, the rough part contains r slots A[0], A[1], ..., A[r-1], each slot consists of c entries. Each entry (fp, time_tag, v) contains a fingerprintfp, which is generated by the hash function h fp Generated about<e,bid> The 8-bit hash value, time tag time_tag and a count value v that records the number of times the data item is inserted. Hash function h k Associated with array A[k], k = 0, 1, ..., r-1. The hash function maps the input element to an index position of array A[k], and then approximates the frequency count through multiple hash functions. When using multiple hash functions, the hash function helps reduce conflicts and increase the accuracy and space efficiency of the data structure.

[0057] Embodiment 2

[0058] Based on the same inventive concept, this embodiment discloses a network stream data insertion method based on the storage method of embodiment 1, comprising:

[0059] For an item arriving at time t, E =<e,bid,t> , calculate the insertion index pos=H(e) of item E in the lean part; locate the bucket at the corresponding position according to the insertion index

[0060] If the bucket The maintained stream key ID is the same as the stream key ID of project E or bucket If the bucket is empty, then add item E =<e,bid,t> Insert into the lean section;

[0061] If the bucket If the stream key ID maintained is different from the stream key ID of project E, the bucket is moved to the first probability. Replace the stream key maintained in with the stream key ID of project E and The stream keys originally maintained in the .hash are driven out to the coarse part.

[0062] Specifically, data insertion includes: when item E=<k,bid,t> When it arrives, calculate the insertion index h(k) of the item in the lean part. Check the status of bucket B: if the stream key maintained in bucket B is k or bucket B is empty, directly insert item E; if the stream key maintained in bucket B is not k, execute the eviction strategy. The eviction strategy determines whether to evict item E to the coarse part. If so, insert the evicted stream key into the coarse part.

[0063] See also Figure 4, insert the Hermetis flow chart for the data flow. Step 1: Check whether E is stored in the lean part. If it is stored (the corresponding decision result is "yes"), then execute step 2: determine whether histogram interval competition occurs. If it is not stored (the corresponding decision result is "no"), then execute step 4: whether to insert E into the coarse part.

[0064] If histogram interval competition occurs (the corresponding decision result is "yes"), step three is executed: whether to remove E to the coarse part.

[0065] A plurality of histogram intervals are maintained in the histogram array. Since an item may have a plurality of different intervals, when there is no empty histogram interval to maintain a new interval, the present invention determines whether to expel a certain interval to the coarse part through the form of histogram interval competition. Step three is specifically as follows: if a histogram competition occurs, first determine whether the items E′ and E maintained in a specific histogram are interchanged according to a certain probability. If they are not interchanged (corresponding to the decision result of "yes"), E is removed to the coarse part; otherwise (corresponding to the decision result of "no"), the original item in this histogram, that is, e′, is inserted into the coarse part, and the new item E is inserted into this position.

[0066] Step 4 is as follows: Check several insertion positions of E in the lean part. If none of these insertion positions has item E, first decide whether to swap the flow k maintained in one of these insertion positions with E according to a certain probability. If not (corresponding to the decision result "yes"), insert E into the rough part; otherwise (corresponding to the decision result "no"), insert the original item in this slot, that is, k, into the rough part, and insert the new item E into this position.

[0067] In the specific implementation process, the algorithm for inserting data into the lean part is shown in Table 1 (Algorithm 1).

[0068] Table 1

[0069]

[0070] Specifically, network traffic data insertion includes the following scenarios:

[0071] Scenario 1.2.1: The stream key e is stored in the bucket B: traverse the histogram intervals of the histogram array part and check whether there is an interval storing the stream value ID bid of e. If there is an interval I0 in bucket B that maintains the histogram interval ID bid, perform the interval update operation shown in Algorithm 3 in Table 3 on I0. Otherwise, if there is an empty interval I1, set the stream value ID maintained by I1 to bid and set T first and T lastSet to t. If there is no empty interval, randomly select an interval I′ with the smallest counter value. Assume that the histogram interval ID maintained in I′ is bid′. Set E′= <e,bid′, t′=I′.T last >Evict to the coarse part and empty the interval I'. Then insert the item E into the interval I'. E' represents an item, e is the stream key of the item E', bid' is the histogram interval ID of E', v' is the count value of E', and t' is the timestamp of E'.

[0072] Scenario 1.2.2: is an empty bucket. At this time, insert the stream key e into the first interval I[0]. Perform the interval update operation shown in Algorithm 3 on interval I[0].

[0073] Scenario 1.2.3: Stores another stream key e′: The count value of the middle flow is V, and the count value of the negative counter is NC. With probability b0 -(V-N-1) The bucket Replace the stream key maintained in with e, and The original flow keys in the middle are driven out into the coarse part.

[0074] Case 1.2.4: If the above operation fails to save e to If e is not found, e is driven to the coarse part, and the coarse part data insertion operation shown in Algorithm 2 is performed, and the count value of the negative counter is updated to NC+v.

[0075] For the three cases 1.2.1, 1.2.2 and 1.2.3, if a data e is inserted into an interval I of the lean part, the interval update operation shown in Algorithm 3 of Table 3 is performed on the interval I. For detailed operations, refer to the data stream aging operation of the lean part.

[0076] For case 1.2.4, the algorithm for inserting data into the rough part is shown in Table 2 (Algorithm 2). By calculating the fingerprint and time tag of the data, the data is inserted into the compact data structure of the rough part. First, the target slot of each row is determined according to the hash function, and the time tag of the slot is checked to see if it has expired. If necessary, it is updated to the current time tag. Then, if the slot fingerprint is consistent with the data fingerprint, the count value is directly updated; if not, the slot content is replaced with a certain probability, thereby achieving probabilistic conflict processing and aging management in the time dimension while ensuring data validity.

[0077] Table 2

[0078]

[0079] In the specific implementation process, the process of Algorithm 2 includes:

[0080] Step 1.4.1, calculate fingerprint fp = h fp (e, bid), time tag time_tag = t / (N / d) mod(d+1). Here N is the time window size, d is the number of sub-windows divided by the time window;

[0081] Step 1.4.2, execute the loop: Forkin[0,r-1]: (here r represents the number of rows of the rough part);

[0082] Step 1.4.3, calculate the slot insertion index idx=h of the kth row of the rough part k (e, bid), candidate slot Slot = Sketch [k] [idx]. Here h k It is the hash function in the data structure in the previous article. As mentioned above, the rough part is modeled as a Sketch, where Sketch refers to the rough part.

[0083] Step 1.4.4, if (time_tag-Slot.time_tag) mod (d+1) == d, then let Slot.time_tag = time_tag; the calculation method of time_tag is described in step 1.4.1, and Slot.time_tag represents the time tag maintained in the candidate slot Slot;

[0084] Step 1.4.5, if Slot.fp = fp, then let Slot.v + = v. The calculation method of fp (fingerprint) is described in step 1.4.1, lot.fp represents the fingerprint maintained in the candidate slot Slot;

[0085] Step 1.4.6, if Slot.fp≠fp, then use the probability Replace the stream recorded by this counter with e.

[0086] After inserting the item E=<e,bid,t> into the lean part, the interval update operation of the lean part will be executed (Algorithm 3), which includes calculating the slice index of the current time and performing interval update operations on the counters: when the time interval exceeds the set threshold, clear all counters and reset the timestamp; when the data is too old, remove the oldest counter and update the time range; otherwise, directly update the current counter and adjust the last timestamp. This process ensures that the counter can efficiently maintain statistical information that changes over time through dynamic clearing and updating.

[0087] In the specific implementation process, the lean part interval update operation can be achieved through the following steps:

[0088] Step 2.1, calculate the time slice index of e

[0089] Step 2.2, if tI.T last >N, then clear all counters in interval I; counter IC[stp]=1, IT first =IT last =t;

[0090] Step 2.3, if Then calculate Clear the oldest clean_step counter of all distance counters IC[stp] in interval I to 0; counter IC[stp] = 1, IT first =tN,IT last =t, where clean_step represents the step size of clearing the counter.

[0091] Step 2.4, except for the first two cases, set the counter IC[stp] = IC[stp] + 1, IT last =t.

[0092] Please refer to Table 3 for the specific algorithm.

[0093] Table 3

[0094]

[0095]

[0096] Embodiment 3

[0097] Based on the same inventive concept, this embodiment discloses a network flow data query method based on the network flow data storage method based on cold and hot flow separation described in Embodiment 1, including:

[0098] Check whether the target flow exists in the lean part. If so, accumulate the count value in the corresponding time slice;

[0099] Access the rough part, by calculating the fingerprint and time tag of the traffic data, find the corresponding slot row by row, if the slot fingerprint matches, update the statistical result to the minimum count value;

[0100] Add the results of the lean part and the rough part and output the final statistical value.

[0101] Specifically, the data query method calculates the statistical value of the target traffic by accessing the data structures of the lean part and the rough part at the same time. First, check whether the target traffic exists in the lean part. If so, accumulate the count value in the corresponding time slice; then access the rough part, calculate the fingerprint and time label of the traffic, and find the corresponding slot row by row. If the slot fingerprint matches, update the statistical result to the minimum count value; finally, add the results of the lean part and the rough part to output the final statistical value. This algorithm combines the accuracy of the lean part and the compactness of the rough part, improving query efficiency and storage utilization.

[0102] The data query method can be implemented through the following steps:

[0103] Step 3.1, let result = 0. result represents the query result of the project, initialized to 0;

[0104] Step 3.2, if the stream key e is in the lean part, query the value of the stream key e in the lean part:

[0105] Step 3.2.1, calculate the time slice index of e

[0106] Step 3.2.2, add result to the d buckets closest to IC[step]: IC[(stp)mod(d+1)], IC[(stp-1)mod(d+1)], ..., IC[(stp-d+1)mod(d+1)] with the count values:

[0107] Step 3.3, query the value of stream key e in the rough part:

[0108] Step 3.3.1, calculate the fingerprint fp and time tag time_tag,

[0109] fp=h fp (e, bid), time_tag=t / (N / d)mod(d+1);

[0110] Step 3.3.2, set coarse_result = ∞. coarse_result represents the query result of the rough part, which is initialized to ∞;

[0111] Step 3.3.3, execute the loop Forkin[0,r-1] where r represents the number of rows in the rough part;

[0112] Step 3.3.4, calculate Candidate slot Slot = Sketch [k] [idx];

[0113] Step 3.3.5, if Slot.fp == fp, then coarse_result = min(coarse_result, Slot.v);

[0114] Step 3.4, if coarse_reslut≠∞, reslut=result+coarse_result;

[0115] Step 3.5, output result.

[0116] The specific algorithm implementation is shown in Table 4.

[0117] Table 4

[0118]

[0119]

[0120] In one embodiment, the integrated data storage, insertion and query specifically include the following steps:

[0121] S1: Determine the data insertion position. For each piece of data that arrives, we first calculate its storage position in the lean part based on the content of the data (stream key).

[0122] S2: Determine the status of the storage location. If the location already stores information of the same stream, traverse all storage areas at this location to find out whether there is an ID space that has already stored this stream, and apply different processing strategies to the results returned by the search.

[0123] S3: If the space to store the ID of this stream is found, the time and count information of this interval will be updated; if the space to store the ID of this stream is not found, but there is an empty storage interval, this interval will be allocated to the new data, and the relevant time record will be set; if there is no free storage interval, the interval with the least number of records will be found, and the data in this space will be expelled, and the expelled data will be placed in the coarse part, and the new data will be inserted into this position if it is empty. Insert the new data into the first storage interval and update the time record if the position stores the information of other streams. Determine whether to replace the existing stream based on probability and put the new data into the lean part. If a replacement occurs, the original stream data will be moved to the coarse part; otherwise, the new data will be directly placed in the coarse part.

[0124] S4: After the data is successfully inserted into a storage interval of the lean part, the count and time of the interval are updated to reflect the latest data situation.

[0125] S5: Perform "aging" processing on the data stored in the lean part according to time. If the current time is more than a certain time away from the last update of the data, the system will clear all counters in this interval; if the earliest time of the data is significantly different from the current time, the oldest counters will be cleared; if the current data does not exceed the aging time threshold, the counter will be directly increased and the last access time will be updated.

[0126] S6: When a piece of data needs to be queried, the system will first search from the lean part. If the corresponding stream data is found in the lean part, the count values ​​in the most recent time slices will be selected and added up as the query result. If there is no data in the lean part, the system will try to find the record of the stream from the rough part, calculate a minimum count value, add it to the query result, and finally obtain the query value of the data by combining the query results of the lean part and the rough part.

[0127] Among them, data insertion includes:

[0128] Data stream insertion: When item E arrives, first calculate the insertion index pos in the lean part. If bucket B[pos] is empty or the stream key already exists, insert the item directly; if different stream keys exist, choose the replacement or eviction strategy based on probability.

[0129] Regularly clear outdated data: The lean part regularly clears outdated data through a combination of scanning and real-time cleaning to ensure the real-time and accuracy of the data.

[0130] Stream key frequent item query: Based on the storage of stream keys in the lean part, frequent items in the current time window are efficiently returned, and the distribution of infrequent items is estimated through the coarse part.

[0131] Among them, the aging data removal strategy combines the global clock mechanism and the window slicing mechanism to effectively support the data aging thread and improve the real-time and accuracy of traffic monitoring. Specifically, it includes:

[0132] Timestamp: Whenever a data item arrives, it is assigned a timestamp indicating the arrival time of the data item. This helps with subsequent data cleanup and query operations.

[0133] Aging strategy: Through the global clock, the system can determine which data is outdated and decide when to clear the outdated data based on the timestamp to ensure the update and accuracy of the data structure.

[0134] Coordinate data streams: A global clock helps the system process data from different streams, ensuring that operations are based on the same time reference when inserting and querying data.

[0135] The data structure consists of the following parts:

[0136] Lean part and rough part, specifically including:

[0137] The lean part is used to store frequent items within the time window, and the coarse part is used to record the distribution of the “fuzzy stream key”.

[0138] The histogram competition mechanism maintains the latest flow histogram interval and regularly evicts the oldest histogram with the smallest counter value to reduce the number of counters in the lean part and improve the overall performance.

[0139] The present invention realizes real-time and accurate measurement of stream-by-stream key distribution through the Hermetis data structure, provides a new solution for network traffic monitoring, and has efficient memory usage and accurate estimation capabilities.

[0140] The data structure provided by the present invention has high accuracy under various types of queries, including:

[0141] Select the optimal model parameters through cross-validation, including learning rate, batch size, etc.;

[0142] The model weights are adjusted using evaluation indicators such as classification accuracy, recall rate, and F1 score to ensure that the model has good accuracy and robustness under low memory configuration when facing data streams with different characteristics.

[0143] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0144] Example:

[0145] In network measurement, quantile estimation refers to estimating the value corresponding to a quantile in a large amount of traffic data. Quantile estimation is used to describe the distribution characteristics of data. Taking the quantile estimation task as an example, the input data of this embodiment is a real-time traffic data set. Before performing quantile estimation, first, the data stream is processed by the Hermetis structure, such as Figure 1 As shown in the figure, the traffic data is decomposed into different flow keys, and its frequent items (hot flow) and infrequent items (cold flow) are maintained separately. In this process, Hermetis uses the lean part to efficiently store frequent flow keys. Figure 2 As shown, the infrequent flow keys are driven to the coarse part for approximate management, such as Figure 3 As shown. Next, the input traffic data is inserted into the stream key by stream key. By calling the Hermetis data insertion algorithm, the stream key and the corresponding count value are inserted into the appropriate bucket in real time, as shown in Figure 4As shown, the system can maintain real-time statistics for each stream key in the growing traffic. By calling the data insertion algorithm of Hermetis, the stream key and the corresponding count value are inserted into the appropriate bucket in real time. In this way, the system can maintain real-time statistics for each stream key in the growing traffic. During the traffic monitoring process, the stream key data of the lean part and the coarse part are updated in real time. In the lean part, the interval update operation is performed regularly to ensure that the count value and timestamp information of the stream key are updated in time to avoid affecting the calculation of the quantile due to data aging.

[0146] When quantile estimation is required, the data stream query algorithm is shown in Table 4. The system first queries the frequent items in the current time window and obtains the count information of the stream key through the query algorithm of the lean part. At the same time, the approximate count values ​​of the infrequent items are obtained from the rough part. Combining these data, the required quantile is calculated using the weighted average method.

[0147] Finally, the results are output. If the estimated quantile exceeds the preset threshold, the system will issue an alarm and record the relevant traffic information for subsequent analysis and processing. On the contrary, if the quantile is within the normal range, it will be allowed to pass through the network normally.

[0148] Through the above steps, this embodiment realizes quantile estimation of real-time traffic data, which not only improves processing efficiency but also ensures accurate monitoring and management of frequent and infrequent flow keys.

[0149] Embodiment 4

[0150] Based on the same inventive concept, this embodiment provides a network flow data storage system based on cold and hot flow separation, including:

[0151] A data structure design module, used to design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part, the lean part is modeled as a hash table with a number of buckets, each bucket includes a flow key ID for recording the maintained flow key, and a negative counter for recording the flow count mapped to the bucket but different from the flow key ID in the bucket, the rough part includes a number of slots, each slot includes a number of entries, each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted;

[0152] The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

[0153] Since the system introduced in the fourth embodiment of the present invention is a system used to implement the storage method of network flow data based on cold and hot flow separation in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0154] Embodiment 5

[0155] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in Embodiment 1 when executing the program.

[0156] Since the computer device introduced in the fifth embodiment of the present invention is a computer device used to implement the method for storing network flow data based on cold and hot flow separation in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0157] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic creative concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for storing network flow data based on cold and hot flow separation, characterized in that: include: Design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part. The lean part is modeled as a hash table with several buckets, each bucket includes a flow key ID for recording the maintained flow key, and a negative counter for recording the flow count mapped to the bucket but different from the flow key ID in the bucket. The rough part includes several slots, each slot includes several entries, and each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted. The lean part of the Hermetis data structure is used to record frequent flows, and the coarse part of the Hermetis data structure is used to record infrequent flows.

2. The method for storing network flow data based on cold and hot flow separation according to claim 1, characterized in that: Each bucket of the lean part also includes multiple histogram interval arrays, wherein each histogram interval includes a histogram interval ID part, a start time, a last time and multiple counters, the start time and the last time respectively mark the timestamps of the earliest arrival flow and the latest arrival flow of the flow key maintained in bucket B, and the hash function H maps the flow key ID to an index of l buckets.

3. The method for storing network flow data based on cold and hot flow separation according to claim 1, characterized in that: The rough part includes a hash function and an array A[k]. The hash function is used to map the input element to an index position of the array A[k], and then the frequency is approximately counted through multiple hash functions.

4. A network flow data insertion method based on the network flow data storage method based on cold and hot flow separation according to any one of claims 1 to 3, characterized in that: include: For an item arriving at time t, E =<e,bid,t> , calculate the insertion index of item E in the lean part pos=H(e); Locate the bucket at the corresponding position according to the insertion index If the bucket The maintained stream key ID is the same as the stream key ID of project E or bucket If the bucket is empty, then add item E =<e,bid,t> Insert into the lean section; If the bucket If the stream key ID maintained is different from the stream key ID of project E, the bucket is moved to the first probability. Replace the stream key maintained in with the stream key ID of project E and The stream keys originally maintained in the .hash are driven out to the coarse part.

5. The network stream data insertion method according to claim 4, characterized in that: If the stream key ID of project E cannot be saved to the bucket , then the stream key ID of item E is evicted to the coarse part.

6. The network stream data insertion method according to claim 4, characterized in that: Set item E =<e,bid,t> Insertion into the lean part includes interval update operations on counters by calculating the slice index of the current time. Specifically, when the time interval exceeds the set threshold, all counters are cleared and the timestamp is reset; when the difference between the earliest arrival time of the data and the stream key maintained in the bucket exceeds the time difference threshold, the oldest counter is removed and the time range is updated; otherwise, the current counter is directly updated and the last timestamp is adjusted.

7. The network stream data insertion method according to claim 5, characterized in that: Evict the stream key ID of item E into the coarse part, including: According to the hash function h fp Determine the target slot for each row and check whether the slot's time stamp has expired. If so, update it to the current time stamp. If the slot fingerprint is consistent with the data fingerprint, the count value is directly updated; if not, the entry content of the slot is replaced with the second probability.

8. A network flow data query method based on the network flow data storage method based on cold and hot flow separation according to any one of claims 1 to 3, characterized in that: include: Check whether the target flow exists in the lean part. If so, accumulate the count value in the corresponding time slice; Access the rough part, by calculating the fingerprint and time tag of the traffic data, find the corresponding slot row by row, if the slot fingerprint matches, update the statistical result to the minimum count value; Add the results of the lean part and the rough part and output the final statistical value.

9. A network flow data storage system based on cold and hot flow separation, characterized in that: include: A data structure design module, used to design a Hermetis data structure, wherein the Hermetis data structure includes a lean part and a rough part, the lean part is modeled as a hash table with a number of buckets, each bucket includes a flow key ID for recording a maintained flow key, and a negative counter for recording a flow count mapped to the bucket but different from the flow key ID in the bucket, the rough part includes a number of slots, each slot includes a number of entries, each entry includes a fingerprint, a time tag, and a count value recording the number of times the data item is inserted; The flow data storage module is used to record frequent flows using the lean part of the Hermetis data structure and to record infrequent flows using the coarse part of the Hermetis data structure.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for storing network flow data based on cold and hot flow separation as described in any one of claims 1 to 3 is implemented.

Images