An efficient approximate counting method for data stream

Abstract

This invention discloses an efficient approximate counting method for data streams. The method comprises: S1: constructing a swap probability data structure; S2: acquiring the data stream; S3: performing parallel insertion, update, and query operations on the data stream based on the swap probability data structure. Data updates include: starting a data update thread to update data for each array; the data update thread includes performing memory update operations using single-instruction multiple-data (SIMD) technology and performing data exchange operations between memory and secondary storage. This invention achieves parallel data insertion, querying, and updating through the swap probability data structure and the approximate sliding window model built upon this structure, thus not affecting throughput. Simultaneously, the swap probability data structure ensures that when the data stored in the approximate sliding window model reaches a set threshold, a data update thread is started to update data for each bucket. Furthermore, the use of SIMD technology and following set memory update rules for SIMD-compatible arrays reduces memory consumption and improves processing speed, while avoiding blocking insertion and query operations, thereby improving query speed.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and in particular to an efficient approximate counting method for data streams. Background Technology

[0002] In the field of network traffic monitoring, with the increase in data traffic and the acceleration of flow rates, real-time data processing has become particularly critical. The Sketch algorithm, a probabilistic data structure widely used in network measurement, utilizes hash functions and equal-probability methods to map elements to contiguous memory spaces. By sacrificing some accuracy, it achieves low space consumption and extremely fast constant-time processing. This characteristic makes the Sketch algorithm well-suited for estimating large data flows in networks and databases. Furthermore, by combining it with a sliding window model, Sketch can retain only the most recent data and continuously update data frequency information, focusing on the latest changes in the data flow, thus better enabling real-time network data monitoring. For example, the Sketch algorithm can use the IP address of a network packet as a unique identifier, calculate its bucket index using a hash function, and insert it into the array of the corresponding bucket according to the bucket index to count the access frequency of that IP address.

[0003] In practice, the importance of historical data decreases over time, but it still consumes a lot of memory. The existing Sketch algorithm solves the problem of updating expired data by introducing an approximate sliding window model. However, introducing a sliding window model increases memory consumption and operation time. At the same time, the data update process usually blocks insertion and query operations, which in turn affects the efficiency and response speed of the system. Summary of the Invention

[0004] This invention provides an efficient approximate counting method for data streams to overcome the technical problem that the existing Sketch algorithm solves the problem of deleting expired data by introducing a sliding window model. However, the introduction of the sliding window model increases memory consumption and operation time, and usually blocks insertion and query operations, thus affecting the efficiency and response speed of the system.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] An efficient approximate counting method for data streams includes:

[0007] S1: Construct a swap probability data structure and an approximate sliding window model, and initialize the swap probability data structure. The swap probability data structure includes a bucket array, which includes several arrays. Each array includes n data items, and the n data items are used to store n different data occurrence frequencies.

[0008] S2: Get the data stream;

[0009] S3: Based on the exchange probability data structure, perform the insertion, update and query operations of data in the data stream in parallel;

[0010] Data insertion includes: inserting data into the corresponding array data items based on the frequency of occurrence of each data item in the data stream according to the unique identifier of each data item;

[0011] The data update includes: determining whether the frequency of occurrence of the data stored in the approximate sliding window model reaches a set threshold; if so, starting a data update thread to update the data for each array; otherwise, not updating the data.

[0012] The data update thread includes performing memory update operations and data exchange operations between memory and secondary storage using single instruction multiple data stream technology.

[0013] The method of using single instruction multiple data stream (SIM) to perform memory update operations includes:

[0014] Establish a single instruction multiple data stream compatible array;

[0015] The data in several of the arrays are stored in the single instruction multiple data stream compatible array;

[0016] The set memory update rules employ Single Instruction Multiple Data (SMD) technology and update the memory of the SMD compatible array according to the set memory update rules.

[0017] The data query includes: querying the frequency of occurrence of the data to be queried based on the data stored in the exchange probability data structure, and the frequency of occurrence of the data is an approximate counting result.

[0018] Furthermore, each data item is a tuple, and the tuple is represented as follows: <h f (e i .id),E,S,C>;

[0019] Among them, h f (e i .id) represents the compressed identifier; E is the exit counter, used to store the frequency of data that is about to be discarded in the approximate sliding window model; S is the sum counter, used to store the sum of the frequency of data in the first m′-1 sub-windows in the approximate sliding window model; C is the cur counter, used to store the frequency of data in the m′-th sub-window, i.e., the current sub-window, in the approximate sliding window model, and m′ is greater than 0.

[0020] Furthermore, in the data update, the established single instruction multiple data stream compatible array includes: an Aid array, an Aexit array, an Asum array, and an A... arranged vertically in sequence. cur Arrays and memory variables X array;

[0021] Wherein, A id The array is used to store the values ​​of n node identifiers, i.e., node.id; the A exit The array is used to store the values ​​recorded by n exit counters; the A sum The array is used to store the values ​​recorded by n sum counters; the A cur The array is used to store the values ​​recorded by n cur counters; the memory variable X array is used to temporarily store A. cur The values ​​recorded in the array.

[0022] Furthermore, the memory update rules include:

[0023] Using A cur The values ​​stored in the array correspond to the updated values ​​in the memory variable X array;

[0024] Update A sum The value A stored at the i-th position in the array sum [i] is:

[0025] A sum [i] = A sum [i]'-A exit [i]+A cur [i]

[0026] In the formula, A sum [i] represents the updated A sum The value recorded at the i-th position in the array, A sum [i]' represents A before the update sum The value recorded at the i-th position in the array, A exit [i] is A exit The value recorded at the i-th position in the array, A cur [i] is A cur The value recorded at the i-th position in the array, i∈[0,n-1], and n is greater than 1;

[0027] A cur The value stored in the array is set to 0.

[0028] Furthermore, the data update process includes performing data exchange operations between memory and secondary storage, including:

[0029] Slide the sub-windows stored in external storage, discarding data from the first sub-window as frequently as possible, and replacing the value of the previous sub-window with the value of the next sub-window in turn, so as to free up the last sub-window;

[0030] Update the value of the last vacated child window using the value stored in the memory variable X array;

[0031] Update the exit counter value using the value of the first child window.

[0032] Furthermore, in the data insertion process, the data is inserted into the corresponding array data item based on the frequency of occurrence of each data item in the data stream, according to the unique identifier of each data item in the data stream. This includes:

[0033] Let the data stream S consist of a series of real-time arriving data, S = {e1, e2, ..., e...} i ...}, each data e i Each has a unique identifier;

[0034] The data e is calculated using the hash function h. i The hash value h(e) corresponding to the identifier i The hash value h(e) i According to the hash value h(e) i Determine data e i The frequency of occurrence should be inserted into which array of the bucket array;

[0035] Using hash function h f For data e i The identifier is compressed to obtain the compressed identifier h. f (e i .id);

[0036] Traverse data e i The frequency of occurrence of the identifier h to be inserted is compared with the frequency of occurrence of the compressed identifier h. f (e i .id) and the data e i The frequency of occurrence is determined by whether there is a matching node.id in the array of nodes to be inserted; if a matching item exists, i.e., node.id = h f (e i If .id), then the cur counter of that node is incremented by 1, indicating that the data e i It reappears in the current sliding window; if no match is found, the search continues sequentially in the array to be inserted. When the first empty node is found, its identifier is defined as h. f (e i.id), and initialize the cur counter of that node to 1, indicating that the data appears for the first time in the current child window.

[0037] Furthermore, in the data query, the frequency of occurrence of the data to be queried is determined based on the data stored in the exchange probability data structure, including:

[0038] The hash value of the data to be queried is calculated using a hash function h, and the array containing the data to be queried is determined based on the hash value of the data to be queried.

[0039] Using hash function h f The identifier of the data to be queried is compressed to obtain the compressed identifier of the data to be queried.

[0040] Iterate through the array containing the data to be queried, and compare the compressed identifier of the data to be queried with the identifier node.id stored in each node of the array containing the data to be queried. If there is a match, calculate the sum of the values ​​recorded by the sum counter and the cur counter, and return it as the query result, indicating the frequency of the data in the current sliding window; if there is no match, the query result returns 0.

[0041] Furthermore, the constructed approximate sliding window model is a time-based approximate sliding window model, and constructing the time-based approximate sliding window model includes:

[0042] Set the total duration of the sliding window to W;

[0043] Divide the sliding window into m equal sub-windows, and record each sub-window in... The frequency of data occurrence per unit time, and the range of each slide is the length of a sub-window;

[0044] In the time-based approximate sliding window model, the frequency of data occurrence in the [1, m-1]th sub-window is stored in external memory, and the frequency of data occurrence in the mth sub-window is stored in memory, where m is greater than 1.

[0045] Furthermore, the constructed approximate sliding window model is a counting-based approximate sliding window model, and constructing the counting-based approximate sliding window model includes:

[0046] Set the total length of the sliding window's counting mechanism to N;

[0047] Divide the sliding window into p equal sub-windows, and record each sub-window in... The frequency of occurrence of a specific data item in each data item, and the sliding range is the length of a sub-window each time;

[0048] In the approximate sliding window model based on counting, the frequency of data occurrence in the [1, p-1]th sub-window is stored in external memory, and the frequency of data occurrence in the pth sub-window is stored in memory, with p being greater than 1.

[0049] Beneficial Effects: This invention designs a low-memory-consumption exchange probability data structure based on the existing Sketch algorithm. Furthermore, an approximate sliding window model is constructed on top of this data structure, with the window sliding process, data insertion, and querying occurring in parallel, thus not affecting throughput. Simultaneously, the exchange probability data structure ensures that when the data stored in the approximate sliding window model reaches a set threshold, a data update thread is initiated to update data in each bucket. Specifically, a single-instruction multiple-data (SIMD) technique is used to perform the memory update operation, including: establishing a SIMD-compatible array and storing data from several arrays into the SIMD-compatible array; and updating the SIMD-compatible array using SIMD and according to the set memory update rules. This memory update process reduces memory consumption and improves processing speed, while avoiding blocking insertion and query operations, thereby improving query speed. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of an efficient approximate counting method for data streams in this invention;

[0052] Figure 2 This is a schematic diagram of the exchange probability data structure in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the exchange probability data structure based on single instruction multiple data stream technology in an embodiment of the present invention;

[0054] Figure 4 This is a graph showing the ARE index under memory changes based on the CAIDA dataset in an embodiment of the present invention.

[0055] Figure 5 This is a graph showing the SPEED metric under memory changes based on the CAIDA dataset in an embodiment of the present invention.

[0056] Figure 6This is a graph showing the ARE index under the change of the number of sliding windows based on the CAIDA dataset in an embodiment of the present invention.

[0057] Figure 7 This is a graph of the SPEED index under the change of the number of sliding windows based on the CAIDA dataset in an embodiment of the present invention;

[0058] Figure 8 This diagram illustrates the acceleration effect achieved using Single Instruction Multiple Data (SID) technology in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] This embodiment provides an efficient approximate counting method for data streams, such as... Figure 1 As shown, the specific steps include:

[0061] S1: Construct a swap probability data structure and an approximate sliding window model, and initialize the swap probability data structure. The swap probability data structure includes a bucket array, which includes several arrays, i.e., several buckets. Each array includes n data items, and the n data items are used to store n different data occurrence frequencies.

[0062] Specifically, in this embodiment, the swap probability data, i.e., the Swap Sketch, is obtained by improving the existing Sketch algorithm. When constructing the swap probability data structure, the number of bucket arrays b and the length n of each array are first calculated according to the memory size limit; then, n data items, i.e., nodes, are created for each array.

[0063] In a specific embodiment, such as Figure 2 As shown, each data item is a tuple, and the tuple is represented as... <h f (e i .id),E,S,C>;

[0064] Among them, hf(e i.id) represents the compressed identifier; E is the exit counter, used to store the frequency of data that is about to be discarded in the approximate sliding window model, that is, the value in the earliest sub-window in the approximate sliding window model; S is the sum counter, used to store the sum of the frequency of data in the first m′-1 sub-windows in the approximate sliding window model; C is the cur counter, used to store the frequency of data in the m′-th sub-window, that is, the current sub-window, in the approximate sliding window model, and m′ is greater than 0.

[0065] Specifically, the Swap Sketch structure is initialized by assigning initial values ​​to all nodes, and setting the exit counter, sum counter, and cur counter in each node to 0.

[0066] Specifically, in this embodiment, the constructed approximate sliding window model is a time-based approximate sliding window model or a count-based approximate sliding window model, including:

[0067] Constructing the time-based approximate sliding window model includes:

[0068] Set the total duration of the sliding window to W;

[0069] Divide the sliding window into m equal sub-windows, and record each sub-window in... The frequency of data occurrences per unit time, with each slide being the length of a sub-window; each sub-window records the frequency of the same data occurrences in different time periods;

[0070] In the time-based approximate sliding window model, the frequency of data occurrence in the [1, m-1]th sub-window is stored in external memory, and the frequency of data occurrence in the mth sub-window is stored in memory, where m is greater than 1.

[0071] Specifically, for example, in a time-based approximate sliding window model, the sliding window only stores the data of the most recent 5 minutes (total length is 5). The sliding window is divided into 5 sub-windows, and each sub-window records the frequency of data occurrence for 1 minute. When the sliding window reaches a threshold, such as when inserting the 6th minute, the frequency of data occurrence for the 1st minute is about to expire, and a data update operation will be triggered.

[0072] Constructing the count-based approximate sliding window model includes:

[0073] Set the total length of the sliding window's counting mechanism to N;

[0074] Divide the sliding window into p equal sub-windows, and record each sub-window in... The frequency of occurrence of a specific data item within a data item, with each slide's range equal to the length of a sub-window; where, The value is rounded down to the nearest integer.

[0075] In the approximate sliding window model based on counting, the frequency of data occurrence in the [1, p-1]th sub-window is stored in external memory, and the frequency of data occurrence in the pth sub-window is stored in memory, with p being greater than 1.

[0076] S2: Get the data stream;

[0077] S3: Based on the exchange probability data structure, perform parallel insertion, update, and query operations on the data stream;

[0078] Data insertion includes: inserting data into the corresponding array data items based on the frequency of occurrence of each data item in the data stream according to the unique identifier of each data item;

[0079] The data update includes: determining whether the frequency of occurrence of the data stored in the approximate sliding window model reaches a set threshold; if so, starting a data update thread to update the data for each array; otherwise, not updating the data.

[0080] The data update thread includes performing memory update operations and data exchange operations between memory and secondary storage using single instruction multiple data stream technology.

[0081] The method of using single instruction multiple data stream (SIM) to perform memory update operations includes:

[0082] Establish a single instruction multiple data stream compatible array;

[0083] The data in several of the arrays are stored in the single instruction multiple data stream compatible array;

[0084] The set memory update rules employ Single Instruction Multiple Data (SMD) technology and update the SMD compatible array according to the set memory update rules. The memory update results are then stored in each array of the bucket array to obtain the updated result.

[0085] Specifically, if the data stored in the time-based approximate sliding window model reaches the maximum total time length W, then sliding begins, thereby starting a data update thread to update the data for each bucket; if the data stored in the count-based approximate sliding window model reaches the maximum total count length N, then sliding begins, thereby starting a data update thread to update the data for each bucket; in practice, any approximate sliding window model can be selected.

[0086] The data query includes: querying the frequency of occurrence of the data to be queried based on the data stored in the exchange probability data structure, and the frequency of occurrence of the data is an approximate counting result.

[0087] Specifically, data querying refers to querying a data item e given by the user. i For example, when it is necessary to query the frequency of its appearance in the current sliding window, it is necessary to calculate the number of times it appears in the most recent N data items (approximate sliding window model based on counting) or in the data items that have arrived in the most recent W time period (approximate sliding window model based on time).

[0088] Specifically, in this embodiment, data insertion, updating, and querying do not affect each other; query operations can be performed at any time, returning an approximate count of the queried data. Furthermore, to further improve the efficiency of update operations, single-instruction multiple-data (SMIDD) technology is used to accelerate this process, thereby reducing time consumption.

[0089] In a specific embodiment, the data insertion process involves inserting the data into the corresponding array data items based on the unique identifier of each data item in the data stream according to its frequency of occurrence, including:

[0090] Let the data stream S consist of a series of real-time arriving data, S = {e1, e2, ..., e...} i ...}, each data e i Each has a unique identifier;

[0091] The data e is calculated using the hash function h. i The hash value h(e) corresponding to the identifier i The hash value h(e) i That is, the bucket index, based on the hash value h(e) i Determine data e i The frequency of occurrence should be inserted into which array of the bucket array;

[0092] Using hash function h f For data e i The identifier is compressed to obtain the compressed identifier h. f (e i .id);

[0093] Traverse data e i The frequency of occurrence of the identifier h to be inserted is compared with the frequency of occurrence of the compressed identifier h. f (e i .id) and the data e i The frequency of occurrence of the identifier `node.id` in the array to be inserted is used to determine whether there is a matching item for the stored identifier `node.id` of each node in the array to be inserted, thereby determining the data e. i Check if it already exists; if a match exists, i.e., node.id = hf (e i If .id), then the cur counter of that node is incremented by 1, indicating that the data e i It reappears in the current sliding window; if no match is found, the search continues sequentially in the array to be inserted. When the first empty node is found, its identifier is defined as h. f (e i .id), and initialize the cur counter of that node to 1, indicating that the data appears for the first time in the current child window.

[0094] In a specific embodiment, the single instruction multiple data stream compatible array (SIMD compatible array) established during the data update includes: an Aid array, an Aexit array, an Asum array, an Acur array, and a memory variable X array arranged vertically in sequence.

[0095] The Aid array stores the values ​​of n node identifiers, i.e., node.id; the Aexit array stores the values ​​of n exit counters; the Asum array stores the values ​​of n sum counters; the Acur array stores the values ​​of n cur counters; and the memory variable X array temporarily stores the values ​​recorded in the Acur array.

[0096] In a specific embodiment, the memory update rules include:

[0097] Update the values ​​in the memory variable X array using the values ​​stored in the Acur array;

[0098] Update the value Asum[i] stored at the i-th position in the Asum array to:

[0099] Asum[i]=Asum[i]'-Aexit[i]+Acur[i]

[0100] In the formula, Asum[i] is the value of the record at the i-th position in the updated Asum array, Asum[i]' is the value of the record at the i-th position in the Asum array before the update, Aexit[i] is the value of the record at the i-th position in the Aexit array, Acur[i] is the value of the record at the i-th position in the Acur array, and i∈[0,n-1].

[0101] Set the value stored in the Acur array to 0.

[0102] Specifically, Asum[i] is the value recorded at the i-th position in the updated Asum array, that is, the value recorded by the sum counter of the i-th node, and its update formula is:

[0103] node[i].S=node[i].S'-node[i].E+node[i].C

[0104] In the formula, node[i].S is the value recorded by the sum counter of the i-th node after the update, node[i].S' is the value recorded by the sum counter of the i-th node before the update, node[i].E is the value recorded by the exit counter of the i-th node, and node[i].C is the value recorded by the cur counter of the i-th node;

[0105] Specifically, since the value recorded by node[i].S before the update includes the value of the expired child window, when discarding data, it is necessary to subtract the value of the discarded node[i].E and add the value of the current child window to update the value of node[i].S, and store it in the sum counter of the i-th node.

[0106] Specifically, in this embodiment, Single Instruction Multiple Data (SIMD) instructions are used in the memory update operation. To effectively utilize SIMD instructions and accelerate the memory update process, this embodiment establishes a SIMD-compatible array based on the bucket array to better align with SIMD instructions and support parallel processing; that is, each array in the bucket array is converted into a SIMD-compatible array. Figure 3 As shown, the SIMD-compatible array includes the Aid array, Aexit array, Asum array, Acur array, and memory variable X array (not shown in the figure) arranged vertically. The vertical grouping structure of the SIMD-compatible array reflects the structure of the nodes in each array of the bucket array. The data corresponding to the same index in the four arrays of the SIMD-compatible array logically corresponds to a node in each array of the bucket array. The data is effectively aligned according to the attributes of each node, so that the values ​​of the nodes in each array of the bucket array are stored in the corresponding positions in the Aid array, Aexit array, Asum array, and Acur array. For example, if the current node data e2 is in the second position in the bucket array, then its values ​​should be stored in the corresponding second positions in the Aid array, Aexit array, Asum array, and Acur array. This reorganization method avoids the original process of processing each node in each array of buckets individually. Instead, it allows Swap Sketch to better align with SIMD instructions, optimizing the processing of SIMD instructions. This enables simultaneous processing of all node data using SIMD instructions, eliminating the need for sequential memory updates for each node, reducing computation time, and improving data update efficiency. Through this adjustment, Swap Sketch can fully utilize SIMD instructions, thereby reducing the time spent on memory update operations.

[0107] For example, consider Figure 3 In node e2, after the data is reorganized into a SIMD-compatible array, the values ​​of each counter in node e2 are as follows:

[0108] Aid[1]=hf(e2); Aexit[1]=14; Asum[1]=38; Acur[1]=0

[0109] By utilizing SIMD instructions, the entire memory update process becomes highly efficient, enabling parallel processing of all node data with minimal overhead and a small number of instructions. This method significantly reduces the time complexity from O(n) (where n represents the length of array A) to O(1). Unlike traditional methods that process each element one by one, the SIMD-based approach allows computation on multiple nodes simultaneously, thus significantly accelerating the update operation. The parallelism provided by SIMD ensures that all updates can be completed in just a few CPU cycles, resulting in a significant efficiency improvement compared to traditional element-by-element processing methods.

[0110] In a specific embodiment, the data update includes performing data exchange operations between memory and secondary storage, including:

[0111] Slide the sub-windows stored in external storage, discarding data from the first sub-window as frequently as possible, and replacing the value of the previous sub-window with the value of the next sub-window in turn, so as to free up the last sub-window;

[0112] Update the value of the last vacated child window using the value stored in the memory variable X array;

[0113] Update the exit counter value using the value of the first child window.

[0114] Specifically, in the data exchange operation between memory and external storage, the latter data is replaced by the former data to simulate the "sliding" process of the standard sliding window model, thereby discarding the value in the first child window. After sliding, the position of the last child window is vacated to save the value temporarily stored in the memory variable X array, that is, the value recorded by the cur counter, and the data of the current first child window is stored in the exit counter.

[0115] Specifically, the update process is as follows: Figure 2As shown, data e2 is currently stored in the second position of array A, and the frequency of data e2 in the sliding window is: node[1].S + node[1].C = 48. At this time, according to the time-based sliding window setting, the earliest inserted data is about to expire, triggering an update operation. The update process is as follows: First, a memory update operation is performed. The first step is to store the value 5 of node[1].C into the memory variable X. The second step is to update the value of node[1].S to node[1].S' - node[1].E + node[1].C. The third step is to set the value of node[1].C to 0. Finally, the values ​​of each counter in node are changed from...<node[1].E:10,node[1].S:43,node[1].C:5> Become<node[1].E:10,node[1].S:38,node[1].C:0> Secondly, the process of swapping between internal and external memory is executed. Each sub-window stored in external memory is slid along, and the value of the next sub-window replaces the value of the previous sub-window. The value of the first sub-window (10) is discarded, and the position of the last sub-window is vacated to store the value in the memory variable X. Finally, the value of node[1].E is updated to 14 using the value of the first sub-window in external memory. At this point, the data stored in external memory changes from {10,14,6,9,4} to {14,6,9,4,5}, and the value of node in memory changes from...<node[1].E10,node[1].S:38,node[1].C:0> It became<node[1].E:14,node[1].S:38,node[1].C:0> At this point, the data exchange process between internal and external storage is complete.

[0116] In a specific embodiment, the data query, which involves querying the frequency of occurrence of the data to be queried based on the data stored in the exchange probability data structure, includes:

[0117] The hash value of the data to be queried is calculated using a hash function h, and the array containing the data to be queried is determined based on the hash value of the data to be queried.

[0118] Using hash function h f The identifier of the data to be queried is compressed to obtain the compressed identifier of the data to be queried.

[0119] Traverse the array containing the data to be queried, and compare the compressed identifier of the data to be queried with the identifier node.id stored in each node of the array containing the data to be queried. If a match is found, calculate the sum of the values ​​recorded by the sum counter and the cur counter, and return it as the query result, indicating the frequency of the data in the current sliding window; if no match is found, the query result returns 0, indicating that the data has not been observed recently.

[0120] Specifically, such as Figure 2 The image shows examples of insert and query operations. Data insertion typically includes two cases: initial insertion and subsequent insertions based on existing insertions. Inserting e1 is an example of the first case. SwapSketch first calculates h(e1) mod b to locate the data's position in the array, then iterates through array A to find the corresponding node. If the node is empty, it sets node.id = h. f (e1) and node[0].C = 1, thus completing the insertion operation. Here, b is the length of an array. Since the value calculated by h(e1) may exceed this length, causing array access out of bounds, this embodiment needs to perform mod b, i.e., take the remainder b, to ensure that the array index is valid and will not cause access out of bounds. Inserting e3 is an example of the second case. That is, Swap Sketch first calculates h(e3) mod b to locate which array the data is in, and then traverses array A. During the traversal, the node.id of the first two nodes does not match hf(e3), but when traversing to the third node, it is found that the id matches. At this time, node[2].C' = node[2].C + 1, and the insertion operation is completed. For the query operation, the result of Query(e3) is 10 + 2 = 12. This is obtained by finding the matching id by traversing the nodes and then recording the value by the counter. If no matching id is found after traversing the entire array A, the query result returns 0.

[0121] Specifically, since the timing of Swap Sketch's insert and query operations accessing memory space is almost uncontested with the timing of its update operations accessing memory space, these two threads do not affect each other. This mechanism allows Swap Sketch to proactively process expired data without blocking Swap Sketch's insert and query operations, greatly improving the system's throughput.

[0122] Beneficial effects:

[0123] (1) Efficient memory structure design: In order to minimize memory consumption when processing large-scale data, this embodiment designs a simplified memory structure, namely the Swap Sketch structure. This structure stores only the data necessary for query processing in memory. Specifically, the Swap Sketch structure stores only the sum of the data occurrence frequencies in these sub-windows in memory, while storing the data occurrence frequencies in the remaining sub-windows (excluding the current sub-window) in external storage (such as hard disk or distributed storage). This structure optimizes memory usage and avoids unnecessary data occupation, thereby reducing memory consumption while ensuring system performance. This method is particularly effective for scenarios with limited memory and can improve the overall efficiency of the system when processing massive amounts of data.

[0124] (2) Independent Update Thread Design: In this embodiment, a dedicated, independently running update thread is designed. This thread is responsible for managing in-memory data updates and data exchange between memory and external storage. When the storage of a sub-window in memory reaches a preset threshold, the update thread is automatically triggered to perform tasks such as data cleanup and data exchange between memory and external storage. This operation is non-blocking, meaning it will not affect the main thread's insertion operation, thus ensuring high throughput and continuous data inflow. In this way, the system can maintain efficient internal data management while continuously processing new data. The design of the data update thread fully utilizes the system's parallelism, improves processing capacity, and enables stable operation under high load conditions.

[0125] (3) Application of Single Instruction Multiple Data Stream (SIMD) technology to update operations: In this embodiment, SIMD instructions are applied to the memory update step in SwapSketch data update to improve the parallelism of memory updates. In traditional single-threaded updates, each operation is usually processed for a single data item. However, with SIMD instructions, multiple data items can be processed simultaneously within one clock cycle, which greatly improves the efficiency of data updates. SIMD instructions can remove expired data in parallel, reducing computation time and processing latency. Due to the efficient parallel computing capability of SIMD instructions, update operations are no longer a bottleneck, thereby further improving the system's throughput and processing capacity.

[0126] In summary, Swap Sketch incorporates an approximate sliding window model, simulating a sliding window, into the hash-table-based sketch algorithm, enabling it to discard expired data. This significantly reduces memory consumption while maintaining high accuracy. Compared to other traditional sketch algorithms, Swap Sketch effectively reduces memory requirements by simplifying its data storage structure and minimizing the storage of redundant and intermediate data. Furthermore, Swap Sketch's update operations are executed by a separate thread, employing a proactive update strategy to ensure that the update process does not block insertion and query operations, thus maintaining high system throughput.

[0127] Specifically, to verify the effectiveness of this embodiment, the CAIDA dataset was used for experimental verification. The CAIDA dataset is a large network traffic dataset containing CAIDA anonymized internet traffic data from 2018. Each packet in the dataset consists of a 5-tuple, including the source IP address, destination IP address, source port, destination port, and protocol type. There are approximately 27 million packets in total, covering 1.3 million unique entries, suitable for network traffic analysis and performance evaluation research.

[0128] Based on the CAIDA dataset, the performance of five Sketch methods—Mic-CM, Mic-CU, Sl-CM, Sl-CU, and Swap Sketch—was evaluated. Tests were conducted with different memory sizes. The experimental metrics were ARE (mean relative error) and throughput. The formula for ARE is:

[0129]

[0130] In the formula, n' is the number of related terms, and f2 represents the actual frequency. This represents the estimated frequency.

[0131] like Figure 4 As shown, Swap Sketch maintains a low and smooth ARE value across different memory sizes, indicating its good robustness. In contrast, the other four Sketches have higher ARE values ​​and fluctuate significantly with changes in memory size, indicating poor performance under low memory constraints.

[0132] like Figure 5 As shown, the throughput of all Sketches remains relatively stable as memory increases, indicating that memory size has a relatively small impact on throughput. It is worth noting that Swap Sketch consistently maintains the highest throughput, being several times, even ten times, faster than Sl-Sketches and Mic-Sketches.

[0133] like Figure 6 As shown, Swap Sketch maintains a low ARE value across different memory sizes, gradually decreasing as the number of sliding windows increases. In contrast, Mic-CM and Mic-CU show more stable changes, but their ARE values ​​are still higher than SwapSketch, while the ARE values ​​of Sl-CM and Sl-CU gradually increase as the number of sliding windows increases.

[0134] like Figure 7 As shown, the throughput of all Sketches remains relatively stable as the number of sliding windows increases, indicating that the number of sliding windows has a small impact on throughput. Notably, Swap Sketch consistently maintains the highest throughput, consistently exceeding that of Sl-Sketches and Mic-Sketches.

[0135] In summary, Swap Sketch maintains a low accuracy (ARE) thanks to its unique structural design, which allows it to run normally by storing only the necessary data in memory, thus ensuring high accuracy under low memory constraints. Swap Sketch also maintains high throughput due to its unique structural design, which allows two threads to perform data insertion and update operations in parallel, thereby achieving high throughput.

[0136] like Figure 8 As shown, by utilizing SIMD instructions, Swap Sketch's memory update operations can be executed in parallel, making full use of the CPU's vector units, reducing memory access frequency, improving resource utilization, significantly reducing operation cycles, and improving performance by about 2 times.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient approximate counting method for data streams, characterized in that, include: S1: Construct a swap probability data structure and an approximate sliding window model, and initialize the swap probability data structure. The swap probability data structure includes a bucket array, which includes several arrays. Each array includes n data items, and the n data items are used to store n different data occurrence frequencies. S2: Get the data stream; S3: Based on the exchange probability data structure, perform the insertion, update and query operations of data in the data stream in parallel; Data insertion includes: inserting data into the corresponding array data items based on the frequency of occurrence of each data item in the data stream according to the unique identifier of each data item; The data update includes: determining whether the frequency of occurrence of the data stored in the approximate sliding window model reaches a set threshold; if so, starting a data update thread to update the data for each array; otherwise, not updating the data. The data update thread includes performing memory update operations and data exchange operations between memory and secondary storage using single instruction multiple data stream technology. The method of using single instruction multiple data stream (SIM) to perform memory update operations includes: Establish a single instruction multiple data stream compatible array; The data in several of the arrays are stored in the single instruction multiple data stream compatible array; The set memory update rules employ Single Instruction Multiple Data (SMD) technology and update the memory of the SMD compatible array according to the set memory update rules. The data query includes: querying the frequency of occurrence of the data to be queried based on the data stored in the exchange probability data structure, and the frequency of occurrence of the data is an approximate counting result.

2. The efficient approximate counting method for data streams according to claim 1, characterized in that, Each data item is a tuple, and the tuple is represented as follows: <h f (e i .id),E,S,C>; Among them, h f (e i .id) represents the compressed identifier; E is the exit counter, used to store the frequency of data that will be discarded in the approximate sliding window model; S is the sum counter, used to store the sum of the frequencies of data in the first m′-1 sub-windows in the approximate sliding window model; C is the cur counter, used to store the frequency of data in the m′-th sub-window, i.e., the current sub-window, in the approximate sliding window model, and m′ is greater than 0.

3. The efficient approximate counting method for data streams according to claim 2, characterized in that, In the data update, the established Single Instruction Multiple Data Stream Compatible (SMDD) array includes: an Aid array, an Aexit array, an Asum array, and an A... arranged vertically in sequence. cur Arrays and memory variables X array; Wherein, A id The array is used to store the values ​​of n node identifiers, i.e., node.id; the A exit The array is used to store the values ​​recorded by n exit counters; the A sum The array is used to store the values ​​recorded by n sum counters; the A cur The array is used to store the values ​​recorded by n cur counters; the memory variable X array is used to temporarily store A. cur The values ​​recorded in the array.

4. The efficient approximate counting method for data streams according to claim 3, characterized in that, The memory update rules include: Using A cur The values ​​stored in the array correspond to the updated values ​​in the memory variable X array; Update A sum The value A stored at the i-th position in the array sum [i] is: A sum [i]=A sum [i]'-A exit [i]+A cur [i] In the formula, A sum [i] represents the updated A sum The value recorded at the i-th position in the array, A sum [i]' represents A before the update sum The value recorded at the i-th position in the array, A exit [i] is A exit The value recorded at the i-th position in the array, A cur [i] is A cur The value recorded at the i-th position in the array, i∈[0,n-1], and n is greater than 1; A cur The value stored in the array is set to 0.

5. The efficient approximate counting method for data streams according to claim 4, characterized in that, The data update process includes data exchange operations between memory and secondary storage, including: Slide the sub-windows stored in external storage, discarding data from the first sub-window as frequently as possible, and replacing the value of the previous sub-window with the value of the next sub-window in turn, so as to free up the last sub-window; Update the value of the last vacated child window using the value stored in the memory variable X array; Update the exit counter value using the value of the first child window.

6. The efficient approximate counting method for data streams according to claim 5, characterized in that, In the data insertion process, the data frequency is inserted into the corresponding array data item based on the unique identifier of each data item in the data stream, including: Let the data stream S consist of a series of real-time arriving data, S = {e1, e2, ..., e...} i ...}, each data e i Each has a unique identifier; The data e is calculated using the hash function h. i The hash value h(e) corresponding to the identifier i The hash value h(e) i According to the hash value h(e) i Determine data e i The frequency of occurrence should be inserted into which array of the bucket array; Using hash function h f For data e i The identifier is compressed to obtain the compressed identifier h. f (e i .id); Traverse data e i The frequency of occurrence of the identifier h to be inserted is compared with the frequency of occurrence of the compressed identifier h. f (e i .id) and the data e i The frequency of occurrence is determined by whether there is a matching node.id in the array of nodes to be inserted; if a matching item exists, i.e., node.id = h f (e i If .id), then the cur counter of that node is incremented by 1, indicating that the data e i It reappears in the current sliding window; if no match is found, the search continues sequentially in the array to be inserted. When the first empty node is found, its identifier is defined as h. f (e i .id), and initialize the cur counter of that node to 1, indicating that the data appears for the first time in the current child window.

7. The efficient approximate counting method for data streams according to claim 6, characterized in that, The data query, which involves querying the frequency of occurrence of the data to be queried based on the data stored in the exchange probability data structure, includes: The hash value of the data to be queried is calculated using a hash function h, and the array containing the data to be queried is determined based on the hash value of the data to be queried. Using hash function h f The identifier of the data to be queried is compressed to obtain the compressed identifier of the data to be queried. Iterate through the array containing the data to be queried, and compare the compressed identifier of the data to be queried with the identifier node.id stored in each node of the array containing the data to be queried. If there is a match, calculate the sum of the values ​​recorded by the sum counter and the cur counter, and return it as the query result, indicating the frequency of the data in the current sliding window; if there is no match, the query result returns 0.

8. The efficient approximate counting method for data streams according to claim 1, characterized in that, The constructed approximate sliding window model is a time-based approximate sliding window model. Constructing the time-based approximate sliding window model includes: Set the total duration of the sliding window to W; Divide the sliding window into m equal sub-windows, and record each sub-window in... The frequency of data occurrence per unit time, and the range of each slide is the length of a sub-window; In the time-based approximate sliding window model, the frequency of data occurrence in the [1, m-1]th sub-window is stored in external memory, and the frequency of data occurrence in the mth sub-window is stored in memory, where m is greater than 1.

9. The efficient approximate counting method for data streams according to claim 1, characterized in that, The constructed approximate sliding window model is a counting-based approximate sliding window model. Constructing the counting-based approximate sliding window model includes: Set the total length of the sliding window's counting mechanism to N; Divide the sliding window into p equal sub-windows, and record each sub-window in... The frequency of occurrence of a specific data item in each data item, and the sliding range is the length of a sub-window each time; In the approximate sliding window model based on counting, the frequency of data occurrence in the [1, p-1]th sub-window is stored in external memory, and the frequency of data occurrence in the pth sub-window is stored in memory, with p being greater than 1.

Images