A method and system for efficiently detecting persistent items in a small memory environment

Abstract

The present invention relates to the field of data stream processing technology, and in particular to a method and system for efficiently detecting persistent items in a small memory environment. The method comprises the following steps: constructing a two-dimensional array data structure: using a two-dimensional array as a data structure, comprising d rows and l columns, and mapping items to specific counter columns; each bucket comprises two fields, one of which is used to store the key of the persistent item, and the other is used as a flag; executing a conservative update strategy: when inserting an item, calculating d hash functions, mapping the item to the position of different counters in d rows, checking the current values of different counters, selecting the smallest count value min, and marking the row as the target row; when updating the counter, only updating the smallest active counter in the current target row. The present invention improves the efficiency and accuracy of detecting persistent items in a small memory environment by introducing a conservative update strategy and a two-dimensional array data structure.

Description

Technical Field

[0001] The present invention relates to the technical field of data stream processing, and in particular to a method and system for efficiently detecting persistent items in a small memory environment. Background Art

[0002] In the era of big data, the processing and analysis of data streams have become increasingly important. Detecting persistent items in data streams is crucial for tasks such as network monitoring and anomaly detection. Persistent items are items that recur across multiple time windows. However, existing detection methods face numerous challenges when processing large-scale data streams. For example, the Small-Space (SS) method employs a "sample and count" strategy, tracking item frequencies via a hash table. However, this approach requires sampling all items (including non-persistent ones), resulting in low space efficiency. The PIE method, while encoding item IDs using reversible Bloom filters and Raptor codes to identify persistent items, also suffers from space efficiency issues, as it encodes both persistent and non-persistent items in each window, resulting in wasted space. The On-Off Sketch (OO) method, while capable of distinguishing persistent and non-persistent items, suffers from counter overestimation caused by hash collisions under low memory conditions, reducing detection accuracy.

[0003] In summary, we can see that the detection of persistent items in existing data streams has the following technical problems:

[0004] 1. Space efficiency: Existing methods require a large amount of memory resources when processing large-scale data streams, which is unacceptable in memory-constrained environments. For example, the SS method requires sampling all items, and the PIE method requires encoding all items, both of which lead to low space efficiency.

[0005] 2. Detection accuracy: Under low memory conditions, the detection accuracy of existing methods decreases significantly. Due to hash collisions, OO methods may misclassify non-persistent items as persistent items, thus reducing detection accuracy.

[0006] 3. Lack of flexibility: Most existing methods assume a fixed time window size, which is unrealistic in practice. Data stream characteristics (such as speed and skewness) may change over time, necessitating a detection method that can adapt to varying time window sizes. Summary of the Invention

[0007] The present invention provides a method and system for efficiently detecting persistent items in a small memory environment, aiming to solve the problems of low space efficiency, poor detection accuracy and insufficient flexibility in the prior art.

[0008] The present invention provides a method for efficiently detecting persistent items in a small memory environment, comprising the following steps:

[0009] Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to a specific counter column. Each bucket corresponds to a counter. Each bucket contains two fields: one field is used to store the key of the persistent item, and the other field is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "on" and switches to "off" once the item is accessed.

[0010] Execute a conservative update strategy: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current values of different counters, select the smallest count value min among them, and mark the row containing the smallest count value min as the target row; when updating counters, only update the smallest active counter in the current target row.

[0011] As a further improvement of the present invention, executing the conservative update strategy includes:

[0012] Update operation: When updating the counter, check the status of the flag: if the flag in the bucket corresponding to the item is "On", it means that the item appears for the first time, increment the counter value of the target row, and switch its flag to "Off"; if the flag is "Off", it means that the item has appeared before, and the counter value remains unchanged.

[0013] As a further improvement of the present invention, executing the conservative update strategy includes:

[0014] Scan update: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

[0015] As a further improvement of the present invention, the method for efficiently detecting persistent items in a small memory environment further includes performing an insert operation:

[0016] When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, one for each sub-item, map the d sub-items to d rows of a two-dimensional array, one for each row, and map each hash function to a counter position in one of the columns of the corresponding row;

[0017] Check whether the item already exists in the specified bucket. If the item exists, update the corresponding counter of the bucket and increment the count value. If the item does not exist, check the status of the flag bit of the current time window of the specified bucket. If the flag bit is "On", record the minimum count value min in the current row of the specified bucket and mark the row as the target row. If the flag bit is "Off", do nothing.

[0018] Scan all rows. If the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value.

[0019] As a further improvement of the present invention, the method for efficiently detecting persistent items in a small memory environment further includes performing a query operation:

[0020] When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the estimated persistence value of the item.

[0021] The present invention also provides a system for efficiently detecting persistent items in a small memory environment, comprising:

[0022] Two-dimensional array data structure module: This module uses a two-dimensional array as a data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket corresponds to a counter, and each bucket contains two fields: one field is used to store the key of the persistent item, and the other field is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "On" and switches to "Off" once the item is accessed.

[0023] Conservative update strategy execution module: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current values of different counters, select the minimum count value min among them, and mark the row containing the minimum count value min as the target row; when updating counters, only update the smallest active counter in the current target row.

[0024] As a further improvement of the present invention, the conservative update strategy execution module includes:

[0025] Update operation execution module: When updating the counter, check the status of the flag: if the flag in the bucket corresponding to the item is "On", it means that the item appears for the first time, increment the counter value of the target row, and switch its flag to "Off"; if the flag is "Off", it means that the item has appeared before, and the counter value remains unchanged.

[0026] As a further improvement of the present invention, the conservative update strategy execution module includes:

[0027] Scan update execution module: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

[0028] As a further improvement of the present invention, the system for efficiently detecting persistent items in a small memory environment further includes an insertion operation execution module:

[0029] When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, one for each sub-item, map the d sub-items to d rows of a two-dimensional array, one for each row, and map each hash function to a counter position in one of the columns of the corresponding row;

[0030] Check whether the item already exists in the specified bucket. If the item exists, update the corresponding counter of the bucket and increment the count value. If the item does not exist, check the status of the flag bit of the current time window of the specified bucket. If the flag bit is "On", record the minimum count value min in the current row of the specified bucket and mark the row as the target row. If the flag bit is "Off", do nothing.

[0031] Scan all rows. If the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value.

[0032] As a further improvement of the present invention, the system for efficiently detecting persistent items in a small memory environment further includes a query operation execution module:

[0033] When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the estimated persistence value of the item.

[0034] The present invention significantly improves the efficiency and accuracy of persistent item detection in small memory environments by introducing a conservative update strategy and a two-dimensional array data structure. Compared to existing methods, this method not only effectively reduces hash collision errors but also adapts to data streams of varying time window sizes, offering greater flexibility and practicality. Furthermore, its advantages in detection accuracy and space efficiency make it ideal for processing large-scale data streams. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is described in further detail below.

[0036] This paper proposes a method and system for efficiently detecting persistent items in a small memory environment. Hereinafter, we refer to it as LightFinder (LF) to represent the efficient detection of persistent items in a data stream in a small memory environment. LF is primarily implemented through the following technical means:

[0037] (1) Conservative-Update (CU) Strategy: LF adopts the CU strategy to minimize hash collision errors. Specifically, when updating counters, LF only updates the smallest active counter in the current row, thus avoiding error accumulation caused by hash collisions.

[0038] (2) Support for variable time windows: LF can handle time windows of different sizes and adapt to different data stream characteristics, which makes it more flexible and accurate in processing skewed data streams.

[0039] (3) Two-dimensional array data structure: LF uses a two-dimensional array as its data structure. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket contains two fields: one for storing the key of the persistent item and the other as a flag that indicates whether the item has been accessed within the current time window. By using a two-dimensional array to store the key and flag of the persistent item, not only does it improve memory usage efficiency, it also reduces the possibility of counter overflow.

[0040] In LF, each row of a two-dimensional array corresponds to a hash function that maps items to specific counter columns. Each bucket is a cell in the two-dimensional array, and each bucket contains two fields: a key for storing the persistent item and a flag. Counter columns are columns of the two-dimensional array, each containing multiple buckets, each corresponding to a counter.

[0041] The mapping operation is implemented through hash functions. LF uses d hash functions to map items to d rows of a two-dimensional array, with each row corresponding to a hash function mapped to a column in that row, i.e., the counter position.

[0042] The key in the bucket is used to store the identifier of the persistent item. In an insert operation, if the item exists in the specified bucket (that is, the key matches), the counter is updated; in a query operation, the key is used to identify the item and retrieve the corresponding counter.

[0043] Example 1:

[0044] The method for efficiently detecting persistent items in a small memory environment includes the following steps:

[0045] Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket corresponds to a counter. Each bucket contains two fields, one of which is used to store the key of the persistent item, and the other is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "On". Once the item is accessed, the flag is switched to "Off".

[0046] To further optimize memory usage, LF checks the status of a flag when updating the counter. If the flag is "On", it means that the item appears for the first time, and the counter will be incremented; if the flag is "Off", it means that the item has appeared before, and the counter remains unchanged.

[0047] Execute the conservative update (CU) strategy:

[0048] Minimum counter selection: When inserting an item, LF first calculates d hash functions to map the item to the location of d different counters. Then, it checks the current count values of the different counters, selects the minimum count value min, and marks the row containing the minimum count value min as the target row.

[0049] Update operation: When updating a counter, the flag status is checked. If the flag in the bucket corresponding to the item is "On," indicating that the item is appearing for the first time, LF increments the counter value for the target row and switches its flag to "Off." If the flag is "Off," indicating that the item has already appeared, the counter value remains unchanged, thus avoiding the accumulation of errors caused by hash collisions. By updating the counter, which records the frequency of an item in the data stream, LF can track the persistence of an item, that is, its appearance in multiple time windows, by incrementing the counter.

[0050] Scan Update: After completing the update operation, LF scans all rows to check whether there are other rows whose counter value matches the minimum counter value min and whose flag is "On". If so, it switches the flag bit of the row to "Off" and increments the counter value of the row. This strategy ensures that only the minimum counter is updated, thereby minimizing the impact of hash collisions.

[0051] Insert operation:

[0052] When an item with a unique ID arrives, the item is split into d sub-items. LF calculates d hash functions for the item, one for each sub-item, and maps the d sub-items to d rows of a two-dimensional array, one for each row, and each hash function is mapped to a counter position in one of the columns of the corresponding row.

[0053] Then, LF checks whether the item already exists in the specified bucket. If the item exists, it updates the counter corresponding to the bucket and increments the count value. If the item does not exist, LF checks the status of the flag bit of the current time window of the specified bucket. If the flag bit is "On", it records the minimum count value min in the current row of the specified bucket and marks the row as the target row. If the flag bit is "Off", no operation is performed.

[0054] Finally, LF scans all the lines. If the counter value of a line matches the minimum count value min and the flag is "On", the flag is switched to "Off" and the counter value is incremented.

[0055] During an insert operation, if the item doesn't exist in the specified bucket, LF checks the flag status and performs the corresponding update operation, but does not store the item in the bucket. The key field in the bucket is used to store the identifier of the persistent item, but during an insert operation, if the item does not exist, LF is mainly concerned with updating the counter and toggling the flag.

[0056] Query operation:

[0057] When querying the persistence of an item, d hash functions are calculated for the item, the corresponding d counters are retrieved, and the minimum count value min among the d counters is taken as the item's persistence estimate. This approach reduces the impact of hash collisions by taking the minimum value, thus providing a more accurate persistence estimate.

[0058] Example 2:

[0059] A system for efficiently detecting persistent items in a small memory environment of the present invention includes:

[0060] Two-dimensional array data structure module: uses a two-dimensional array as the data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket corresponds to a counter, and each bucket contains two fields, one of which is used to store the key of the persistent item, and the other is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "On" and switches to "Off" once the item is accessed.

[0061] Conservative update strategy execution module: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current values of different counters, select the minimum count value min among them, and mark the row containing the minimum count value min as the target row; when updating counters, only update the smallest active counter in the current target row.

[0062] The conservative update strategy execution module includes:

[0063] Update operation execution module: When updating the counter, check the status of the flag: if the flag in the bucket corresponding to the item is "On", it means that the item appears for the first time, increment the counter value of the target row, and switch its flag to "Off"; if the flag is "Off", it means that the item has appeared before, and the counter value remains unchanged.

[0064] Scan update execution module: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

[0065] Insert operation execution module:

[0066] When an item with a unique ID arrives, it is split into d subitems. D hash functions are calculated for the item, one for each subitem. These d subitems are then mapped to d rows of a two-dimensional array, one for each hash function. Each hash function is mapped to a counter position in one of the columns of the corresponding row. The item's unique ID is used to identify the item itself. During insert operations, the LF uses the hash function to map the item ID to the counter position. During query operations, the hash function is also used to locate the counter to retrieve the item's persistent information.

[0067] Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket and increment the count value. If the item does not exist, check the flag status of the current time window of the specified bucket. If the flag is "On", record the minimum count value min in the current row of the specified bucket and mark the row as the target row. If the flag is "Off", do nothing.

[0068] Scan all rows. If the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value.

[0069] Query operation execution module:

[0070] When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the estimated persistence value of the item.

[0071] Example 3, operation example.

[0072] Example 1: Insert operation.

[0073] Assume there is a data stream containing multiple items, each with a unique ID. To insert an item using LF, the steps are as follows:

[0074] Initialize the LF data structure, set d=3, l=1000, that is, a two-dimensional array with 3 rows and 1000 columns.

[0075] When item e1 arrives, split item e1 into 3 sub-items, calculate 3 hash functions h1(e1), h2(e1), h3(e1), each sub-item corresponds to a hash value, and map the 3 sub-items of item e1 to 3 different counter positions C1[h1(e1)], C2[h2(e1)], C3[h3(e1)] respectively.

[0076] Checks whether e1 already exists in the specified bucket. If not, checks the flag status of the current time window. Assuming the flag is "On", records the minimum count value min in the current row and marks the row as the target row.

[0077] Scan all rows, and if the counter value of a row matches min and the flag is "On", switch the flag to "Off" and increment the counter.

[0078] Example 2: Query operation.

[0079] Suppose we want to query the persistence of project e2. The specific steps are as follows:

[0080] Calculate three hash functions h1(e2), h2(e2), and h3(e2), retrieve the corresponding three counters C1[h1(e2)], C2[h2(e2)], and C3[h3(e2)], and take the minimum value of these three counters as the persistence estimate of e2.

[0081] The method and system for efficiently detecting persistent items in a small memory environment of the present invention have the following advantages:

[0082] (1) Improved space efficiency, enabling accurate detection of persistent items with limited memory resources. LF performs well in small memory environments. For example, when the memory is only 10% of the memory required by On-Off Sketch (OO), LF's mean absolute error (AAE) is almost halved and the average relative error (ARE) is reduced by 10 times. This shows that LF significantly outperforms existing methods in terms of space efficiency.

[0083] (2) Improve detection accuracy and reduce the impact of hash collisions on detection results. LF effectively reduces the impact of hash collisions on detection results through the CU strategy, thereby improving detection accuracy. Under different time window sizes and memory configurations, LF's detection accuracy is superior to existing methods.

[0084] (3) Improved flexibility and adaptability to data streams with different time window sizes. LF supports variable time windows and can adapt to the characteristics of different data streams, which makes it more flexible and adaptable in practical applications.

[0085] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for efficiently detecting persistent items in a small memory environment, characterized in that: The following steps are involved: Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to a specific counter column. Each bucket corresponds to a counter. Each bucket contains two fields: one field is used to store the key of the persistent item, and the other field is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "on" and switches to "off" once the item is accessed. Execute a conservative update strategy: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current count values of different counters, select the minimum count value min, and mark the row with the minimum count value as the target row; when updating counters, only update the minimum active counter in the current target row; It also includes performing insert operations: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, one for each sub-item, map the d sub-items to d rows of a two-dimensional array, one for each row, and map each hash function to a counter position in one of the columns of the corresponding row; Checks whether the item already exists in the specified bucket. If so, updates the corresponding counter of the bucket and increments the count value. If not, checks the status of the flag bit in the current time window of the specified bucket. If the flag bit is "On", records the minimum count value min in the current row of the specified bucket and marks the row as the target row. If the flag bit is "Off", no operation is performed. Scan all rows. If the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value. Executing a conservative update strategy involves: Update operation: When updating the counter, check the status of the flag bit: If the flag bit in the bucket corresponding to the item is "On", it means that the item appears for the first time, and the counter value of the target row is incremented and its flag bit is switched to "Off". If the flag bit is "Off", it means that the item has appeared before, and the counter value remains unchanged. Executing a conservative update strategy involves: Scan update: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

2. The method for efficiently detecting persistent items in a small memory environment according to claim 1, characterized in that: It also includes query operations: When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the estimated persistence value of the item.

3. A system for efficiently detecting persistent items in a small memory environment, characterized in that: include Two-dimensional array data structure module: This module uses a two-dimensional array as a data structure, consisting of d rows and l columns. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket corresponds to a counter, and each bucket contains two fields: one field is used to store the key of the persistent item, and the other field is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "On" and switches to "Off" once the item is accessed. Conservative update strategy execution module: When inserting an item, it calculates d hash functions, maps the item to the position of different counters in d rows, checks the current count values of different counters, selects the minimum count value min, and marks the row with the minimum count value as the target row; when updating counters, it only updates the minimum active counter in the current target row; Also includes the insert operation execution module: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, one for each sub-item, map the d sub-items to d rows of a two-dimensional array, one for each row, and map each hash function to a counter position in one of the columns of the corresponding row; Checks whether the item already exists in the specified bucket. If so, updates the corresponding counter of the bucket and increments the count value. If not, checks the status of the flag bit in the current time window of the specified bucket. If the flag bit is "On", records the minimum count value min in the current row of the specified bucket and marks the row as the target row. If the flag bit is "Off", no operation is performed. Scan all rows. If the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value. The conservative update strategy execution module includes: Update operation execution module: When updating the counter, check the status of the flag bit: If the flag bit in the bucket corresponding to the item is "On", it means that the item appears for the first time, the counter value of the target row is incremented and its flag bit is switched to "Off". If the flag bit is "Off", it means that the item has appeared before, and the counter value remains unchanged; The conservative update strategy execution module includes: Scan update execution module: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

4. The system for efficiently detecting persistent items in a small memory environment according to claim 3, characterized in that: It also includes query operation execution modules: When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the estimated persistence value of the item.