Method and system for efficiently detecting persistent items in small memory environment

Abstract

The invention relates to the technical field of data stream processing, in particular to a method and system for efficiently detecting persistent items in a small memory environment. The method comprises the following steps of: constructing a two-dimensional array data structure: adopting a two-dimensional array as a data structure which comprises d rows and l columns, and mapping items to a specific counter column; each bucket comprises two fields, one field is used for storing a key of a persistent item, and the other field is used as a flag bit; executing a conservative updating strategy: when an item is inserted, calculating d hash functions, mapping the item to the positions of different counters in d rows, checking the current values of the different counters, selecting the minimum count value min, and marking the row as a target row; and when the counters are updated, only the smallest active counter in the current target row is updated. According to the method, by introducing a conservative updating strategy and a two-dimensional array data structure, the efficiency and precision of detecting the persistent items in a small memory environment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data stream processing, and particularly to a method and system for efficiently detecting persistent items in a small memory environment. Background Art

[0002] In the big data era, the processing and analysis of data streams have become particularly important. The detection of persistent items in a data stream is crucial for tasks such as network monitoring and anomaly detection. Persistent items refer to items that repeatedly appear in multiple time windows. However, existing detection methods face many challenges when dealing with large-scale data streams. For example, the Small-Space (SS) method adopts a "sampling and counting" strategy, tracking item frequencies through a hash table, but this method requires sampling all items (including non-persistent items), resulting in low space efficiency. Although the PIE method encodes item IDs through a reversible Bloom filter and Raptor codes to identify persistent items, it also has a space efficiency problem because it encodes both persistent and non-persistent items in each window, causing space waste. The On-Off Sketch (OO) method can distinguish between persistent and non-persistent items, but in a low-memory condition, due to the problem of overestimating counters caused by hash collisions, the detection accuracy is reduced.

[0003] In summary, the following technical problems exist in the detection of persistent items in existing data streams: 1. Space efficiency problem: When existing methods deal with large-scale data streams, they require a large amount of memory resources, which is unacceptable in memory-constrained environments. For example, the SS method needs to sample all items, and the PIE method needs to encode all items, both of which result in low space efficiency.

[0004] 2. Detection accuracy problem: In a low-memory condition, the detection accuracy of existing methods will decrease significantly. Due to the hash collision problem, the OO method may misjudge non-persistent items as persistent items, thus reducing the detection accuracy.

[0005] 3. Lack of flexibility: Most existing methods assume that the time window size is fixed, which is unrealistic in practical applications. The characteristics of data streams (such as speed, skewness, etc.) may change over time, so a detection method that can adapt to different time window sizes is needed. Summary of the Invention

[0006] The present invention provides a method and system for efficiently detecting persistent items in a small memory environment, aiming to solve the problems of low space efficiency, poor detection accuracy, and lack of flexibility existing in the prior art.

[0007] The present invention provides a method for efficiently detecting persistent items in a small memory environment, comprising the following steps: Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, which contains d rows and l columns. Each row corresponds to a hash function that maps an item to a specific counter column; each bucket corresponds to a counter, and each bucket contains two fields. One field is used to store the key of the persistent item, and the other field is used as a flag bit to indicate whether the item has been accessed within the current time window; the flag bit is initially set to "On", and once the item is accessed, the flag bit is switched to "Off". Execute a conservative update strategy: When inserting an item, calculate d hash functions to map the item to d different counter positions in the rows, check the current values of the different counters, select the smallest count value min among them, and mark the row containing the smallest count value min as the target row; when updating the counter, only update the smallest active counter in the current target row.

[0008] As a further improvement of the present invention, executing the conservative update strategy includes: Update operation: When updating the counter, check the status of the flag bit: If the flag bit in the bucket corresponding to the item is "On", it means the item appears for the first time. Increment the counter value of the target row and switch its flag bit to "Off"; if the flag bit is "Off", it means the item has appeared before, and the counter value remains unchanged.

[0009] As a further improvement of the present invention, executing the conservative update strategy includes: Scan and update: After completing the update operation, scan all rows to check if there are other rows whose counter values match the smallest count value min and the flag bit is "On". If so, switch the flag bit of that row to "Off" and increment the counter value of that row.

[0010] As a further improvement of the present invention, the method for efficiently detecting persistent items in a small memory environment further includes performing an insertion operation: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions of the item. Each sub-item corresponds to a hash value, and map the d sub-items to d rows in the two-dimensional array respectively. Each row corresponds to a hash function, and each hash function maps to the position of a counter in one column of the corresponding row; Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket, and increment the count value; if the item does not exist, check the status of the flag bit in the current time window of the specified bucket. If the flag bit is "On", record the smallest count value min in the current row of the specified bucket and mark that row as the target row; if the flag bit is "Off", do nothing. Scan all rows. If the counter value of a certain row matches the minimum counter value min and the flag bit is "On", then switch the flag bit to "Off" and increment the counter value.

[0011] As a further improvement of the present invention, a method for efficiently detecting persistent items in a small memory environment further includes performing a query operation: When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the minimum counter value min among the d counters as the persistence estimation value of the item.

[0012] The present invention also provides a system for efficiently detecting persistent items in a small memory environment, including Two-dimensional array data structure module: Using a two-dimensional array as the data structure, which contains d rows and l columns. Each row corresponds to a hash function and maps an item to a specific counter column; each bucket corresponds to a counter, and each bucket contains two fields. One field is used to store the key of the persistent item, and the other field is used as a flag bit to indicate whether the item has been accessed within the current time window; the flag bit is initially set to "On", and once the item is accessed, the flag bit is switched to "Off". Conservative update strategy execution module: When inserting an item, calculate d hash functions, map the item to the positions of d different counters in different rows, check the current values of different counters, select the minimum counter value min among them, and mark the row containing the minimum counter value min as the target row; when updating the counter, only update the minimum active counter in the current target row.

[0013] As a further improvement of the present invention, the conservative update strategy execution module includes: Update operation execution module: When updating the counter, check the status of the flag bit: If the flag bit in the bucket corresponding to the item is "On", it means the item appears for the first time, increment the counter value of the target row and switch its flag bit to "Off"; if the flag bit is "Off", it means the item has appeared before, and the counter value remains unchanged.

[0014] As a further improvement of the present invention, the conservative update strategy execution module includes: Scan update execution module: After the update operation is completed, scan all rows to check whether there are other rows whose counter values match the minimum counter value min and the flag bit is "On". If so, switch the flag bit of this row to "Off" and increment the counter value of this row.

[0015] As a further improvement of the present invention, the system for efficiently detecting persistent items in a small memory environment further includes an insert operation execution module: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, with each sub-item corresponding to a hash value. Map the d sub-items to d rows of a two-dimensional array respectively, where each row corresponds to a hash function, and each hash function maps to the position of a counter in a corresponding column of the row; Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket, and increment the count value. If the item does not exist, check the flag status of the current time window of the specified bucket. If the flag is "On", record the minimum count value min in the current row of the specified bucket and mark this row as the target row. If the flag is "Off", do nothing; Scan all rows. If the counter value of a certain row matches the minimum count value min and the flag is "On", then switch the flag to "Off" and increment the counter value.

[0016] As a further improvement of the present invention, a system for efficiently detecting persistent items in a small memory environment further includes a query operation execution module: When querying the persistence of an item, calculate d hash functions for the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the persistence estimation value of the item.

[0017] The beneficial effects of the present invention are as follows: By introducing a conservative update strategy and a two-dimensional array data structure, the present invention significantly improves the efficiency and accuracy of detecting persistent items in a small memory environment. Compared with existing methods, the present invention can not only effectively reduce hash collision errors, but also adapt to data streams of different time window sizes, with higher flexibility and practicality. In addition, the advantages of the present invention in terms of detection accuracy and space efficiency make it an ideal choice for processing large-scale data streams. Detailed implementation manners

[0018] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below.

[0019] The present invention proposes a method and system for efficiently detecting persistent items in a small memory environment. Hereinafter, the name LightFinder (LF) is used to represent efficiently detecting persistent items in a data stream in a small memory environment. LF is mainly implemented by the following technical means: (1) Conservative-Update (CU) strategy: LF adopts the CU strategy to minimize hash collision errors. Specifically, when LF updates the counter, it only updates the smallest active counter in the current row, thus avoiding the accumulation of errors caused by hash collisions.

[0020] (2) Support for variable time windows: LF can handle time windows of different sizes and adapt to different data stream characteristics, which makes it more flexible and accurate in processing skewed data streams.

[0021] (3) Two-dimensional array data structure: LF uses a two-dimensional array as the data structure. Each row corresponds to a hash function that maps items to specific counter columns. Each bucket contains two fields: one for storing the key of the persistent item and the other as a flag bit indicating whether the item has been accessed within the current time window. By using a two-dimensional array to store the keys and flag bits of persistent items, not only is the memory usage efficiency improved, but the possibility of counter overflow is also reduced.

[0022] In LF, each row of the two-dimensional array corresponds to a hash function that maps items to specific counter columns. Each bucket, which is a cell in the two-dimensional array, contains two fields: one for storing the key of the persistent item and the other as a flag bit. The counter column refers to the column of the two-dimensional array, and each column contains multiple buckets, with each bucket corresponding to a counter.

[0023] The mapping operation is implemented through hash functions. LF uses d hash functions to map items to d rows of the two-dimensional array. Each row corresponds to a hash function and maps to a certain column in that row, which is the counter position.

[0024] The key in the bucket is used to store the identifier of the persistent item. In the insertion operation, if the item exists in the specified bucket (i.e., the keys match), the counter is updated; in the query operation, the item is identified by the key and the corresponding counter is retrieved.

[0025] Example 1:

[0026] A method for efficiently detecting persistent items in a small memory environment includes the following steps: Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, which contains d rows and l columns. Each row corresponds to a hash function that maps items to specific counter columns; each bucket corresponds to a counter, and each bucket contains two fields, where one field is used to store the key of the persistent item and the other field is used as a flag bit to indicate whether the item has been accessed within the current time window; the flag bit is initially set to "On", and once the item is accessed, the flag bit is switched to "Off".

[0027] To further optimize memory usage, LF checks the status of the flag bit when updating the counter. If the flag bit is "On", it means the item appears for the first time and the counter will be incremented; if the flag bit is "Off", it means the item has appeared before and the counter remains unchanged.

[0028] Execute the conservative update (CU) strategy: Minimum counter selection: When inserting an item, LF first calculates d hash functions to map the item to the positions of d different counters. Then, it checks the current counter values of the different counters, selects the minimum counter value min among them, and marks the row containing the minimum counter value min as the target row.

[0029] Update operation: When updating the counter, check the status of the flag bit: If the flag bit in the bucket corresponding to the item is "On", indicating that the item appears for the first time, LF increments the counter value of the target row and switches its flag bit to "Off"; If the flag bit is "Off", indicating that the item has appeared before, the counter value remains unchanged, thus avoiding the error accumulation caused by hash collisions. By updating the counter, it is used to record the appearance frequency of the item in the data stream. By incrementing the counter, LF can track the persistence of the item, that is, the appearance of the item in multiple time windows.

[0030] Scan and update: After completing the update operation, LF scans all rows to check if there are other rows whose counter values match the minimum counter value min and the flag bit is "On". If so, switch the flag bit of this row to "Off" and increment the counter value of this row. This strategy ensures that only the minimum counter is updated, thus minimizing the impact of hash collisions.

[0031] Insert operation: When an item with a unique ID arrives, the item is split into d sub-items. LF calculates the d hash functions of the item, each sub-item corresponds to a hash value, and maps the d sub-items to d rows of the two-dimensional array respectively. Each row corresponds to a hash function, and each hash function maps to the position of a counter in a column corresponding to the row; Then, LF checks if the item already exists in the specified bucket. If the item exists, update the counter corresponding to this bucket and increment the counter value; If the item does not exist, LF checks the status of the flag bit in the current time window of the specified bucket. If the flag bit is "On", record the minimum counter value min in the current row of the specified bucket and mark this row as the target row; If the flag bit is "Off", do nothing; Finally, LF scans all rows. If the counter value of a certain row matches the minimum counter value min and the flag bit is "On", switch the flag bit to "Off" and increment the counter value.

[0032] In the insertion operation, if the item does not exist in the specified bucket, LF checks the status of the flag bit and performs corresponding update operations, but does not store the item in the bucket. The key field in the bucket is used to store the identifier of the persistent item. However, in the insertion operation, if the item does not exist, LF mainly focuses on the update of the counter and the toggling of the flag bit.

[0033] Query operation: When querying the persistence of an item, d hash functions of the item are calculated, the corresponding d counters are retrieved, and the minimum count value min among the d counters is taken as the estimated persistence value of the item. This approach reduces the impact of hash collisions by taking the minimum value, thereby providing a more accurate persistence estimate.

[0034] Embodiment 2:

[0035] A system for efficiently detecting persistent items in a small memory environment according to the present invention includes: Two-dimensional array data structure module: A two-dimensional array is used as the data structure, which contains d rows and l columns. Each row corresponds to a hash function that maps an item to a specific counter column; each bucket corresponds to a counter, and each bucket contains two fields. One field is used to store the key of the persistent item, and the other field is used as a flag bit to indicate whether the item has been accessed within the current time window; the flag bit is initially set to "On", and once the item is accessed, the flag bit is toggled to "Off".

[0036] Conservative update strategy execution module: When inserting an item, d hash functions are calculated to map the item to the positions of d different counters, the current values of the different counters are checked, the minimum count value min is selected, and the row containing the minimum count value min is marked as the target row; when updating the counter, only the smallest active counter in the current target row is updated.

[0037] The conservative update strategy execution module includes: Update operation execution module: When updating the counter, the status of the flag bit is checked: If the flag bit in the bucket corresponding to the item is "On", it means the item appears for the first time, the counter value of the target row is incremented, and its flag bit is toggled to "Off"; if the flag bit is "Off", it means the item has appeared before, and the counter value remains unchanged.

[0038] Scan update execution module: After the update operation is completed, all rows are scanned to check if there are other rows whose counter values match the minimum count value min and the flag bit is "On". If so, the flag bit of that row is toggled to "Off", and the counter value of that row is incremented.

[0039] Insertion operation execution module: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions for the item, with each sub-item corresponding to a hash value. Map the d sub-items to d rows of a two-dimensional array respectively, where each row corresponds to a hash function, and each hash function maps to the position of a counter in a corresponding column of the row. The unique ID of the item is used to identify the item itself. In the insertion operation, LF maps the item ID to the counter position through the hash function; in the query operation, it also locates the counter through the hash function to retrieve the persistence information of the item.

[0040] Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket, incrementing the count value; if the item does not exist, check the status of the flag bit in the current time window of the specified bucket. If the flag bit is "On", record the minimum count value min in the current row of the specified bucket and mark this row as the target row; if the flag bit is "Off", do nothing.

[0041] Scan all rows. If the counter value of a certain row matches the minimum count value min and the flag bit is "On", switch the flag bit to "Off" and increment the counter value.

[0042] Query operation execution module: When querying the persistence of an item, calculate d hash functions for the item, retrieve the corresponding d counters, and take the minimum count value min among the d counters as the persistence estimate value of the item.

[0043] Example 3, operation example.

[0044] Example 1: Insertion operation.

[0045] Suppose there is a data stream containing multiple items, and each item has a unique ID. Use LF for the insertion operation, and the specific steps are as follows: Initialize the data structure of LF, set d = 3 and l = 1000, that is, a two-dimensional array with 3 rows and 1000 columns.

[0046] When item e1 arrives, split item e1 into 3 sub-items, calculate 3 hash functions h1(e1), h2(e1), h3(e1), with each sub-item corresponding to a hash value. Map the 3 sub-items of item e1 to 3 different counter positions C1[h1(e1)], C2[h2(e1)], C3[h3(e1)] respectively.

[0047] Check whether e1 already exists in the specified bucket. If it does not exist, check the status of the flag bit in the current time window. Suppose the flag bit is "On", record the minimum count value min in the current row and mark this row as the target row.

[0048] Scan all lines. If the counter value of a certain line matches min and the flag is "On", then switch the flag to "Off" and increment the counter.

[0049] Example 2: Query operation.

[0050] Suppose we want to query the persistence of item e2. The specific steps are as follows: Calculate three hash functions h1(e2), h2(e2), h3(e2), retrieve the corresponding three counters C1[h1(e2)], C2[h2(e2)], C3[h3(e2)], and take the minimum value of these three counters as the estimated value of the persistence of e2.

[0051] The method and system for efficiently detecting persistent items in a small memory environment of the present invention have the following advantages: (1) Improve space efficiency and be able to accurately detect persistent items with limited memory resources. LF performs well in a small memory environment. For example, when the memory is only 10% of the memory required by On-Off Sketch (OO), the average absolute error (AAE) of LF is almost halved, and the average relative error (ARE) is reduced by 10 times. This shows that LF is significantly superior to existing methods in terms of space efficiency.

[0052] (2) Improve detection accuracy and reduce the impact of hash collisions on the detection results. LF effectively reduces the impact of hash collisions on the detection results through the CU strategy, thereby improving the detection accuracy. Under different time window sizes and memory configurations, the detection accuracy of LF is better than that of existing methods.

[0053] (3) Improve flexibility and be able to adapt to data streams with different time window sizes. LF supports variable time windows and can adapt to the characteristic changes of different data streams, which makes it have higher flexibility and adaptability in practical applications.

[0054] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for efficiently detecting persistent items in a small memory environment, characterized in that: The following steps are involved: Construct a two-dimensional array data structure: Use a two-dimensional array as the data structure, which contains d rows and l columns. Each row corresponds to a hash function that maps the item to a specific counter column. Each bucket corresponds to a counter. Each bucket contains two fields, one of which is used to store the key of the persistent item, and the other is a flag that indicates whether the item has been accessed within the current time window. The flag is initially set to "On". Once the item is accessed, the flag is switched to "Off". Execute a conservative update strategy: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current count values ​​of different counters, select the smallest count value min, and mark the row containing the smallest count value min as the target row; when updating the counter, only update the smallest active counter in the current target row.

2. The method for efficiently detecting persistent items in a small memory environment according to claim 1, characterized in that: Executing a conservative update strategy involves: Update operation: When updating the counter, check the status of the flag: if the flag in the bucket corresponding to the item is "On", it means that the item appears for the first time, increment the counter value of the target row, and switch its flag to "Off"; if the flag is "Off", it means that the item has appeared before, and the counter value remains unchanged.

3. The method for efficiently detecting persistent items in a small memory environment according to claim 2, characterized in that: Executing a conservative update strategy involves: Scan update: After the update operation is completed, scan all rows to check whether there are other rows whose counter values ​​match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

4. The method for efficiently detecting persistent items in a small memory environment according to claim 1, characterized in that: It also includes performing insert operations: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions of the item, each sub-item corresponds to a hash value, map the d sub-items to d rows of the two-dimensional array, each row corresponds to a hash function, and each hash function is mapped to the counter position of one of the columns of the corresponding row; Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket and increment the count value. If the item does not exist, check the flag status of the current time window of the specified bucket. If the flag is "On", record the minimum count value min in the current row of the specified bucket and mark the row as the target row. If the flag is "Off", do nothing. Scan all rows, and if the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value.

5. The method for efficiently detecting persistent items in a small memory environment according to claim 1, characterized in that: It also includes query operations: When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the smallest count value min among the d counters as the estimated persistence value of the item.

6. A system for efficiently detecting persistent items in a small memory environment, characterized in that: include Two-dimensional array data structure module: uses a two-dimensional array as the data structure, which contains d rows and l columns. Each row corresponds to a hash function that maps the item to a specific counter column. Each bucket corresponds to a counter. Each bucket contains two fields, one of which is used to store the key of the persistent item, and the other is used as a flag to indicate whether the item has been accessed within the current time window. The flag is initially set to "On". Once the item is accessed, the flag is switched to "Off". Conservative update strategy execution module: When inserting an item, calculate d hash functions, map the item to the position of different counters in d rows, check the current count values ​​of different counters, select the smallest count value min, and mark the row containing the smallest count value min as the target row; when updating the counter, only update the smallest active counter in the current target row.

7. The system for efficiently detecting persistent items in a small memory environment according to claim 6, characterized in that: The conservative update strategy execution module includes: Update operation execution module: When updating the counter, check the status of the flag: if the flag in the bucket corresponding to the item is "On", it means that the item appears for the first time, increment the counter value of the target row, and switch its flag to "Off"; if the flag is "Off", it means that the item has appeared before, and the counter value remains unchanged.

8. The system for efficiently detecting persistent items in a small memory environment according to claim 7, characterized in that: The conservative update strategy execution module includes: Scan update execution module: After the update operation is completed, scan all rows to check whether there are other rows whose counter values ​​match the minimum count value min and whose flag is "On". If so, switch the flag of the row to "Off" and increment the counter value of the row.

9. The system for efficiently detecting persistent items in a small memory environment according to claim 6, characterized in that: It also includes the insert operation execution module: When an item with a unique ID arrives, split the item into d sub-items, calculate d hash functions of the item, each sub-item corresponds to a hash value, map the d sub-items to d rows of the two-dimensional array, each row corresponds to a hash function, and each hash function is mapped to the counter position of one of the columns of the corresponding row; Check whether the item already exists in the specified bucket. If the item exists, update the counter corresponding to the bucket and increment the count value. If the item does not exist, check the flag status of the current time window of the specified bucket. If the flag is "On", record the minimum count value min in the current row of the specified bucket and mark the row as the target row. If the flag is "Off", do nothing. Scan all rows, and if the counter value of a row matches the minimum count value min and the flag is "On", switch the flag to "Off" and increment the counter value.

10. The system for efficiently detecting persistent items in a small memory environment according to claim 6, characterized in that: It also includes query operation execution modules: When querying the persistence of an item, calculate d hash functions of the item, retrieve the corresponding d counters, and take the smallest count value min among the d counters as the estimated persistence value of the item.