A method and apparatus for querying a persistent item in a distributed stream

Abstract

The present disclosure relates to a method and device for querying a persistent item in a distributed stream. The method comprises: calculating d hash functions according to the ID of a target item to obtain d counters corresponding to the mapping, the target item being an item for which a persistence estimate is queried this time, the persistence estimate being used to represent the persistence of the target item, the persistence representing the number of time windows in which the target item has appeared in the distributed stream; calculating the cardinality estimate of an HLL counter in the d counters respectively to obtain the count value of each counter; and taking the counter with the minimum value among the d counters as the result of the persistence estimate of the target item queried this time. The present disclosure can provide accurate cross-site persistence estimation and persistent item query, and realize an anomaly detection task under privacy protection conditions.

Description

Technical Field

[0001] This disclosure relates to streaming data processing technology, and more particularly to a method and apparatus for querying persistent items in a distributed stream. Background Technology

[0002] With the rapid development of the internet, the generation of massive amounts of real-time data has led to increasingly faster streaming data speeds. How to store and process this massive amount of data within limited space has become a major challenge. To address the problem of information overload, approximate processing techniques for streaming data are finding increasingly widespread applications, with sketch models receiving particular attention. Under limited storage space, instead of storing and indexing every single event, we process them quickly and create a summary.

[0003] In existing technologies, streaming data approximation processing techniques primarily focus on frequency, cardinality, and quantiles. However, in certain scenarios, such as telecommunications companies wanting to track activity on their networks to identify overall network performance and detect anomalous behavior, a different approach is needed.

[0004] Some malicious actors attempt to evade frequency detection by distributing anomalous attacks across numerous different time periods, rendering traditional frequency detection methods ineffective in accurately capturing such anomalies. In many cases, because the input data stream may contain sensitive information, such as descriptions of personal behavior, preferences, or characteristics, users may be unwilling to disclose their true information, thus posing numerous security risks during data collection, storage, and use. How to collaborate across multiple sites to achieve anomaly detection under privacy-preserving conditions has become a pressing issue. Summary of the Invention

[0005] This disclosure provides a method and apparatus for querying persistent items in a distributed stream, to at least solve the problem in the prior art that it is impossible to accurately detect the scattered behavior of items across multiple windows of a data stream. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a method for querying persistent items in a distributed stream is provided, comprising:

[0007] A data structure for persistence estimation is provided, wherein the data structure for persistence estimation consists of d counter arrays, wherein each counter array consists of l counters, and for each of the l counters, each counter holds an HLL counter, wherein the HLL counter is an array consisting of multiple data units, wherein the value in the data unit is 1 or 0, and d and l are both positive integers;

[0008] The query method includes:

[0009] Calculate d hash functions h1, h2...h based on the target project's ID. d d counters corresponding to the mapping are obtained. The target item is the item whose persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target item. The persistence represents the number of time windows in which the target item appears in the distributed stream.

[0010] Calculate the base estimate of the HLL counter in each of the d counters to obtain the count value of each counter;

[0011] The counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query.

[0012] Optionally, the data structure for the persistence estimate may also include an item insertion operation performed according to the following steps:

[0013] Calculate d hash functions h1, h2...h based on the project ID. d ;

[0014] According to the d hash functions h1, h2...h d The mapping relationship is used to map each item to a corresponding d counter;

[0015] Calculate the new hash function h′ based on the current time window t;

[0016] According to the new hash function h′, the items are mapped to the data units of the HLL counter.

[0017] Optionally, mapping the items to the data units of the HLL counter according to the new hash function h′ includes:

[0018] Initialize all data cells to 0;

[0019] In the d counters corresponding to the project, according to the new hash function h′, the project is mapped to the data unit of the HLL counter held by each of the d counters, and the value of the data unit to which the project is mapped is updated;

[0020] If the value of this data cell is 0, it means that the target item has not yet accessed the counter in the current time window. When mapping the item, the value in the data cell will be set to 1.

[0021] If the value of this data cell is 1, it means that the target item in the current time window has already accessed the counter, and the value in the data cell will not be changed when mapping the item.

[0022] Optionally, this also includes merging data from different sites by following these steps:

[0023] The data in each data unit of the HLL counter between different site counters are combined to obtain the merged HLL counter;

[0024] Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0025] According to a second aspect of the present disclosure, a method for querying persistent items in a distributed stream is provided, comprising: a data structure applied to query persistent items, the data structure for querying persistent items including a counter structure and a bucket structure, the counter structure being a counter array consisting of l counters, each counter holding an HLL counter, the HLL counter being an array consisting of multiple data units, the value in the data unit being 1 or 0, and l being a positive integer;

[0026] The bucket structure consists of l buckets, which correspond to l counters in the counter structure. Each bucket contains multiple key-value pairs, where the key is the ID of the item and the value is the HLL counter corresponding to the item.

[0027] The query method includes:

[0028] Determine the given threshold carried in the query request initiated by the user terminal;

[0029] Iterate through all the buckets in the bucket structure;

[0030] The IDs of items whose corresponding counter values ​​are greater than the given threshold are reported as persistent items for this query.

[0031] Optionally, the process may also include inserting items into the data structure of the query persistence item according to the following steps:

[0032] When performing the first insertion operation on each item, insert that item into the counter structure;

[0033] When performing subsequent insert operations on each item, check if the item has already been recorded in the bucket structure:

[0034] If the item is already recorded in the bucket structure, update the HLL counter corresponding to the item in the bucket structure; if the item is not recorded in the bucket structure, update the counter C[i] corresponding to the item in the counter structure.

[0035] Compare the counter C[i] corresponding to each item in the counter structure with the smallest counter B[i] in the bucket structure. min Comparison:

[0036] If C[i] > B[i] min Then B[i] will be...min Set the key to the project's ID and swap C[i] and B[i]. min If C[i] ≤ B[i] min If not, no further operations will be performed.

[0037] Optionally, the counter C[i] corresponding to the item in the update counter structure includes:

[0038] Calculate the hash function h1 based on the project ID;

[0039] According to the mapping relationship of the hash function h1, the items are mapped to the counters in the counter structure;

[0040] Based on the current project's time window t, the project is mapped to the corresponding HLL counter's data unit according to the new hash function h′.

[0041] Optionally, this also includes merging data from different sites by following these steps:

[0042] For merging counters in a counter structure:

[0043] The data in each data unit of the HLL counter between different site counters are combined to obtain the merged HLL counter;

[0044] Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter;

[0045] For merging counters in a bucket structure:

[0046] If all items appear in the bucket structure of each site participating in the merger, then the data in each data unit of the HLL counter corresponding to the item is combined to obtain the merged HLL counter;

[0047] If an item only appears in the bucket structure of certain sites, and no other counters for that item appear in the bucket structure, then the HLL counter held by the counter in the counter structure is used to participate in the merging operation. After merging, the item with the higher persistence estimate is taken as the merged HLL counter.

[0048] Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0049] According to a third aspect of the present disclosure, a query apparatus for persistent items in a distributed stream is provided, applied to a data structure for persistence estimation. The data structure for persistence estimation consists of d counter arrays, each counter array consisting of l counters. For each of the l counters, each counter holds an HLL counter, where the HLL counter is an array of multiple data units, the values ​​of which are 1 or 0. d and l are both positive integers, including:

[0050] The mapping module is configured to perform the calculation of d hash functions h1, h2...h based on the ID of the target item. d d counters corresponding to the mapping are obtained. The target item is the item whose persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target item. The persistence represents the number of time windows in which the target item appears in the distributed stream.

[0051] The cardinality estimation module is configured to perform cardinality estimation for the HLL counters in the d counters respectively, and obtain the count value of each counter;

[0052] The counter comparison module is configured to take the counter with the smallest value among the d counters as the result of the persistence estimate of the target item in this query.

[0053] According to a fourth aspect of the present disclosure, a query apparatus for persistent items in a distributed stream is provided, which is applied to querying a data structure of persistent items. The data structure for querying persistent items includes a counter structure and a bucket structure. The counter structure is a counter array, which consists of l counters. For the l counters, each counter holds an HLL counter. The HLL counter is an array consisting of multiple data units, where the value in the data unit is 1 or 0, and l is a positive integer.

[0054] The bucket structure consists of l buckets, each corresponding to a counter in the counter structure. Each bucket contains multiple key-value pairs, where the key is the item's ID and the value is the HLL counter corresponding to that item, including:

[0055] The threshold determination module is configured to determine the given threshold carried in the query request initiated by the user terminal.

[0056] The traversal module is configured to traverse all buckets in the bucket structure.

[0057] The query module is configured to use the IDs of items whose corresponding counter values ​​in the execution report are greater than the given threshold as persistent items for this query.

[0058] In this embodiment of the invention, a persistence estimation data structure (Persistence Sketch) is proposed. This data structure consists of d counter arrays, each containing l counters. Each of the l counters holds an HLL counter, which is an array of multiple data units, where the value in each data unit is 1 or 0. Both d and l are positive integers. The proposed Persistence Sketch is an accurate sketch model for persistence research. Compared to traditional sampling algorithms, Persistence Sketch significantly improves the accuracy of persistence estimation, substantially reduces estimation errors, and better performs tasks such as network monitoring and anomaly detection. Furthermore, Persistence Sketch is a sketch model suitable for distributed environments, solving the problem of existing sketch models being unmergeable. It can losslessly merge sketch models generated by different sites, providing a high-accuracy sketch model in distributed environments. Regarding the HLL counter structure held by the counters in this structure, as a cardinality counting algorithm, the HLL counter can perform element deduplication and counting in a very small space, ensuring low memory consumption and high statistical accuracy even with large data volumes. Based on this data structure, this invention proposes a persistence-oriented query method. First, it calculates d hash functions h1, h2...h based on the ID of the target item. d The algorithm obtains d counters corresponding to the mapping, where the target item is the item for which persistence is estimated in this query. The persistence estimate represents the persistence of the target item, which is the number of time windows in which the target item appears in the distributed stream. Then, the cardinality estimate of the HLL counters in the d counters is calculated to obtain the count value of each counter. Finally, the counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query. This invention can accurately detect the scattered behavior of items in multiple windows of the data stream without significantly improving the algorithm's time complexity. It also solves the defect of existing sketch models that do not support persistence deduplication, and the query method can support instantaneous queries.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0061] Figure 1This is a schematic diagram of a data structure for persistence estimation provided in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the data structure of the HLL counter provided in an embodiment of the present invention.

[0063] Figure 3 This is a flowchart of a method for querying persistent items in a distributed stream, provided by an embodiment of the present invention.

[0064] Figure 4 This is a schematic diagram of a data structure for querying persistent items provided in an embodiment of the present invention.

[0065] Figure 5 This is a flowchart of a method for querying persistent items in a distributed stream, provided by an embodiment of the present invention.

[0066] Figure 6 This is a schematic diagram of multi-site data merging provided by an embodiment of the present invention.

[0067] Figure 7 This is a schematic diagram of a device for querying persistent items in a distributed stream, provided in an embodiment of the present invention.

[0068] Figure 8 This is a schematic diagram of a device for querying persistent items in a distributed stream, provided in an embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0070] This invention provides a method and apparatus for querying persistent items in a distributed stream, and applies a data structure for persistence estimation. In order to describe the data structure for persistence estimation proposed in this invention, the prior art involved in the embodiments of this invention will be briefly introduced first.

[0071] On-Off Sketch is a sketch model for handling data flow. It leverages the periodic incrementing persistence of an element, compressing the increment when multiple elements are mapped to the same counter. By adding a status field (On or Off) to each counter, when an element arrives, the status field is checked first. If it's Off, the counter doesn't count; it only counts when the status field is On, incrementing the counter by 1. However, when mapping and inserting items, we found a problem with merging multiple On-Off Sketch models. When an item is accessed on multiple sites within a certain time window, the counters on different sites will all count within that time window, resulting in an accumulation. For example, in 20 sites, due to item access within a time window, the counters for the item on all 20 sites will increment by 1. When merging multiple models, this results in an increment of 20, leading to a higher actual count than expected, and a deviation in the item persistence record. On-Off Sketch cannot guarantee that items are counted only once within the same time window when merging multiple sketch models, thus causing count discrepancies. Based on this, this application proposes an improved data structure for persistence estimation. We discard the original status field in On-Off Sketch, avoid multiple accumulations of item accesses, and set an HLL counter in each counter. Figure 2 This is a schematic diagram of the data structure of the HLL counter provided in an embodiment of the present invention. Figure 2 As shown, the HLL counter is an array structure with multiple data units, each storing either a binary 0 or a 1, thus it can be viewed as a fixed-length binary string. Modifications to the values ​​of these data units reflect whether an item has been accessed. If an item is accessed multiple times within the same time window, the value in each HLL counter unit is changed at most once (e.g., the first access to an item within the same time window sets 0 to 1, while subsequent accesses keep the corresponding time window's value at 1). In this way, the HLL counter effectively records only one access operation for the item within that time window, unlike the counter in On-OffSketch which continuously increments. Furthermore, this significantly reduces model error during subsequent multi-site merging.

[0072] Figure 1 This is a schematic diagram of a data structure for persistence estimation provided in an embodiment of the present invention. For example... Figure 1 As shown:

[0073] The persistence estimation data structure (Persistence Sketch) consists of d counter arrays, each of which consists of l counters. For each of the l counters, there is an HLL counter, which is an array of multiple data units, where the value of each data unit is 1 or 0. Both d and l are positive integers.

[0074] In Persistence Sketch, the value of d is calculated using ln(1 / δ), typically between 2 and 5; the value of l is calculated using e / ε. Here, δ and ε are error parameters of the Sketch model. Compared to the On-Off Sketch model, Persistence Sketch omits the ON and OFF state fields and sets an HLL counter in each counter. Figure 2 This is a schematic diagram of the data structure of the HLL counter provided in an embodiment of the present invention. Figure 2 As shown, the HLL counter is an array structure with multiple data units. Each data unit stores either binary 0 or 1, and can therefore be viewed as a binary string of fixed length. Modifications to the values ​​of these data units reflect whether an item has been accessed. If an item is accessed multiple times within the same time window, the value of each unit in the HLL counter is changed at most once (e.g., the first access to an item within the same time window sets 0 to 1, while subsequent accesses keep the corresponding time window's value at 1). In this way, the HLL counter effectively records only one access operation for an item within that time window, unlike the counter in On-Off Sketch which continuously increments. Furthermore, this significantly reduces model error during subsequent multi-site merging.

[0075] The primary application scenarios for Persistence Sketch are distributed data stream processing, particularly those requiring persistent analysis and querying. Here are a few practical application scenarios to illustrate this. For telecommunications operators, tracking network activity is crucial for monitoring overall network performance and identifying anomalies. Persistence Sketch helps operators identify these anomalies and provides accurate persistent statistical results. For example, a telecommunications operator can use Persistence Sketch to monitor data traffic on its network. Through persistent estimation and querying, it can accurately detect dispersed behaviors across multiple time windows, such as malicious attacks or abnormal traffic. This allows the operator to take timely measures to protect network security and stability. In another scenario, a large e-commerce platform can use Persistence Sketch to analyze user purchasing behavior. Through persistent estimation and querying, it can understand user purchasing habits and preferences across different time windows, providing valuable information for personalized recommendations and marketing strategies.

[0076] Figure 3 This is a flowchart of a method for querying persistent items in a distributed stream, as provided in an embodiment of the present invention. Figure 3 As shown:

[0077] Step S110: Calculate d hash functions based on the ID of the target project to obtain d corresponding counters. The target project is the project whose durability is estimated in this query. The durability estimate is used to represent the durability of the target project. The durability represents the number of time windows in which the target project appears in the distributed stream.

[0078] Step S120: Calculate the base estimate of the HLL counter in each of the d counters to obtain the count value of each counter.

[0079] Step S130: The counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query.

[0080] In this embodiment of the invention, a method for querying persistent items in a distributed stream is proposed based on a data structure for persistence estimation. First, d hash functions h1, h2...h are calculated based on the ID of the target item. dThe algorithm obtains d counters corresponding to the mapping, where the target item is the item for which the persistence is estimated in this query. The persistence estimate represents the persistence of the target item, which is the number of time windows in which the target item appears in the distributed stream. Then, the cardinality estimate of the HLL counters in the d counters is calculated to obtain the count value of each counter. Finally, the counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query. This invention, without significantly increasing the time complexity of the algorithm, can not only accurately detect the scattered behavior of items in multiple windows of the data stream, but also solves the defect of existing sketch models that do not support model merging in a distributed environment. Furthermore, this query method can support instantaneous queries.

[0081] The following section introduces the data structure based on persistence estimation and the specific implementation of the persistence estimation query method.

[0082] Step S110 is involved.

[0083] Calculate d hash functions based on the ID of the target project to obtain d corresponding counters. The target project is the project for which the persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target project. The persistence represents the number of time windows in which the target project appears in the distributed stream.

[0084] As a feature of streaming data processing, persistence differs from frequency. It highlights the recurring behavior of an item across different time periods in the data stream, accurately detecting scattered behavior across multiple windows. Persistence can be used to detect potential cybersecurity malicious behavior or click fraud. For example, given an item *e* and a data stream *S* with *T* non-overlapping consecutive time windows, the persistence of *e* is defined as the number of time windows in which *e* appears. For instance, if a person visits a website *f* times in a year, and visits the website for *p* days, *f* is the frequency of the website, and *p* is the persistence of the website, indicating the person's preference for that website. If the user's focus in streaming data is frequency, anomalous attacks will distribute their behavior across different time windows to circumvent this feature. In this case, it becomes difficult to detect such distributed attacks using the frequency of the streaming data. This application addresses the persistence feature in distributed data streams by using a counter to record the number of time windows in which anomalous attack behavior occurs. By continuously recording the attack, such scattered anomalous behavior can be easily captured.

[0085] When project e i When performing an insert operation, Persistence Sketch will adjust the settings based on the project name. iGiven the ID, calculate d hash functions equal to the number of counter array elements. A hash function is a function that maps an input of arbitrary size to a hash value of fixed size, such that each element (e.g., the item e to be inserted in this invention) is hashed. i Each element corresponds to a unique storage location within the structure, enabling element-to-element mapping. Following the mapping relationship of d hash functions, each item can be mapped to one of d counters, with each of these d counters belonging to a different counter array.

[0086] Step S120 is involved.

[0087] Calculate the base estimate of the HLL counter in each of the d counters to obtain the count value of each counter.

[0088] Cardinality estimation is a type of probabilistic algorithm that can estimate the cardinality with significantly less time and space consumption than precise calculation, even with a certain degree of error. The structure of an HLL counter has been briefly described earlier; it can be viewed as a binary string of fixed length. The basic idea of ​​the cardinality estimation algorithm is to decompose the binary string into different bits, each bit representing a specific weight. These weights are then added together to obtain a decimal value, which is displayed as the final count value of the counter.

[0089] This involves step S130.

[0090] The counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query.

[0091] The hash function corresponding to the minimum counter may produce an estimated frequency that is less than or equal to the actual frequency. When multiple hash functions map to different counters, choosing the minimum counter can reduce accumulated error. Because each counter independently records the frequency of elements, choosing the minimum value minimizes accumulated error. Choosing the minimum counter as the final estimate is a conservative strategy that minimizes estimation error and ensures that the frequency of elements is not overestimated. This is very useful in various applications, especially in scenarios where the frequency of elements in a data stream needs to be estimated.

[0092] Based on the persistence estimation data structure proposed in this invention, the method further includes inserting items into the persistence estimation data structure according to the following steps:

[0093] Step S210: Calculate d hash functions h1, h2...h based on the project ID. d .

[0094] Step S220, according to the d hash functions h1, h2...h d The mapping relationship is used to map each item to a corresponding d counter.

[0095] Step S230: Calculate the new hash function h′ based on the current time window t.

[0096] Step S240: Map the items to the data units of the HLL counter according to the new hash function h′.

[0097] In the data structure for persistence estimation, the order of item query and insertion operations is generally as follows: data is inserted first, Persistence Sketch performs the corresponding operation, and then the user queries based on the processing results. Query operations are also allowed during the insertion process. For example, if the current data stream size is already 2GB, a user querying at this point will retrieve the statistical results of the current 2GB of data. When an item is mapped to its corresponding counter using a hash function, a new hash function h′ needs to be calculated based on the time window t in which the item data is located. Then, based on the mapping relationship of h′, the item is mapped to the data unit of the HLL counter. In this way, by modifying the value of the counter unit in the HLL counter, regardless of how many times the item mapped to a certain time window appears, we can ensure that each counter increases at most once in each time window. Persistence Sketch eliminates the error caused by the repeated occurrence of the same item within the same time window, reducing the overestimation of persistence values. Furthermore, the setting of HLL counters also has certain advantages in terms of space efficiency. The main advantage of the HLL counter data structure is its high space efficiency; regardless of the size of the dataset, the space used by the HLL data structure is constant. Traditionally, to accurately calculate cardinality, it's necessary to store all observed unique elements, which is impractical in big data environments. However, the HLL data structure only needs to store a fixed-size binary array, thus saving significant space. Furthermore, merging HLL counters is simple: two or more HLL counters can be easily merged, making them well-suited for distributed system workloads.

[0098] The following describes the data structure based on persistence estimation, the HLL counter merging scheme, and the merging of data from different sites according to the following steps:

[0099] Step S310: Perform a union operation on the data in each data unit of the HLL counter between different station counters to obtain the merged HLL counter.

[0100] Step S320: Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0101] Step S310 is involved.

[0102] The data in each data unit of the HLL counter between different site counters are combined to obtain the merged HLL counter.

[0103] The union operation mentioned here is equivalent to an OR logical operation: 1 and 0 combined equals 1, 0 and 0 combined equals 0, and 1 and 1 combined equals 1. To ensure that the merged HLL data structure can still accurately estimate the cardinality, a union operation needs to be performed to preserve the maximum leading zero information. In this way, the merged data structure will try to maintain the maximum number of leading zeros in each unit, thus maintaining accurate cardinality estimation.

[0104] This involves step S320.

[0105] Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0106] The merged HLL data structure typically shares the same configuration parameters (e.g., the number of hash functions and the size of the bit array) to ensure consistent performance and accuracy. However, the contents of the bit array are appropriately "unioned" to maintain the accuracy of the estimate, meaning the merged HLL structure may differ slightly from any of the original structures. Merging multiple HLL data structures results in a new HLL data structure designed to more accurately estimate the overall cardinality of all data before merging, rather than retaining multiple original structures. This helps maintain accuracy and saves memory. The method for obtaining the counter value through cardinality estimation has already been explained, and will not be repeated here. In a telecommunications operator's application scenario, suppose a telecommunications operator has multiple geographically distributed data centers, each responsible for monitoring and recording traffic data on its local network. To gain a comprehensive understanding of the persistence of the entire network, the telecommunications operator wants to merge the Persistence Sketches from each data center to obtain overall persistence statistics. First, each data center uses the Persistence Sketch to perform persistence estimation and querying of local traffic. Each data center's Persistence Sketch records the persistence information of local traffic and provides a query interface for retrieving traffic items with high persistence. Then, the telecom operator merges the persistence sketches from each data center. This merged sketch allows the operator to conduct persistence studies and queries across the entire network. For example, they can query high-persistence traffic items to understand key traffic characteristics across the network; or they can perform persistence estimates to assess anomalous behavior or potential risks. By merging persistence sketches, telecom operators can integrate data from various data centers, enabling persistence studies and queries across the entire network. This provides a more comprehensive understanding of network operations, allows for timely detection of anomalous behavior, and enables appropriate measures to protect network security and stability.

[0107] Figure 4 This is a schematic diagram of a data structure for querying persistent items provided in an embodiment of the present invention.

[0108] like Figure 4 The data structure for querying persistent items includes a counter structure and a bucket structure. The counter structure is a counter array consisting of l counters. Each of the l counters holds an HLL counter, which is an array consisting of multiple data units. The value in each data unit is 1 or 0, and l is a positive integer.

[0109] The bucket structure consists of l buckets, which correspond to l counters in the counter structure. Each bucket contains multiple key-value pairs, where the key is the ID of the item and the value is the HLL counter corresponding to the item.

[0110] In the persistence estimation problem, the item IDs are not stored, indicating that the persistence estimation data structure cannot be directly used to query persistent items. Therefore, this invention proposes an improvement on the persistence estimation data structure to facilitate the search for persistent items. The core of the data structure for querying persistent items is to distinguish whether an item is persistent through persistence estimation, and to discard the IDs of non-persistent items, storing only the IDs of persistent items to reduce memory consumption.

[0111] The data structure used for querying persistent items is an improved array of counters, mapped using a hash function by default. Compared to the data structure used for persistence estimation, the data structure for finding persistent items uses only one array of counters. This array has the same structure as the counter array in the persistence estimation data structure, except that the number of counters is reduced from `d` to one. The counter structure in this array has been described in detail earlier. In this array, an additional bucket is used for each counter. By reducing the number of arrays, we can improve throughput. By using an additional bucket for each counter, we can separate persistent and non-persistent items and only record the ID of the persistent item. If an item's persistence is high enough, its counter will not be the smallest in the bucket and therefore will not be replaced, thus avoiding hash collisions and reducing errors.

[0112] Based on the data structure of persistent items, query operations can be performed on persistent items. Figure 5 This is a flowchart of a method for querying persistent items in a distributed stream, provided by an embodiment of the present invention.

[0113] like Figure 5 As shown, the query method, following steps S410-S430, is applied to the data structure for querying persistent items:

[0114] Step S410: Determine the given threshold carried in the query request initiated by the user terminal.

[0115] The given threshold is defined by the user terminal. The defined persistence threshold is generally within the persistence range that corresponds to the bucket structure, and is used to filter out the user-defined "persistent items".

[0116] Step S420: Traverse all buckets in the bucket structure.

[0117] Step S430: Report the IDs of items whose corresponding counter values ​​are greater than the given threshold, as persistent items for this query.

[0118] The value of the counter corresponds to the persistence of the stored item. The larger the counter value, the greater the persistence of the stored item, and vice versa.

[0119] Similar to the data structure used for persistence estimation, the data structure for querying persistent items performs item insertion operations according to the following steps:

[0120] Step S510: When performing the first insertion operation on each item, insert the item into the counter structure.

[0121] Step S520: When performing subsequent insertion operations on each item, check whether the item has already been recorded in the bucket structure:

[0122] If the item is already recorded in the bucket structure, update the HLL counter corresponding to the item in the bucket structure; if the item is not recorded in the bucket structure, update the counter C[i] corresponding to the item in the counter structure.

[0123] Step S530: Match the counter C[i] corresponding to the item in the counter structure with the smallest counter B[i] in the bucket structure. min Comparison:

[0124] If C[i] > B[i] min Then B[i] will be... min Set the key to the project's ID and swap C[i] and B[i]. min If C[i] ≤ B[i] min If not, no further operations will be performed.

[0125] C[i] and B[i] min By comparing their sizes, persistent and non-persistent items can be separated, and only the IDs of persistent items are recorded. If C[i] > B[i] min This indicates that the item's persistence is large enough to be inserted into the bucket structure, therefore the item needs to be moved from the counter structure to the bucket structure; if C[i] ≤ B[i] min This indicates that the item's persistence is insufficient, and it remains stored in the counter structure used to record non-persistent items, without any operation. In other words, if an item's persistence is high enough, its counter will not be the smallest one in the bucket, and therefore will not be replaced, thus avoiding the impact of hash collisions and reducing errors. This can improve the precision and accuracy of querying persistent items.

[0126] In step S530, updating the counter C[i] corresponding to the item in the counter structure is equivalent to inserting the corresponding mapping of the item in the counter structure. Therefore, it is the same as the scheme of inserting the item in the persistence estimation data structure, and is implemented according to steps S610-S630:

[0127] Step S610: Calculate the hash function h1 based on the project ID.

[0128] Step S620: According to the mapping relationship of the hash function, the item is mapped to the counter in the counter structure.

[0129] Step S630: Based on the current project's time window t, map the project to the corresponding HLL counter's data unit according to the new hash function h′.

[0130] Figure 6 This is a schematic diagram of multi-site data merging provided by an embodiment of the present invention, as shown below. Figure 6 As shown: Based on the data structure of persistent queries, the HLL counter merging scheme follows the steps.

[0131] For merging counters in a counter structure:

[0132] Step S710: Perform a union operation on the data in each data unit of the HLL counter between different station counters to obtain the merged HLL counter.

[0133] Step S720: Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0134] For merging counters in a bucket structure:

[0135] Step S730: If all items appear in the bucket structure of each site participating in the merging, then perform a union operation on the data in each data unit of the HLL counter corresponding to the item to obtain the merged HLL counter.

[0136] In a distributed environment with data from multiple sites, due to the varying persistence of different projects, not every project is recorded in the bucket structure of each site participating in the merging. When a project appears in the bucket structure of each site participating in the merging, a AND operation is performed on the data in each data unit of the HLL counter corresponding to the project. This AND operation on the data in each data unit of the HLL counter is consistent with the operation on merging the HLL counter in the persistence estimation data structure mentioned earlier, which is to perform an OR operation on the binary value in each data unit.

[0137] Step S740: If an item only appears in the bucket structure of some sites, and no other counters for the item appear in the bucket structure, then the HLL counter held by the counter in the counter structure is used to participate in the merging operation. After merging, the item with the higher persistence estimate is taken as the merged HLL counter.

[0138] If an item only appears in the bucket structure of the persistent data structure of some of the sites participating in the merge, and the counter for that item does not appear in the bucket structures of other sites, then the HLL counter in the counter structure is used to participate in the merge operation. After the merge, the items with higher persistence estimates are selected as the merged HLL counters. The number of these top items is the same as the number of key-value pairs in the bucket structure. The understanding of the top items can be considered from two aspects:

[0139] From the perspective of array indexing: In the merged counter array, positions with smaller indices are considered to be earlier. Items with higher persistence estimates are placed at smaller indices, making them easier to access.

[0140] From the perspective of persistence estimates: Items with higher persistence estimates indicate that they appear more frequently in different time windows of the data stream. Placing items with higher persistence estimates earlier in the list allows for faster identification of these frequently occurring items, leading to more accurate persistence estimation and persistent item lookups.

[0141] Step S741: Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

[0142] In summary, the data structure for querying persistent items in this invention is based on the bucket structure of the persistent item data structure (Persistence Sketch), which has the following technical effects and advantages:

[0143] First, the bucket structure enables precise querying of persistent items. Each bucket consists of a counting unit and a persistent item storage unit. The counting unit stores information about non-persistent items, while the persistent item storage unit stores information about persistent items. When inserting new stream information, the persistent item storage unit is queried first. If the information for that item exists, the data in the persistent item storage unit is updated directly; otherwise, the counting unit is updated first, and then the persistence values ​​of the counting unit and the persistent item storage unit are compared. If the persistence value of the counting unit is higher than the minimum value in the persistent item storage unit, the items are swapped. This allows for precise querying of persistent items.

[0144] Secondly, this data structure supports data merging across multiple sites, enabling persistence studies in a distributed environment. The bucket-based merging operation avoids data loss and maintains the same properties and accuracy as independent PersistenceSketch operations. This allows for accurate persistence estimation and persistent item queries in a distributed environment.

[0145] Third, the bucket structure uses a combination of counting units and persistent item storage units, resulting in lower memory consumption. This saves storage resources and enables the storage and processing of massive amounts of data within a limited space.

[0146] Based on the data structure for querying persistent items, a given threshold is determined in the query request initiated by the user terminal; all buckets in the bucket structure are traversed; and the IDs of items whose corresponding counter values ​​are greater than the given threshold are reported as the persistent items for this query. This data structure for querying persistent items enables accurate persistent item queries with relatively low memory consumption.

[0147] Based on the above Figure 3 The present invention provides a method for querying persistent items in a distributed stream, and also provides a device for querying persistent items in a distributed stream, applied to a data structure for persistence estimation. The data structure for persistence estimation consists of d counter arrays, each counter array consisting of l counters. Each of the l counters holds an HLL counter, which is an array of multiple data units, where the value in each data unit is 1 or 0. d and l are both positive integers. Figure 7 As shown, the device for querying persistent items in this distributed stream may include:

[0148] Hash mapping module 810 is used to calculate d hash functions h1, h2...h based on the ID of the target item. d d counters corresponding to the mapping are obtained. The target item is the item whose persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target item. The persistence represents the number of time windows in which the target item appears in the distributed stream.

[0149] The base estimation module 820 is used to calculate the base estimate of the HLL counters in the d counters respectively, and obtain the count value of each counter;

[0150] The counter comparison module 830 is used to take the counter with the smallest value among the d counters as the result of the persistence estimate of the target item in this query.

[0151] In summary, in this embodiment of the invention, d hash functions h1, h2...h are first calculated based on the ID of the target project.d The algorithm obtains d counters corresponding to the mapping, where the target item is the item for which persistence is estimated in this query. The persistence estimate represents the persistence of the target item, which is the number of time windows in which the target item appears in the distributed stream. Then, the cardinality estimate of the HLL counters in the d counters is calculated to obtain the count value of each counter. Finally, the counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query. This invention can accurately detect the scattered behavior of items in multiple windows of the data stream without significantly improving the algorithm's time complexity. It also solves the defect of existing sketch models that do not support persistence deduplication, and the query method can support instantaneous queries.

[0152] Based on the above Figure 5 The present invention provides a query device for persistent items in a distributed stream, which is applied to querying the data structure of persistent items. The data structure for querying persistent items includes a counter structure and a bucket structure. The counter structure is a counter array, which consists of l counters. For the l counters, each counter holds an HLL counter. The HLL counter is an array composed of multiple data units, where the value in the data unit is 1 or 0, and l is a positive integer.

[0153] The bucket structure consists of l buckets, each corresponding to a counter in the counter structure. Each bucket contains multiple key-value pairs, where the key is the item's ID and the value is the HLL counter corresponding to that item. Figure 8 As shown, the device for querying persistent items in this distributed stream may include:

[0154] The threshold determination module 910 is used to determine the given threshold carried in the query request initiated by the user terminal;

[0155] Traversal module 920 is used to traverse all buckets in the bucket structure;

[0156] The query module 930 is used to report the IDs of items whose corresponding counter values ​​are greater than the given threshold as persistent items for this query.

[0157] In summary, in this embodiment of the invention, the given threshold carried in the query request initiated by the user terminal is first determined, then all buckets in the bucket structure are traversed, and finally the IDs of items whose corresponding counter values ​​are greater than the given threshold are reported as persistent items for this query. Without significantly increasing the time complexity of the algorithm, this invention can not only accurately detect the scattered behavior of items on multiple windows of the data stream, but also solve the defect that the existing sketch model does not support persistent deduplication. Furthermore, this query method can support instantaneous queries.

[0158] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0159] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for querying persistent items in a distributed stream, characterized in that, A data structure for persistence estimation is provided, wherein the data structure for persistence estimation consists of d counter arrays, wherein each counter array consists of l counters, and for each of the l counters, each counter holds an HLL counter, wherein the HLL counter is an array consisting of multiple data units, wherein the value in the data unit is 1 or 0, and d and l are both positive integers; The query method includes: Calculate d hash functions h1, h2...h based on the target project's ID. d d counters corresponding to the mapping are obtained. The target item is the item whose persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target item. The persistence represents the number of time windows in which the target item appears in the distributed stream. Calculate the base estimate of the HLL counter in each of the d counters to obtain the count value of each counter; The counter with the smallest value among the d counters is taken as the result of the persistence estimate of the target item in this query.

2. The method according to claim 1, characterized in that, It also includes inserting items into the data structure for the persistence estimate according to the following steps: Calculate d hash functions h1, h2...h based on the project ID. d ; According to the d hash functions h1, h2...h d The mapping relationship is used to map each item to a corresponding d counter; Calculate the new hash function h′ based on the current time window t; According to the new hash function h′, the items are mapped to the data units of the HLL counter.

3. The method according to claim 2, characterized in that, The step of mapping the items to the data units of the HLL counter according to the new hash function h′ includes: Initialize all data cells to 0; In the d counters corresponding to the project, according to the new hash function h′, the project is mapped to the data unit of the HLL counter held by each of the d counters, and the value of the data unit to which the project is mapped is updated; If the value of this data cell is 0, it means that the target item has not yet accessed the counter in the current time window. When mapping the item, the value in the data cell will be set to 1. If the value of this data cell is 1, it means that the target item in the current time window has already accessed the counter, and the value in the data cell will not be changed when mapping the item.

4. The method according to claim 1, characterized in that, This also includes merging data from different sites using the following steps: The data in each data unit of the HLL counter between different site counters are combined to obtain the merged HLL counter; Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

5. A method for querying persistent items in a distributed stream, characterized in that, A data structure for querying persistent items, comprising a counter structure and a bucket structure, wherein the counter structure is a counter array consisting of l counters, and each of the l counters holds an HLL counter, wherein the HLL counter is an array consisting of multiple data units, the value of which is 1 or 0, and l is a positive integer; The bucket structure consists of l buckets, which correspond to l counters in the counter structure. Each bucket contains multiple key-value pairs, where the key is the ID of the item and the value is the HLL counter corresponding to the item. The query method includes: Determine the given threshold carried in the query request initiated by the user terminal; Iterate through all the buckets in the bucket structure; The IDs of items whose corresponding counter values ​​are greater than the given threshold are reported as persistent items for this query.

6. The method according to claim 5, characterized in that, It also includes inserting items into the data structure of the query persistence item according to the following steps: When performing the first insertion operation on each item, insert that item into the counter structure; When performing subsequent insert operations on each item, check if the item has already been recorded in the bucket structure: If the item is already recorded in the bucket structure, update the HLL counter corresponding to the item in the bucket structure; if the item is not recorded in the bucket structure, update the counter C[i] corresponding to the item in the counter structure. Compare the counter C[i] corresponding to each item in the counter structure with the smallest counter B[i] in the bucket structure. min Comparison: If C[i] > B[i] min Then B[i] will be... min Set the key to the project's ID and swap C[i] and B[i]. min If C[i] ≤ B[i] min If not, no further operations will be performed.

7. The method according to claim 6, characterized in that, The counter C[i] corresponding to the item in the updated counter structure includes: Calculate the hash function h1 based on the project ID; According to the mapping relationship of the hash function h1, the items are mapped to the counters in the counter structure; Based on the current project's time window t, the project is mapped to the corresponding HLL counter's data unit according to the new hash function h′.

8. The method according to claim 5, characterized in that, This also includes merging data from different sites using the following steps: For merging counters in a counter structure: The data in each data unit of the HLL counter between different site counters are combined to obtain the merged HLL counter; Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter; For merging counters in a bucket structure: If all items appear in the bucket structure of each site participating in the merger, then the data in each data unit of the HLL counter corresponding to the item is combined to obtain the merged HLL counter; If an item only appears in the bucket structure of certain sites, and no other counters for that item appear in the bucket structure, then the HLL counter held by the counter in the counter structure is used to participate in the merging operation. After merging, the item with the higher persistence estimate is taken as the merged HLL counter. Calculate the base estimate of the merged HLL counter to obtain the count value of the merged counter.

9. A device for querying persistent items in a distributed stream, characterized in that, A data structure for persistence estimation is provided, comprising d arrays of counters, each array consisting of l counters. Each of the l counters holds an HLL counter, which is an array of multiple data units, where the value in each data unit is 1 or 0. d and l are both positive integers, including: The mapping module is configured to perform the calculation of d hash functions h1, h2...h based on the ID of the target item. d d counters corresponding to the mapping are obtained. The target item is the item whose persistence is estimated in this query. The persistence estimate is used to represent the persistence of the target item. The persistence represents the number of time windows in which the target item appears in the distributed stream. The cardinality estimation module is configured to perform cardinality estimation for the HLL counters in the d counters respectively, and obtain the count value of each counter; The counter comparison module is configured to take the counter with the smallest value among the d counters as the result of the persistence estimate of the target item in this query.

10. A device for querying persistent items in a distributed stream, characterized in that, A data structure for querying persistent items, comprising a counter structure and a bucket structure, wherein the counter structure is a counter array consisting of l counters, and each of the l counters holds an HLL counter, wherein the HLL counter is an array consisting of multiple data units, the value of which is 1 or 0, and l is a positive integer; The bucket structure consists of l buckets, each corresponding to a counter in the counter structure. Each bucket contains multiple key-value pairs, where the key is the item's ID and the value is the HLL counter corresponding to that item, including: The threshold determination module is configured to determine the given threshold carried in the query request initiated by the user terminal. The traversal module is configured to traverse all buckets in the bucket structure. The query module is configured to use the IDs of items whose corresponding counter values ​​in the execution report are greater than the given threshold as persistent items for this query.

Images