Method and device for tracking frequent key

The Dynamic Dyadic Count-Min Sketch addresses the limitations of static universe data structures by allowing key length flexibility and reducing hash functions, enhancing efficiency and memory usage for tracking frequent keys in variable-length key scenarios.

WO2024248649A1PCT designated stage expired Publication Date: 2024-12-05HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/RU2023/000154
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing data structures for tracking frequent keys, such as Dyadic Count-Min Sketch (DCM), face challenges with a static universe, requiring a fixed key length at creation time, which limits their applicability and efficiency, especially when dealing with variable-length keys or growing key lengths.

Method used

The Dynamic Dyadic Count-Min Sketch (DDCM) method allows for tracking frequent keys of various lengths by extending the hash function domain without changing the hash value, enabling the data structure to scale with growing key lengths and reducing the number of hash functions, thereby supporting keys of different lengths and reducing memory usage.

Benefits of technology

The DDCM efficiently tracks frequent keys with lower memory usage and faster update and query times, making it suitable for applications with variable key lengths and reducing the need for large memory, such as network traffic monitoring and stream processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure RU2023000154_05122024_PF_FP_ABST
    Figure RU2023000154_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a method and a device for tracking a frequent key within limited memory, which can support keys of various lengths and expand with a growing key length. The method includes: receiving, a first key with a length of p; and performing, update and query operations based on a hash function, where a domain of the hash function is a first integer range; receiving, a second key with a length of p+1; performing, update and query operations based on the hash function, wherein the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key, where p is an integer.
Need to check novelty before this filing date? Find Prior Art

Description

METHOD AND DEVICE FOR TRACKING FREQUENT KEYTECHNICAL FIELD

[0001] Embodiments of the present invention relate to the field of cloud computing technologies, and more specifically, to a method and device for tracking a frequent key.BACKGROUND

[0002] Timely reacting to network traffic anomalies (like DoS attacks), tracking most active network participants, most active network streams, popular topics or posts in social networks, popular orders in online shops, and frequent values in data bases for query optimization-all of these can be attributed to the task of tracking a frequent item in a data stream.

[0003] A tracked item often has an identifier (ID) of a certain length, for example, an Internet Protocol (IP) address, a user ID, a post ID, or a connection ID. For example, an IPv6 address is an identifier of 128 bits in length. Such an identifier can be called a key and a set of all possible keys can be called a universe. Generally, a number of a frequent key may increase (when a tracked event occurs) as well as decrease (when the event leaves a time window of interest).

[0004] A Dyadic Count-Min Sketch (DCM) is a random approximate data structure for tracking the frequent key within a memory logarithmic in an universe size. However, the DCM has a crucial disadvantage: a static universe, in other words, the universe has to be fixed at creation time of the data structure, which is well-recognized in the literature and is reiterated even in the recent 2020 text book “Small Summaries for Big Data” by Cormode and Yi.SUMMARY

[0005] Embodiments of the present application provide a method for tracking a frequent key within limited memory, which can support keys of various lengths and expand with agrowing key length p.

[0006] According to a first aspect, a method for tracking a frequent key is provided, comprising: receiving, a first key with a length of p, wherein p is an integer; performing, update and query operations based on a hash function, wherein a domain of the hash function is a first integer range; receiving, a second key with a length of p + 1; performing, update and query operations based on the hash function, wherein the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key.

[0007] According to the method proposed by the present application, since the domain of the hash function can be extended from the first integer range to the second integer range without changing the hash value of the number in the first integer range, a data structure scales as required when a key of a previously unseen length is added. In other words, the method supports the keys of various length and expands with the growing key length. Therefore, there is more scene that the method can be applied for.

[0008] Therefore, the key length is not a priority restricted in the method, and the method is applicable to a key with a growing length. In addition, a number of used hash functions grows logarithmically with a maximum key length.

[0009] In some implementation manners of the first aspect, p is the maximum key length that be processed before receiving the second key with a length of p + 1. At each point in time, there is a maximum length p of the key seen so far. When receiving the second key with the length of p + 1, the data structure scales, guaranteeing that a number of used random hash functions grow with a largest logarithm in p and a number of memory use grow with a largest linear in p.

[0010] In some implementation manners of the first aspect, the hash function is an incrementally computable hash function.

[0011] In some implementation manners of the first aspect, the method further comprising: dividing, Count-Min sketches (CMs) Ao, A1,... , Ap+1used by the update and query operations into classes, the CMs within a same class use a same randomly chosen set of hash functions.

[0012] In some implementation manners of the first aspect, Ap+1including a hash value of the second key.

[0013] According to this embodiment, it is allowed that many CMs share the same randomly chosen set of hash functions. This lowers a running time of updating and querying, lowers the memory use, and lowers a number of random bits required to be chosen, since these are all proportional to the number of used hash functions.

[0014] In some implementation manners of the first aspect, the class of CMs is where ε is an inaccuracy of CMs and Q is a failure probability ofCMs, and ε and Q are both taken by the user.

[0015] In some implementation manners of the first aspect, a hash function of log 1 / 5 is used for each CM Alt..., Ap, and a hash functino of is used for CM Ao, where δis a constant failure probability of each of the CMs

[0016] In some implementation manners of the first aspect, wherein the first integer range , and the second integer range

[0017] According to a second aspect, a chip (or a chip system) is provided. The chip has a function of implementing the method in the first aspect and any embodiments of the first aspect. The function may be implemented by using a hardware structure. Alternatively, the chip or chip system includes an interface and a plurality of circuits.

[0018] According to a third aspect, a computing device is provided. The computing device has a function of implementing the method in the first aspect and any possible implementation manners of the first aspect. The function may be implemented by using a hardware, or may be implemented by executing a corresponding software by the hardware. The hardware or the software includes one or more units corresponding to the foregoing function.

[0019] In some implementation manners of the third aspect, the computing device comprises a processor and a memory coupled with the processor. The memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the method according to the first aspect or any implementation manners of the first aspect is implemented.

[0020] According to a fourth aspect, a computing device cluster is provided. The computingdevice cluster comprises at least one computing device, wherein the computing device comprises a processor and a memory coupled with the processor. The memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the method according to the first aspect or any implementation manners of the first aspect is executed.

[0021] According to a fifth aspect, a computer readable storage medium is provided, where the computer readable storage medium stores a computer instruction, and when the computer instruction runs on a computer, the method according to the first aspect or any implementation manners of the first aspect is executed.

[0022] According to a sixth aspect, a computer program product is provided, where the computer program product includes a computer program code, and when the computer program code runs on a computer, the method according to the first aspect or any implementation manners of the first aspect is executed.DESCRIPTION OF DRAWINGS

[0023] One or more embodiments are exemplarily described by corresponding accompanying drawings, and these exemplary illustrations and accompanying drawings constitute no limitation on the embodiments. Elements with the same reference numerals in the accompanying drawings are illustrated as similar elements, and the drawings are not limiting to scale, in which:

[0024] FIG. 1 is a schematic diagram of a method (100) for tracking a frequent key.

[0025] FIG. 2 is a schematic diagram of a universe growth of DDCM.

[0026] FIG. 3 is a schematic diagram of a change of an update speed and a query speed in dependence of a key length of p for a method DDCM and DCM.

[0027] FIG. 4 is a schematic block diagram of a chip or a chip system according to an embodiment of the present application.DESCRIPTION OF EMBODIMENTS

[0028] In order to understand features and technical contents of embodiments of the present disclosure in detail, implementations of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, and the attached drawings are only for reference and illustration purposes, and are not intended to limit the embodiments of the present disclosure. In the following technical descriptions, for ease of explanation, numerous details are set forth to provide a thorough understanding of the disclosed embodiments. One or more embodiments, however, may be practiced without these details. In other cases, well-known structures and apparatuses may be shown simplified in order to simplify the drawings.

[0029] Related technologies and concepts are introduced here firstly in order to better understand the technique solution proposed by the present application.

[0030] 1. Tracking a frequent key

[0031] Timely reacting to network traffic anomalies (like DoS attacks), tracking most active network participants, most active network streams, popular topics or posts in social networks, popular orders in online shops, and frequent values in data bases for query optimization-all of these can be attributed to the task of tracking a frequent item in a data stream.

[0032] Generally, a number of an item may increase (when a tracked event occurs) as well as decrease (when the event leaves a time window of interest). A tracked item often has an identifier (ID) of a certain length, for example, an Internet Protocol (IP) address, a user ID, a post ID, or a connection ID. For example, an Internet Protocol Version 6 (IPv6) address is an identifier of 128 bits in length. Such an identifier can be called a key and a set of all possible keys can be called a universe.

[0033] In the task that tracking a frequent key in a stream of key insertion and deletion events, the set U of possible keys is also called the universe, which is assumed to be the set of all p-bit words for a natural number p, or, equivalently, the set of all integers in the range U = {0, ..., 2P— 1},

[0034] 2. A basic data structure for tracking a frequent key.

[0035] For tracking a frequent key in a stream, a data structure supporting the followingthree operations is required:Initialization: takes a key length of p, initializes a count fs= 0 for each key s E U = {0 2P- 1}.Update: takes a key s ∈ U and an integer c and increments the count fsof s by c. The integer c is possibly positive or negative. Note that a negative value c represents a decrement. It is assumed that an updating is maintained invariant at fs> 0. It is denoted that a total number of all keys by n, which may increase or decrease after an update operation.Query: takes a number and outputs the key s ∈ U with thatis, with a frequency

[0036] It can be mathematically proved that any data structure supporting the above “update” and “query” operations requires at least n or at least |(U| bits of memory (|U| is a size of the set U, that is a number of elements in the set U). In the context of analyzing big data, this is infeasible, since both the universe U and the count n of events may be large, for example, when counting network packets arriving from various IPv6 addresses, the data structure will quickly exhaust main memory. Auxiliary memory may not be available or too slow for keeping up with a rapid update and query. For example, an embedded device or a programmable switche, which may be attractive for network traffic monitoring, usually do not have auxiliary memory. In contrast, a full-fledged stream processing engine like Apache Flink allows the use of auxiliary memory (“state”), but its content has to be frequently snapshotted in order to be migrated or restored when a computational node of a cluster fails.

[0037] 3. An approximate random data structure

[0038] When dropping the requirement for an exact answer to a “query” operation, it can be fitted a data structure for tracking a frequent key into a few megabytes of memory, regardless of a universe size and a stream length. This makes such a data structure suitable for tracking a frequent item even in an embedded device or a programmable switch. The key approach is an approximate and random data structure. Typically, it supports three operations:Initialization: works similar to the Initialization operation described in “2. A basic data structure for tracking a frequent key”, but takes two additional numerical parameters:• an inaccuracy ε ∈ (0,1], and• a failure probability Q E (0,1],Update: works similar to the Update operation described in “2. A basic data structure for tracking a frequent key”.Query: takes a frequency and guarantees that• all keys with frequency at least are output,• keys with frequency less than are output with probability at most Q only.

[0039] These guarantees are mathematically proved to hold in the worst case on arbitrary data. No previous knowledge on the data is required. In particular, the probability bounds do not come from assuming random input data of any particular distribution, but from random choices made by the update and query algorithms. The data structures usually consist of several counters, their number is proportional to

[0040] It is noted that all logarithms in the present application have to be understood as base-2 logarithms. For example, log 2 = 1. In addition, by [x] and [x], it is denoted that the number x rounded down or rounded up to the nearest integer, respectively.

[0041] This means that a very low failure probability like 210is achievable using a few counters only. Notably, the number of counters does not depend on the data stream length. Indeed, in experiments, data streams of arbitrary length may be tracked using several megabytes of memory.

[0042] 4. A Count-Min Sketch (CM)

[0043] Based on the above data structures, Count-Min Sketch (CM) and Dyadic Count-Min Sketch (DCM) have been developed for approximating the counts of keys. CMs are used as a building block in the present application, without going into its realization details.

[0044] The CM is a data structure for approximating the counts of keys. A CM A supports three operations:Initialization: takes a key length p, an inaccuracy , a failure probabilityQ ∈ (0,1], and initializes a count fs= 0 for each .Update: takes a key s ∈ U and an integer c and increments the count fsof s by c.Query: takes a key s ∈ U and returns an estimate A[s] of fs. The estimate isguaranteed to satisfy With probability at least 1 — Q, it also satisfiesεn, where n is a total count of all keys.

[0045] Cormode and Muthukrishnan (2005) have published a construction of a CM using 2 1 1-log- counters and log- randomly chosen hash functions. A running time for updating and querying a key s is dominated by the time for computing all of the hash functions on the key s.

[0046] It is important to note that the CM does not store any keys s nor any of the counts fsexplicitly. Thus, it is impossible to obtain a list of all keys stored in a CM and their exact counts. One only obtains an estimate A[s] of its count.

[0047] 5. A Dyadic Count-Min Sketch (DCM)

[0048] The DCM is based on the CM and the DCM realizes the interface described in “3. An approximate random data structure, that is the DCM is a random approximate data structure for tracking a frequent key. A universe U of a DCM is a set of all binary words of length p. That is, In the DCM, CMs A support three operations:“Initialization”: takes a key length p , an inaccuracy ε G (0,1) , a failure probability Q G (0,1] , and initializes CMs each with an independentlyrandomly chosen set of hash functions. A universe of each Ai is a binary word of length i, which is naturally identified with an integer in the range {0, ..., 2i— 1}.It is enough to know that will estimate a total count of keys starting with ani-bit prefix x and for an empty prefix x will just track a total count n of all keys.“Update”: takes a p-bit key s and an integer c, and, for each prefix length i G {0, ...,p}, increments the count of the i-bit prefix x of s in A, by c.“Query”: takes a frequency and, by calling the following procedureRecurse with the empty prefix x as input, outputs all keys of which the frequency is estimated to be at least :Procedure RecurseInput: binary word x of length !Output: keys starting with prefix x and having estimated frequency at least <p. BeginIf then beginIf i = p then output xElse beginCall Recurse with binary word x = x0Call Recurse with binary word x = xlEndEndEnd

[0049] S ince the universe U of keys for the DCM is the set of all p-bit words, there isTo achieve query answers with inaccuracy at most ε, the inaccuracy of each of the CMs is also chosen to be ε. In order to achieve a failure probability of at most Q,two variants have been suggested in the literature:1. In the original variant of Cormode and Muthukrishnan (that is also called DCM variant 2005 in the present application), the CMs Ao, ...,Apare created with failure probability each- This leads to a use of random hash functions andcounters.2. The improved variant of Larsen et al. (that is also called DCM variant 2016 in the present application) creates the CM Apwith a failure probability and the CMswith a failure probability each. This leads to a use of randomhash functions and counters.

[0050] It can be seen from the above that the performance of updating and querying of a DCM is determined by the number of hash functions: each query or update of a CM leads to the computation of each of its hash functions on the queried or updated key. Obviously, in an answer for a query for a frequency in a DCM, there will never be more than p keys inthe output. Thus, for both variants, the time for an update operation or a query operation for a constant frequency is proportional to computing all hash functions on all prefixes of aconstant number of keys.

[0051] However, the DCM has a crucial disadvantage: the static universe, which is well- recognized in the literature and is reiterated even in the recent 2020 text book “Small Summaries for Big Data” by Cormode and Yi. This disadvantage is described in detail below.

[0052] Both realizations of the DCM require the universe, that is, the number of bits in thekeys, to be known at a creation time. This makes their use very limited to, for example, tracking a frequent IP address or a frequent product and a user ID, but makes them not usable for stream processing in general.

[0053] It is generally impossible to give an upper bound on the key length in a data stream in advance, at initialization time of a data structure. For example, popular topics, words or “hash tags” to be tracked in a social network, do not have a bounded length. It may be tempting to just choose a large key length p to accommodate for most use cases. But, on the one hand, this will unnecessarily waste a lot of memory and time, since a large number of random hash functions will have to be generated, stored, and computed. On the other hand, it is unclear how the stream processing system is supposed to behave when suddenly a too long key arrives.

[0054] A previously known DCM consists of CMs for key length p =log | U|. The universe of each CM Ai is the set of i-bit words and is an estimate of thenumber of keys of which the first i bits coincide with those of x. Allowing keys of variable lengths introduces several technical problems.1. Which of the CMs tracks the count of a key with less than p bits? A possible solution is to make all keys have a length p by adding a leading zero, but this makes the keys 1 and 01 indistinguishable, for example.2. When adding a key of length p + l, anew CM Ap+1has to be added. Howto interpret the estimates that are given by the CMs Ao, Ap? Do they count keys of length p or keys of length p + 1? Recomputing the estimates of already existing keys is not possible since none of the CMs stores an actual key and its actual count.3. For an original 2005 DCM: the number of hash functions of each of the CMs depends onAdding a key of length p + 1 requires not only the introduction of a new CM Ap+1, but also the introduction of a new hash function for each CM Ao, ... ,Ap. The CMs, however, do not store any keys explicitly, so that computing the value of old keys under new hash functions is impossible.

[0055] An obvious solution is using a separate DCM for each key length. Since the key length ranges from 0 to, this will lead to using log | U\ DCMs, which contain a totalCMs. Thus, this would increase the number of hash functions, counters, and running time by an additional factor of |.

[0056] In addition, except the problem of static universe, the large number of random hash functions are required is another disadvantage of the DCM. The large number of random hash functions required for the DCM implies at least two problems:1. For choosing a hash function at random, it needs to generate a number of random bits that are proportional to the key length. The random bits chosen for each of the hash functions have to be stored in memory in order to compute values of hash functions on keys later. The large number of random hash functions will lead to the large number of stored random bits, which will take up huge memory.2. The running time of updates and queries of the DCM is proportional to the time for computing all hash functions on all prefixes of updated or returned keys. Its memory use is also proportional to the number of used hash functions.

[0057] For example, a 2016 variant of the DCM uses hash functions forcovering a universe U = {0, ..., 2P— 1} of p-bit keys. Each hash function is computed for each prefix of each key involved in an update or query operation. Computing the hash value of a key of length takes time. Thus, the update and query time of theDCM is actually proportional to which is quadratic in the key length. This also means that the number of random bits that need to be generated andstored for the hash functions is quadratic in p. The number of random hash functions required for an original 2005 version of the DCM is even larger.

[0058] In view of all of above, a method and device for tracking a frequent key are proposed, which could support keys of various lengths and expand with a growing key length. In addition, it is helpful for lowering the number of hash functions.

[0059] The new data structure in the present application called Dynamic Dyadic Count-Min Sketch (DDCM) builds on the 2016 variant of the DCM and solves both of its problems: the data structure scales as required when a key of a previously unseen length is added. At the same time, a DDCM uses strictly less hash functions.

[0060] The scenario of the present application is introduced at first.

[0061] In one scenario, the new data structure could realize as an operator in a stream processing engine.

[0062] Specifically, the DDCM may be used to implement an operator that will give a stream processing engine the possibility to track a popular topic in a discussion, a popular order in a online shop, a frequently appearing event and a condition in a production line (machine monitoring). Whenever a new tuple arrives in a stream, the user extracts an interesting key and feeds it to the operator. Whenever a tuple is retracted or leaves an interesting time window, the user feeds a decrement update to the operator. The operator will answer queries for a frequent key at any request time.

[0063] The advantage of using a DDCM instead of a DCM or instead of keeping counts in a hash table is that the DDCM will typically fit into the main memory of the server running the operator, so that the customer and cloud provider can save on memory cost. Migrating or restoring the work of the operator on a server failure will be cheap, since no large memory contents has to be migrated or snapshotted. Compared to a DCM-based solution, the customer will also profit from a higher profit.

[0064] Since the DDCM does not require a bound on key length, its use cases are more versatile than those of a previously known DCM. The streaming system will be well-behaved at all times, and there is no need to handle exceptions for too long keys.

[0065] In another scenario, the new data structure could realize on a programmable network switch or an embedded device.

[0066] Specifically, a lower memory footprint of the DDCM over the known DCM makes it suitable for realization on the embedded device or the programmable network switch without external memory. This allows for tracking highly loaded network connections or detecting the source of network anomalies like DoS attacks directly within the network hardware, allowing for a higher throughput of updating to the data structure and allowing for lower power consumption than realization on a full-fledged server.

[0067] The following describes a proposed solution of the present application in more detail.

[0068] FIG. 1 is a schematic diagram of a method (100) for tracking a frequent key. The method (100) specifically includes the following steps:

[0069] Step 110: receiving a first key with a length of p.

[0070] Step 120: performing update and query operations based on a hash function, where a domain of the hash function is a first integer range.

[0071] Step 130: receiving a first key with a length of p + 1.

[0072] Step 140: performing, update and query operations based on the hash function, wherein the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key.

[0073] Exemplarily, the first integer range U is U = (0,1,2, ... , 2P— 1], and the second integer range U' is U' = {0,1,2, ..., 2P+1— 1}.

[0074] The DCM over a universe U of p -bit words could conveniently identify its universe with an integer in the range {0, ... , 2P— 1}. This identification becomes ambiguous in presence of a variable- length key. For example, the keys 1 and 01 will be represented by a same number. For a one-to-one correspondence between keys and numbers, each p-bit key s is represented as a number with p + 1 bits in the present application, of which the most significant bit is 1 and the other bits are taken from s. ft is denoted that this number is Is. The empty key s is represented by Is = 1.

[0075] Like the DCM, the DDCM holding a key of length up to p also consists of CMs / loMi, ...,Ap, but uses them differently. Indeed, the DDCM will maintain the following invariant based on the DCM: A^_i[lx] estimates the total count of keys of length T of which a length-i prefix is x. When a key of length p + 1 arrives, a new CM Ap+1to the DDCM is added. For the invariant to hold, the universe size of each previously created CM Akincreases by a factor 2p+1-k: Aknow additionally estimates the total count of keys of length p + 1 that have prefixes of length p + 1 — k.

[0076] Denoting the total count of keys of length / of which a length-i prefix is x as Q[lx], FIG.2 illustrates a correspondence for p = 2. It can be said that FIG.2 shows a final result of growing from p = 2 to p + 1 = 3.

[0077] As shown in FIG.2, when a key of length p + 1 = 3 arrives, a new CM A3is added to the DDCM. For the invariant to hold, the universe size of each previously created CM Akincreases by a factor 23~k. Aknow additionally estimates the total count of keys of length p + 1 = 3 that have prefixes of length 3 — k. Four steps of this universe growth are indicated by the rectangles in the FIG. 2 from left to right. By this way, an indexation of CMs seems inconvenient at first, but conveniently allows the data structure to grow.

[0078] In the present application, the universe growth of the CMs will be done fullyautomatically by a suitable choice of a family of random hash functions, where a domain of the hash function can be extended from an integer range U — {0, ... , 2P— 1} to an integer range U' = {0, ... , 2P+1— 1} without changing a hash value of a number in an old range U.

[0079] For example, the hash function is an incrementally computable hash function.

[0080] Specifically, an example for such an incrementally computable hash function has been described already in 1979 by Carter and Wegman: a number s of which a binary representation is s1(s2, —> sp, and it is mapped to a q-bit hash valuewhere d® is a randomly chosen vector of q bits and is their combinationvia bitwise exclusive “or”. Inside a CM, h(s) is used to index an array in memory, so that h(s) and a random vector d® fit into a machine word. More specifically, one can choosewhereis the inaccuracy of the CM.

[0081] Extending the universe of the hash function h(s) to a number with p + 1 bits is as simple as choosing a new random q-bit vector d^p+1\ The procedure for computing h(s) can randomly choose and store an additional random vector on demand whenever the number of already stored vectors is lower than the number of bits in the number that the hash value is computed for. Using the incrementally computable hash function realized in this way in the CMs of a DDCM allows to consider CMs as a black box of which the universe is the set of all natural numbers. No further action is needed to extend their universe for growing the DDCM.

[0082] According to the present application, since the domains of these hash functions can be extended from the first integer range U = {0,1,2, ... , 2P— 1} to the second integer range U' = {0,1,2, ... , 2P+1— 1} without changing the hash value of the number in the first range U, the data structure scales as required when the key of the previously unseen length is added. In other words, the method supports the key of various lengths and expands with the growing key length.

[0083] Therefore, there is more scene that the method can be applied for.

[0084] Additionally, CMs used by the update and query operations are partitioned into classes, the CMs within the same class use the same randomly chosen set of hash functions.

[0085] Like the DCM, for the DDCM to guarantee a failure probability of at most Q, theCMs >1^ ... ,Ap+1with any constant failure probability 6 and the CM Aowith afailure probability are chosen. The known DCM chooses the hash function for eachCM independently at random. This leads to the use of lrandom hash function for each CM A-^, , Ap+1, and to O hash function for Ao. In DDCM, a random set of hashfunctions only for Ao, and the remaining CMs At, ...,Ap+1use the same number of hash functions. In addition, the remaining CMs Alt... , Ap+1are split into classes. Each class chooses its set of hash functions independently at random. Two CMs of the same class use the same set of hash functions.

[0086] In detail, the CM Aogets its own, independently and randomly chosen set of hash functions. The other CMs Ar, ... ,Ap+1in a DDCM are partitioned into classes. The class of Akis denoted by H(k). CMs within the same class use the same randomly chosen set of hash functions. The base idea is to cycle through a fixed number of classes. For example, when cycling through three classes, the CM Akwill get class number H(k) = 3 — (k mod 3) . However, in order to meet required inaccuracy bounds and failure probabilities, the length of the cycle has to grow with the length of the keys held in the DDCM. Since the key length is unknown in advance, an additional class is added into the cycle every now and then.

[0087] In detail, the CM Ap+1including a hash value of the second key.

[0088] In particular, an assignment of CMs to classes that is parameterized by a natural number d and a sequence of natural numbers is used. The CMs Akis split into consecutive blocks. Block i has a size ) and cycles the classes from class d + idown to 1 for times. Assume that each of the CMs Alf... , Ap+1has a constant failure probability Then, it can be proved that, if one wants to satisfy the same inaccuracybound ε and failure probability Q as the known DCM with probability at least 1 — O’, then one can chooseand, for any such thatand CMs could be assigned to ) classes, for each of which aconstant number of hash functions could be chosen independently at random. For example, and can be chosen.

[0089] Summarizing and maintaining the same inaccuracy bound ε and failure probability bound Q as the known DCM and DDCM reduces the number of required hash functions to

[0090] It is noted that, for all practical purposes, an item of meets the following:Because there are no more than 229particles (and, hence, keys) in the observable physical universe.

[0091] The DDCM allows many CMs to share the same randomly chosen set of hash functions. This lowers the running time of updates and queries, lowers the memory use, and lowers the number of random bits required to be chosen, since these are all proportional to the number of used hash functions.

[0092] However, the DDCM allows a two-sided error instead of a one-sided error: with a probability Q, a query for keys of frequencymay not find all keys of frequency <p or may output some keys of frequency less than The previously known DCM has only onesided error, it guarantees to always output all keys of frequency

[0093] Method 100 is a general description about DDCM, embodiments about it will be introduced specifically later in the present application.

[0094] The example implementation of the DDCM is almost identical to the implementation of the DCM, it just takes into account the possibility for adding more CMs and incrementally computes hash values. One needs to implement three operations:Initialize: takes an inaccuracy a failure probabilityinitializes a CM Xowith inaccuracy ε and a failure probability , and sets maximumlength p = 0.Update: takes a f-bit key s and an integer c and updates the count of s by c using the following procedure:Procedure: UpdateInput: key s of length €, natural number c.BeginIf £ then begininitialize new C with inaccuracy ε and failure probabilityeach Cuses the hash functions for class Hfk) as described in method 100. set new maximum length p to ■£EndFor each prefix lengthLet x be the length-i prefix of sFor each hash function h in the classes doIf i = 1 then compute 7t(lx)Else compute fi(lx) from 7t(ly) for the prefix y of x of length i — 1EndIncrement[lx] by c, using the precomputed hash values.EndEndQuery: takes a frequency and outputs all keys of which the frequency isestimated to be at least. The procedure is as the following:Procedure: QueryInput: frequencyp [ ]Output: all keys with estimated frequency at least <p.BeginFor each key length call Recurse with empty word x, length f,and zero hash valuesEndProcedure RecurseInput: binary word x of length i, key length f, h(lx) for each hash function h in the classesOutput: keys of length f, starting with prefix x, and having estimated frequency at least (p.BeginIfthen begin the CM checks this usingprecomputed valuesIf i = f then output xElse beginFor each hash function h in the classesdoFrom incrementally compute andEndCall Recurse with binary word x = xO, lengthand hash values h(lxO)Call Recurse with binary word x = xl, lengthand hash values h(lxl)EndEndEnd

[0095] It is worth noting that the recursive implementation of the query procedure serves the purpose of an easily understandable and correct sample implementation. In order to prevent overflows of the function call stack, it makes sense to implement the query procedure using iteration, explicitly handling a stack data structure on the heap.

[0096] The following Table 1 compares the difference between the method 100 and various variants of DCM. It can be seen from Table 1 how the number of counters, the number of hash functions, update time and query time depend on the universe size |t7|, allowed inaccuracy ε G (0,1), failure probability Q G (0,1], and query frequency <p G [0,1] (constant factors are ignored).Table 1

[0097] The difference between the DDCM and the state of the art has also be compared experimentally. To this end, prototypes of DDCM and DCM are implemented as an operator in the stream processing engine Apache Flink, version 2.12. Flink 2.12 was running on Java 8, Ubuntu 20.04, powered by an Intel Core i7-9700K CPU at 3.6 GHz. The prototype uses only one out of the eight CPU cores. For both data structures, we set an inaccuracy bound ε —0.0001, failure probability Q = 0.01, and query all keys with frequency at least <p = 0.001. The following Table 2 and FIG. 3 compare the performance of the update and query operations of DDCM to DCM in dependence of key length p. The Table 2 also shows the total count of all keys n and the number of frequent keys. FIG.3(a) shows the change of update speedin dependence of key length p. FIG.3(b) shows the change of query speed in dependence of key length p.Table 2

[0098] It can be seen from FIG.3 that for keys of at least 100 bits of length, the throughput of DDCM can be observed to be at least two times higher than that of DCM. The advantage of DDCM grows for larger key lengths due to the asymptotically lower number of hash functions involved in the computations. The throughput is about four times higher for keys of a length more than 400 bits.

[0099] All related contents of steps in a method embodiment have been described in detail in the foregoing system embodiment, and in order to avoid repetition, details are not repeated here.

[0100] FIG.4 is a schematic block diagram of a chip or a chip system according to an embodiment of the present application.

[0101] As shown in FIG.4, the chip (or the chip system) may include an interface 11 and a plurality of circuits 12. Alternatively, the interface 11 is configured to receive a first key with alength of p. The plurality of circuits 12 is configured to perform the method 100, such as to perform update and query operations based on a hash function, where a domain of the hash function is a first integer range. And the interface 11 is further configured to receive a second key with a length of p + 1. The plurality of circuits 12 is further configured to perform the method 100, such as to perform update and query operations based on the hash function, where the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key.

[0102] Alternatively, the interface 11 includes an input interface and an output interface, where the input interface is configured to receive the first key with a length of p and receive the second key with a length of p + 1, and the output interface is configured to output the querying result of the plurality of circuits 12. As an embodiment, the interface 11 is an interface circuit.

[0103] As an embodiment, the function of the chip is implemented by hardware, such as the plurality of circuits 12, and the hardware includes one or more corresponding structures, such as multiplexers, controllers, re-coders and logical gates and so on.

[0104] Alternatively, the hash function is an incrementally computable hash function.

[0105] Alternatively, the plurality of circuits 12 is further configured to divide Count-Min sketches (CMs) Ao, Alt...,Apused by the update and query operations into classes, the CMs within a same class use a same randomly chosen set of the hash function, and Ap+1including a hash value of the second key.

[0106] Alternatively, the class of the CMs is ε is tan inaccuracyof the CMs and Q is a failure probability of the CMs, and s and Q are both taken by the user.

[0107] Alternatively, the first integer range U is U = {0,1,2, ..., 2P— 1} , the second integer range U' is U' = {0,1,2, ..., 2P+1— 1}.

[0108] The present application also provides a computing device or a computing apparatus including at least one processor coupled to at least one memory, and the at least one processor is configured to execute computer programs or instructions stored in the at least one memory to cause the device to execute the method for tracking frequent keys as provided in the present application.

[0109] The present application also provides a computing device cluster. The computing device cluster comprises at least one computing device, and the computing device including at least one processor coupled to at least one memory, and the at least one processor is configured to execute computer programs or instructions stored in the at least one memory to cause the device to execute the method for tracking frequent keys as provided in the present application.

[0110] The present application also provides a computing device including a processor and a communication interface, where the communication interface is configured to receive data and / or information, and transmit the received data and / or information to the processor, and the processor is configured to process the data and / or information, and the communication interface is also configured to output the data and / or information that has been processed by the processor, so that the method for tracking frequent keys as provided in the present application is executed.

[0111] The present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer instruction, and when the computer instruction is executed on a computer, the method for tracking a frequent key provided by the present application is executed.

[0112] The present application also provides a computer program product, where the computer program product includes a computer program code, and when the computer program code runs on a computer, the method for tracking a frequent key provided by the present application is executed.

[0113] Optionally, the foregoing processor may be one or more, and the memory may be one or more.

[0114] In the foregoing embodiments, the processor and the memory may be coupled, or may be integrated together.

[0115] In the foregoing embodiments, the processor may be a central processing unit (central processing unit, CPU), a general purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or another programmable logic device, a discrete gate or a transistor logic device, a discrete hardware component, a microprocessor, one or more integrated circuits configured to control program execution in the solutions of the present application, or the like. For example,the processor may include a digital signal processor device, a microprocessor device, an analog- to-digital converter, and a digital-to-analog converter. The processor may allocate control and signal processing functions of mobile devices between these devices based on respective functions of the devices. In addition, the processor may include a function for operating one or more software programs, and the software program may be stored in a memory. The function of the processor may be implemented by hardware, or may be implemented by executing corresponding software by hardware. The hardware or the software includes one or more units corresponding to the foregoing function.

[0116] The memory may be a read-only memory (read-only memory, ROM) or another type of static storage device that is capable of storing static information and a static instruction, or a random access memory (random access memory, RAM) or another type of dynamic storage device that is capable of storing information and an instruction, or may be an electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), a compact disc read-only memory (compact disc read-only memory, CD- ROM) or other compact disc storage, optical disc storage (which includes a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, and the like), a disk storage medium or another magnetic storage device, or any other medium that can be used to carry or store expected program code having an instruction or a data structure form and that can be accessed by a computer. It should be noted that the memory in the systems and the methods described in this specification includes but is not limited to these memories and a memory of any other appropriate type.

[0117] In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0118] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0119] In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0120] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in the present application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disc.

[0121] The foregoing descriptions are merely specific implementations of the present application, but are not intended to limit the protection scope of the present application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in the present application shall fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A method for tracking a frequent key, comprising: receiving, a first key with a length of p, wherein p is an integer; performing, update and query operations based on a hash function, wherein a domain of the hash function is a first integer range; receiving, a second key with a length of p + 1; performing, update and query operations based on the hash function, wherein the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key.

2. The method according to claim 1, wherein the hash function is an incrementally computable hash function.

3. The method according to claim 1 or 2, the method further comprising: dividing, Count-Min sketches (CMs) A0,A1, ... , Ap+1used by the update and query operations into classes, wherein the CMs within a same class use a same set of hash functions, and Ap+1including a hash value of the second key.

4. The method according to claim 3, wherein the class of the CMs isε is an inaccuracy of the CMs and Q is a failure probability of the CMs, and E andQ are both taken by the user.

5. The method according to any one of claims 1 to 4, wherein the first integer range U is U = {0,1,2, ... , 2P— 1}, and the second integer range U' is U' = {0,1,2, ... , 2p+1— 1}.

6. A chip, comprising: an input interface, configured to receive a first key with a length of p, wherein p is an integer; a plurality of circuits, configured to perform update and query operations based on a hash function, wherein a domain of the hash function is a first integer range; the input interface is further configured to receive a second key with a length of p + 1; the plurality of circuits are further configured to perform update and query operationsbased on the hash function, wherein the domain of the hash function can be extended from the first integer range to a second integer range without changing a hash value of the first key.

7. The chip according to claim 6, wherein the hash function is an incrementally computable hash function.

8. The chip according to claim 5 or 6, wherein the plurality of circuits are further configured to: divide Count-Min sketches (CMs) Ao, Alt..., Apused by the update and query operations into classes, the CMs within a same class use a same randomly chosen set of the hash function, and Ap+1including a hash value of the second key.

9. The chip according to claim 8, wherein the class of the CMs isε is tan inaccuracy of the CMs and Q is a failure probability of the CMs, and ε and Q are both taken by the user.

10. T The chip according to any one of claims 6 to 9, wherein the first integer range U is U = {0,1,2, ... , 2P— 1}, and the second integer range U' is U' = {0,1,2, ... , 2p+1— 1}.

11. A computing device cluster, comprising at least one computing device, wherein the computing device comprising a processor and a memory coupled with the processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method according to any one of claims 1 to 5.

12. A computer readable storage medium, wherein the computer readable storage medium stores a computer instruction, and when the computer instruction runs on a computer, the method according to any one of claims 1 to 5 is executed.

13. A computer program product, wherein the computer program product comprises a computer program code, and when the computer program code runs on a computer, the method according to any one of claims 1 to 5 is executed.