Stream processing method and computing device

Abstract

A stream processing method includes: obtaining a candidate generation sketch including k blocks; obtaining a verification sketch including k´ blocks; receiving records in a stream with insertions and / or retractions, each of the records including a key and a count that are in correspondence; updating the sketches by transferring each record to one of the k blocks of the candidate generation sketch and one of the k´ blocks of the verification sketch; querying the k blocks of the candidate generation sketch to determine a set of candidate keys; dividing the set of candidate keys into k´ sub-sets of candidate keys; testing each sub-set of candidate keys by using one of the k´ blocks of the verification sketch to obtain a verified current list of frequent keys; and outputting a current list of frequent keys in the insertion stream and / or retraction stream.

Description

STREAM PROCESSING METHOD AND COMPUTING DEVICETECHNICAL FIELD

[0001] The present disclosure relates to the field of computing technologies, and in particular, to a stream processing method and a computing device.BACKGROUND

[0002] At present, for the task of tracking frequent items in a stream, an approximate randomized data structure used to track frequent keys is fit into hundreds of megabytes of memory, so that the data structure is suitable for tracking frequent keys even in embedded devices or programmable switches. Existing methods either do not support retractions, or only support limited retractions, or only support a given key frequency query but cannot output a list of the most frequent items in the presence of retractions, or are not cache efficient, that is, do not use memory hierarchy to their advantage.SUMMARY

[0003] Some embodiments of the present disclosure provide a stream processing method and a computing device, thereby reducing memory consumption, outputting a list of the most frequent keys, and improving stream processing efficiency.

[0004] In a first aspect, a stream processing method is provided. The stream processing method includes: obtaining a candidate generation sketch, the candidate generation sketch including k blocks (where each block can also be referred to as candidate generation sub-sketch), k being an integer greater than or equal to 1 ; obtaining a verification sketch, the verification sketch including k' blocks (where each block can also be referred to as verification sub-sketch), k' being an integer greater than or equal to 1 ; receiving a plurality of records in a insertion stream and / or retraction stream, each of the records including a key and a count that are in correspondence; updating the key and the count by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k' blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch; querying the k blocks of the candidate generation sketch to determine a set of candidate keys; dividing the set of candidate keys into k' sub-sets of candidate keys, the k' subsets of candidate keys being in one-to-one correspondence with the k' blocks of the verification sketch; testing each sub-set of candidate keys by using one block of the verification sketch of the k' blocks of the verification sketch to obtain a verified current list of frequent keys; and outputting the current list of frequent keys in the stream with insertions and / or retractions.

[0005] In the scheme provided in some embodiments of the present disclosure, by obtaining the candidate generation sketch including k blocks, each of which consists of a set of majority testers and the verification sketch including k' blocks each of which consists of a set of counters, during the update, the record in the received input stream is transferred to one target block of the candidate generation sketch and one target block of the verification sketch corresponding to the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is continuously loaded into faster memory and corresponding update processing is carried out in the memory. Therefore, one block may be put into faster memory, and during the update, only one block of the target candidate generation sketch and one block of the target verification sketch need to be accessed. In addition, although the entire data structure is stored in a slower, larger memory level, one required block may be fully loaded into faster memory during the update; and thus, the update may be executed in faster memory and takes less time. During the query each block of candidate sketch is processed sequentially and needs to be processed once, thereafter each block of candidate sketch needs to be processed at most once. Thus, queries as well as updates benefit from loading one required block at a time into faster memory where the execution takes less time.

[0006] In one possible implementation, updating the key and the count by transferring each record to the target block of the candidate generation sketch and the target block of the verification sketch, includes: determining an r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on a predefined first function, wherein r E {1, ... , k] ; transferring each record to the target block of the candidate generation sketch; determining an r'-th block of the verification sketch as the target block of the verification sketch based on a predefined second function, wherein r' E {1, ..., k'} ; transferring each record to the target block of the verification sketch; updating majority testers corresponding to the key in each record in the target block of the candidate generation sketch; and updating the counters corresponding to the record’s key in the target block of the verification sketch.

[0007] During the update, the record in the received input stream is transferred to one target block of the candidate generation sketch and one target block of the verification sketch corresponding to the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is continuously loaded into faster memory and corresponding update processing is carried out in the memory.

[0008] In another possible implementation, determining the r -th block of the candidate generation sketch as the target block of the candidate generation sketch based on the predefined first function, includes: for a record (s, c) with key s and count c, determining that r = h0(s) based on a hash function h0: U -» {1, ... , k }, the r-th block of the candidate generation sketch being thetarget block of the candidate generation sketch; and determining the r'-th block of the verification sketch as the target block of the verification sketch based on the predefined second function, includes: for a record (s, c) with key s and count c, determining that r' = / t'0(s) based on a hash function h'o: U -> {1, ... , k'}, the r'-th block of the verification sketch being the target block of the verification sketch.

[0009] In another possible implementation, for each record (s, c) with key s and count c, updating majority testers corresponding to the key s in the target block of the candidate generation sketch, includes: obtaining an array of majority testers Mijr, i = 1,j = 1, ... , w / k corresponding to the target block of the candidate generation sketch r G {1, ... , kj; and updating the majority testers by sending d copies of the corresponding record to the array of majority testers selected via hash functions h^ U{1, ... , w / k}, i = 1, ... , d.

[0010] In the embodiments of the present disclosure, each of majority testers is used to provide candidate keys during the later query. Compared with the existing NAGT, in the embodiments of the present disclosure, the majority testers are not used for verification, so that the number of the majority testers needed in the stream processing is smaller, and the small number of the majority testers makes the memory consumption of storing the majority testers less, thereby reducing the memory consumption of stream processing, and improving the processing efficiency. Moreover, less majority testers lead to a smaller set of candidate keys they generate during query, which in turn leads to less time for verifying those keys, and hence improves processing efficiency.

[0011] In yet another possible implementation, for each record (s, c) with key s and count c, updating the counters accordingly in the target block of the verification sketch, includes: obtaining an array of counters Cijr, i = 1, ... , d! , j = 1, ... , w' / k' corresponding to the target block of the verification sketch r' 6 {1, ... , k'}; and updating the counters Cih^s)rlselected via hash functions h'l'. U -» {1, ... , w' / k'}, i = 1, ... , d', by incrementing them by count c.

[0012] In one possible implementation, obtaining the candidate generation sketch and obtaining the verification sketch includes: obtaining a set of parameters; determining a total width and a total depth of the candidate generation sketch, and a number k of blocks of the candidate generation sketch based on the set of parameters; and determining a total width and a total depth of the verification sketch, and a number k' of blocks of the verification sketch based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

[0013] In another possible implementation, the set of parameters includes at least one or more of: a maximum length of a key, a minimum frequency that will be of interest during run of a streaming program, an inaccuracy, a failure probability, or a block size; determining the total width and the totaldepth of the candidate generation sketch based on the set of parameters includes: determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during run of the streaming program, the failure probability, and the block size; determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability; and determining the total width of the candidate generation sketch based on the total depth of the candidate generation sketch, and the correspondence relationship between the total width and the total depth of the candidate generation sketch.

[0014] In another possible implementation, the correspondence relationship between the total width and the total depth of the candidate generation sketch includes: w is proportional to J^ln^ • — ; wherein p is the maximum length of the key,is the minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

[0015] In another possible implementation, determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability, includes: d is proportional to log~ \ wherein C> is the minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, and d is the total depth of the candidate generation sketch.

[0016] In one possible implementation, determining the total width and the total depth of the verification sketch based on the set of parameters as well as the total width and the total depth of the candidate generation sketch, includes: determining a correspondence relationship between the total width and the total depth of the verification sketch based on the inaccuracy, the failure probability, and the block size; determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch; and determining the total width of the verification sketch based on the total depth of the verification sketch and the correspondence relationship between the total width and the total depth of the verification sketch.

[0017] In another possible implementation, the correspondence relationship between the total width and the total depth of the verification sketch includes: w is proportional towherein £ is the inaccuracy, g is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0018] In one possible implementation, determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch includes: d' is proportional to log wherein Q is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

[0019] In one possible implementation, each block of the candidate generation sketch has a width of w / k and a depth of d, wherein w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch; each block of the verification sketch has a width of w' / k' and a depth of d', wherein w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0020] In one possible implementation, determining the number k of blocks of the candidate generation sketches based on the set of parameters, includes: the number k of blocks of the candidate generation sketch is proportional towherein p is the maximum length of the key, and B is theblock size; and determining the number k' of blocks of the verification sketch based on the set ofparameters, includes: the number k' of blocks of the verification sketch is proportional to

[0021] In the embodiments of the present disclosure, due to the trade-off of the inaccuracy, block size, total width and total depth of the candidate generation sketch, and total width and the total depth of the verification sketch, the throughput may be improved. In addition, it may ensure that the overall memory consumption has a logarithmic relationship with the length of the stream, may further reduce memory consumption, and may expand the application scenario of the stream processing method. Moreover, for large-capacity stream, it may further ensure the reliability and stability of stream processing, and may ensure the quality of service.

[0022] In one possible implementation, testing each sub-set of candidate keys by using one block of the verification sketch out of the k’ blocks of the verification sketch to obtain the verified current list of frequent keys, includes: outputting a candidate key only if the values of all counters, that correspond to this candidate key in the corresponding block of the verification sketch, is greater than or equal to the predefined threshold, each sub-set of candidate keys including a plurality of candidate keys.

[0023] In a second aspect, a computing device is provided. The computing device includes a memory and at least one processor connected to the memory. The memory is configured to store computer instructions. The computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a candidate generation sketch, the candidate generation sketch including k blocks, k being an integer greater than or equal to 1 ; obtaining a verificationsketch, the verification sketch including k' blocks, k' being an integer greater than or equal to 1 ; receiving a plurality of records in a insertion stream and / or retraction stream, each of the records including a key and a count that are in correspondence; updating the sketches by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k' blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch; querying the k blocks of the candidate generation sketch-to determine a set of candidate keys; dividing the set of candidate keys into k' sub-sets of candidate keys, the k' sub-sets of candidate keys being in one- to-one correspondence with the k1blocks of the verification sketch; testing each sub-set of candidate keys by using one block of the verification sketch of the k' blocks of the verification sketches to obtain a verified current list of frequent keys; and outputting the current list of frequent keys in the stream with insertions and / or retractions.

[0024] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c determining an r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on a predefined first function, wherein r G {1, ... , k}; transferring record (s, c) to the target block of the candidate generation sketch; determining an r'-th block of the verification sketch as the target block of the verification sketch based on a predefined second function, wherein r' G {1, ... , k'}; transferring record (s, c) to the target block of the verification sketch; updating majority testers corresponding to the key s in the target block of the candidate generation sketch; and updating the counters corresponding to the key s in the target block of the verification sketch.

[0025] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining that for a record (s, c) with key s and count c, r = h0(s) based on a hash function hQ\ U -> {1, ... , k }, the r-th block of the candidate generation sketch being the target block of the candidate generation sketch; and determining that r' = h'0(s) based on a hash function h'o: U -» {1, ... , k'}, the r'-th block of the verification sketch being the target block of the verification sketch.

[0026] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c, obtaining an array of majority testers Mijr, i = 1, ..., d, j = 1, ... , w / k corresponding to the target block of the candidate generation sketch r G {1, ... , k); and updating the majority testers by sending d copies of the corresponding record to the array of majority testers MiZl.(S)rselected via hash functions h^ U {1, ... , w / k}, i = 1, ... , d.

[0027] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c, obtaining an array of counters C[jr, i = 1, j = 1, ... , w' / k' corresponding to the target block of the verification sketch r' 6 {1, , / c'}; and updating the counters Cih^rlselected via hash functions h'i'. U -> {1, ... , w' / k'}, i = 1, ... , d' , by incrementing them by count c.

[0028] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a set of parameters; determining a total width and a total depth of the candidate generation sketch, and a number k of blocks of the candidate generation sketch based on the set of parameters; and determining a total width and a total depth of the verification sketch, and a number k' of blocks of the verification sketch based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

[0029] In one possible implementation, the set of parameters includes at least one or more of: a maximum length of a key, a minimum frequency that will be of interest during run of a streaming program, an inaccuracy, a failure probability, or a block size; the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during run of the streaming program, the failure probability, and the block size; determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability; and determining the total width of the candidate generation sketch based on the total depth of the candidate generation sketch, and the correspondence relationship between the total width and the total depth of the candidate generation sketch.

[0030] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: w is proportional towherein p is the maximum length of the key, O is the minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

[0031] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: d is proportional to ; wherein is the minimum frequency that will be of interest during run of the streaming program, q is the failure probability, and d is the total depth of the candidate generation sketch.

[0032] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the verification sketch based on the inaccuracy, the failure probability, and the block size; determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch; and determining the total width of the verification sketch based on the total depth of the verification sketch and the correspondence relationship between the total width and the total depth of the verification sketch.

[0033] In one possible implementation, the computer instructions, when executed by the at leastwherein s is the inaccuracy, Q is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0034] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: d' is proportional to l°g~; wherein Q is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

[0035] In one possible implementation, each block of the candidate generation sketch has a width of w / k and a depth of d, wherein w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch; each block of the verification sketch has a width of w' / k' and a depth of d' , wherein w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0036] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: the number k of blocks of the candidate generation sketch is proportional towherein p is the maximum length of the key, and B is theblock size; and the number k' of blocks of the verification sketch is proportional to

[0037] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: outputting a candidate key only if the values of all counters, that correspond to this candidate key in the corresponding block of the verification sketch, is greater than or equal to the predefined threshold, each sub-set of candidate keys including a plurality of candidate keys.

[0038] In a third aspect, a computer program product including instructions is provided. The instructions, when executed by a computer, cause the computer to perform the stream processing method in the first aspect or in any of the possible implementations in the first aspect.

[0039] In a fourth aspect, a computer-readable storage medium is provided, which has stored a computer program or instructions that, when executed by a computer, cause the computer to perform the stream processing method in the first aspect or in any of the possible implementations in the first aspect.

[0040] In a fifth aspect, a computing device cluster is provided, which includes at least one computing device each including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, to cause the computing device cluster to perform the stream processing method in the first aspect or in any of the possible implementations in the first aspect.

[0041] The technical effects of any design method from the second to the fifth aspects can be referred to the first aspect or the different designs of the first aspect, which will not be detailed here.

[0042] On the basis of the implementations provided in the above aspects, the present disclosure is able to further provide more implementations by further combination.BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG. 1 is a flow diagram of a stream processing method, in accordance with embodiments of the present disclosure;

[0044] FIG. 2 is a flow diagram of another stream processing method, in accordance with embodiments of the present disclosure;

[0045] FIG. 3 is a flow diagram of a process of obtaining a candidate generation method and a verification method, in accordance with embodiments of the present disclosure;

[0046] FIG. 4 is a schematic diagram of a process of transferring a record during update, in accordance with embodiments of the present disclosure;

[0047] FIG. 5 is a schematic diagram of a further process of transferring a record during update, in accordance with embodiments of the present disclosure;

[0048] FIG. 6 is a schematic diagram showing a structure of a computing device, in accordance with embodiments of the present disclosure; and

[0049] FIG. 7 is a schematic diagram showing a structure of a computing device cluster, in accordance with embodiments of the present disclosure.DETAILED DESCRIPTION

[0050] In order to facilitate understanding, the main terms involved in embodiments of the present disclosure are explained first.

[0051] Key, which is an identifier with a certain length that the tracked item has, such as IP address, user ID, post ID, or connection ID. For example, the IPv6 address is an identifier with a length of 128 bits.

[0052] A universal family H of functions is a set of functions h: U -» P, and if a randomfunction from H, for any two keys x =£ y G U, is chosen, it holds that Pr[h(x) = h(y)] < — . Thatis to say, distinct keys from U collide under a random function h G H with probability at most 1 / | F|. Several simple constructions of effectively computable universal hash functions are known and presented. A randomly chosen functions from a universal family are referred to as the random hash function.

[0053] Herein, it is assumed that log is base-2 logarithm, and In is base-e logarithm. That is, log 2 = 1 and In e = 1. [x] and [x] represent the number x rounded down or rounded up to the nearest integer, respectively.

[0054] Timely reacting to network traffic anomalies (like DoS attacks), tracking most active network participants, most active network streams, popular topics or posts in social networks, popular orders in online shops, frequent values of IPv6 addresses in data bases for query optimization - all of these boil down to the task of tracking frequent items in a data stream. In general, the count of the item may increase (when a tracked event occurs) or decrease. The identifiers, with a specific length, of the tracked frequent items are referred to as frequent keys, "tracking frequent items" can be understood as "tracking frequent keys".

[0055] The task is tracking frequent keys in a stream of key insertion and retraction events. Usually, the set U of possible keys, also called universe, is assumed to be the set of all p-bit words for some natural number p, or equivalently, the set of all integers in the range U = {0, ... , 2P— 1}. For tracking frequent keys in a stream, a data structure supporting initialization, update and query is required.

[0056] Initialization refers to taking a key with length p and initializing a count fs= 0 for each key s G U = {0, ... , 2P— 1}.

[0057] Update refers to taking a key s G U and an integer c (possibly negative) and incrementing the count fsof 5 by c.

[0058] It should be noted that negative values c represent a decrement (retraction). It is assumed that updates maintain the invariant fs> 0. The total count of all keys is represented by n (which may increase or decrease after an update operation).

[0059] Query refers to taking a number <p E [0,1], and outputting the keys s E U with fs> (pn, that is, with frequency > q>.

[0060] Methods of tracking frequent items include exact methods and approximate methods corresponding to approximate data structures.

[0061] Exact methods of tracking keys in a stream with backtracking (retraction) require at least n or at least | U | bits of memory. In the context of analyzing big data, this is infeasible, because both the universe U and the count n ofevents may be large. For example, when network packets arriving from various IPv6 addresses are counted, | U\ = 2128, and millions of packets may arrive per second. This data structure will quickly exhaust main memory, and auxiliary memory may not be available or too slow for keeping up with rapid updates and queries. For example, embedded devices or programmable switches, which may be attractive for network traffic monitoring, usually do not have auxiliary memory. In contrast, full-fledged stream processing engines like Apache Flink allow for the use of auxiliary memory ("state"), but when a computational node of a cluster fails, snapshot has to be frequently performed on data content in order to migrated or restored.

[0062] Methods of tracking frequent items include an exact method and an approximate method corresponding to an approximate randomized data structure. The exact method of tracking keys in a stream with backtracking (retraction) requires memory proportional to the length of the stream, which is not feasible for analyzing big data streams.

[0063] Approximate methods include using approximate randomized data structures, which sacrifices the accuracy of the answers in a controlled manner in order to be able to work while consuming a limited amount of memory, e.g. few hundred megabytes, regardless of the universe size and stream length.

[0064] Most known existing approximate randomized data structure for key frequency estimation do not support retraction, or only support limited retraction, or only support a given key frequency query, but cannot output a list of the most frequent items in the presence of retraction.

[0065] Still, there are several approximate randomized data structures for tracking frequent keys that do support retractions. These data structures include non-adaptive group testing (NAGT), dyadic count-min sketch (DCM) and dynamic dyadic count-Min sketch (DDCM).

[0066] NAGT, DCM and DDCM data structures for each operation (update or query) perform a significant number of I / Os to random addresses given by a number of hash functions these data structures use, which does not allow to use caches effectively and negatively impacts throughput.[ 00671 On the other hand, there is an approximate randomized data structure called Buffered Count Min sketch which can be considered as cache-efficient. It is arranged in blocks, such that for each operation only access to one block is required. However, this data structure is only designed to output an approximate count of a given key, but not the list of all frequent keys.

[0068] Thus, there are the following problems. It was previously mathematically proven that any exact method for tracking frequent keys in streams with retractions would require the amount of memory proportional to the stream size, which is infeasible for analyzing high-volume data streams arriving at several GB / s.

[0069] Existing approximate methods for tracking keys in streams with retractions in the related art involve many memory accesses to random addresses, which makes it impossible to efficiently utilize memory caches, and thus negatively impacts throughput.

[0070] In addition, the existing the existing cache-efficient method only solves the count estimation problem, but cannot output the list of all frequent keys.

[0071] Embodiments of the present disclosure provide a stream processing method that allows to approximately output a list of the most frequent keys cache-efficiently, in feasible memory, with smooth user-definable trade-off between solution accuracy, memory usage, and fast cache memory size. In the stream processing method, a candidate generation sketch including k blocks and a verification sketch including k' blocks are obtained; each block of the candidate generation sketch consists of a fixed number of majority testers and each block of verification sketch consists of fixed number of counters, where the number of majority testers and counters depend only on user defined constant parameters but not on the stream size; during the update, a record in a received input stream (stream with insertions and / or retractions) is transferred to one target block of the candidate generation sketch and one target block of the verification sketch out of the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; during the query, each block of the candidate generation sketch is processed sequentially and outputs a set of candidate keys, further the total set of candidate keys is partitioned into k' subsets of candidate keys in one to one correspondence with k' blocks of the verification sketch, each subset of candidate keys is then sequentially tested against the corresponding block of the verification sketch, and the keys, that passed the verification sketches, are then output as the current list of frequent keys. Here, by making the data structure consisting of blocks and processing these blocks in an efficient way during updates and queries the cache-efficiency is achieved. By making the size of the sketches depend on the user selected constant parameters the memory consumption becomes feasible. Finally, since the user selected parameters reflect the relationship between the output accuracy, overall memory consumption, and used cache memory size, the smooth user-definable trade-off for the QoS is provided

[0072] Implementations of the stream processing method provided in the embodiments of the present disclosure will be described in details below with reference to the accompanying drawings.

[0073] FIG. l is a flow chart of the stream processing method, in accordance with embodiments of the present disclosure, and as shown in FIG. 1, the method includes the following steps.

[0074] In step 101, a candidate generation sketch is obtained, the candidate generation sketch including k blocks, k being an integer greater than or equal to 1.

[0075] In step 102, a verification sketch is obtained, the verification sketch including k' blocks, k' being an integer greater than or equal to 1.

[0076] In step 103, a plurality of records in a stream with insertions and / or retractions are received, each of the records including a key and a count that are in correspondence.

[0077] In step 104, the sketches are updated by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k1blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch.

[0078] During the update, the record in the received input stream is transferred to one target block of the candidate generation sketch and one target block of the verification sketch corresponding to the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is continuously loaded into faster memory and corresponding update processing is carried out in the memory. Therefore, one block may be put into faster memory, and during the update, only one block of the target candidate generation sketch and one block of the target verification sketch need to be accessed. In addition, although the entire data structure is stored in a slower, larger memory level, one required block may be fully loaded into faster memory during the update; and thus, the update may be executed in faster memory and takes less time.

[0079] In step 105, during the query, the k blocks of the candidate generation sketch are queried sequentially to determine a set of candidate keys.

[0080] In step 106, the set of candidate keys are divided into k' sub-sets of candidate keys, and the k' sub-sets of candidate keys are in one to one correspondence with the k' blocks of the verification sketch.

[0081] In step 107, each sub-set of candidate keys is tested by using one block of the verification sketch out of the krblocks of the verification sketch, so as to obtain a verified current list of frequent keys.

[0082] In step 108, the current list of frequent keys in the stream with retractions and / or insertions is output.

[0083] In the embodiments of the present disclosure, by obtaining the candidate generation sketch including k blocks-and the verification sketch including k' blocks, during the update, the record in the received input stream is transferred to one target block of the candidate generation sketch and one target block of the verification sketch out of the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is consequently loaded into faster memory and corresponding update processing is carried out in the memory. During the query each block of candidate sketch is processed sequentially and needs to be processed once, thereafter each block of candidate sketch needs to be processed at most once. Therefore, each time one block needs to be accessed it may be put into faster memory, where during the update only one block of the target candidate generation sketch and one block of the target verification sketch need to be accessed, while during the query each block of the candidate generation sketch should be accessed once and each block of the verification sketch should be accessed at most once. In addition, although the entire data structure is stored in a slower, larger memory level, one required block may be fully loaded into faster memory during the update or query; and thus, the update or query may be executed in faster memory and takes less time.

[0084] FIG. 2 is a flow diagram illustrating another stream processing method, in accordance with the embodiments of the present disclosure;

[0085] In step 201, a candidate generation sketch is obtained; the candidate generation sketch includes k blocks, each of which consists of a set of majority testers, and k is an integer greater than or equal to 1.

[0086] In the embodiments of the present disclosure, each block of the candidate generation sketch has a width of w / k and a depth of d; w is the total width of the candidate generation sketch; and d is the total depth of the candidate generation sketch.

[0087] In step 202, a verification sketch is obtained; the verification sketch includes k’ blocks, each of which consists of a set of counters, and k’ is an integer greater than or equal to 1.

[0088] In the embodiments of the present disclosure, each block of the verification sketch has a width of w' / k' and a depth of d', w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0089] In the embodiments of the present disclosure, a data structure composed of the candidate generation sketch and the verification sketch may be referred to as blocked NAGT+.

[0090] FIG.3 is a flow diagram illustrating a process of obtaining the candidate generation sketch and the verification sketch provided in the embodiments of the present disclosure. As shownin FIG. 3, the process of obtaining the candidate generation sketch and the verification sketch includes the following sub-steps SI to S3.

[0091] In sub-step S 1 : a set of parameters is obtained.

[0092] The set of parameters includes at least one or more of: a maximum length of a key, an inaccuracy, a failure probability, or a block size, a minimum frequency that will be of interest during run of a streaming program. In addition, the set of parameters may further include an upper bound on a frequency of most frequent key, otherwise it will default to 1.In the embodiments of the present disclosure, the above process of obtaining the set of parameters may include the following content. In the present disclosure, an additional operator may be used to realize the stream processing method. The additional operator is a usual building block of programs in the stream processing, and is used in cloud-native big data analysis applications. A program in the stream processing consists of data streams, which are ordered sequences of data units (tuples, events, and / or records), and transformations, performed by the operators. An operator takes on or more streams as input, applies specific transformations or functions (e.g., filtering, aggregation, etc.) to each incoming data unit, produces one or more output streams, and can communicate with an internal memory called the operator's state.

[0093] The operators together with their input and output streams form a directed acyclic graph called a logical query plan of a program. The logical query plan is further optimized by the stream processing system, then transformed into a physical query plan by assigning the operators with their states to computational nodes on a cluster; next, the program is executed.

[0094] Many standard transformations performed by operators in stream processing are usually provided as built-in primitives or additional library modules. In the embodiments of the present disclosure, an additional library module provides a user interface (UI) for setting up the operator and tuning the operator's set of parameters and a new operator itself, and the new operator can: receive a record (which may be, for example, a key insertion, a key retraction or a query with parameter (p for outputting all keys with frequency at most <p) from the input stream; update the data structure in its state in case of key insertion and key retraction events obtain an approximate set of required frequent keys from the data structure and submit it to the output stream.

[0095] In the embodiments of the present disclosure, handling key retractions in the stream is important for a data structure to work correctly in a larger number of scenarios. Retractions in a stream naturally arise the following three scenarios: one is when changing a source data base for the stream; one is when a job in a stream processing recovers from a failure and all calculations for events processed since the last snapshot is supposed to be removed; and one is when a key in the stream serves as a result of nested queries.

[0096] Consider the following example of a nested query. When analyzing for an online store a stream of interactions with items, among the items that have been sold between one and three times, the top- 10 frequently viewed items need to be determined. Assuming that the stream consists of records in the form (s, c), the key s is the ID of the viewed item; and c = 1 if it was sold, and 0 otherwise. The stream processing program may consist of a pair of consecutive operators.

[0097] A first operator obtains the input stream. For an item with an ID of s, the total purchase count Psincrements by c, and the total purchase count Psis queried; if 1 < Ps< 3, the record (s, c) is passed to a second operator; the output stream of the first operator will be the input stream for the second operator; and the second operator increments by 1 the total view count Vsfor the item with ID of s, and outputs the top- 10 frequently viewed items from the second operator's database.

[0098] If a frequently viewed item that has already been purchased three times is purchased again, the second operator should stop determining the frequently viewed item as a candidate for output. Thus, the second operator has to support record retraction.

[0099] In the embodiments of the present disclosure, a proposed operator may be set (e.g., by the user) by selecting a set of parameters that includes the maximum length of a key, the upper bound (Pi on the frequency of the most frequent key, the minimum frequency that will be of interest during the run of the streaming program, inaccuracy, failure probability and block size. Here, the maximum length of the key refers to the maximum length of the key expected to occur in the data stream; the upper bound on the frequency of the most frequent key can be the upper bound (by default <Pi = 1) that can be expected and driven by some prior knowledge that the user may have about the data in the stream (smaller values of (primprove the tradeoff between resource consumption and accuracy); the minimum frequencythat will be of interest during the run of the streaming program satisfies6 [0,1]; the inaccuracy £ satisfies £ G (0,1]; the failure probability p satisfies p G (0,1]; and the block size B is in bytes.

[0100] Optionally, through the following sub-steps Al to A6, tuning these parameters in the graphical user interface (GUI) will be prompted (e.g., to the user).

[0101] In sub-step Al, the maximum length p of the key expected to occur in the stream and the upper bound on the expected frequency of the most frequent key are defined.

[0102] In sub-step A2, the desired inaccuracy £ G (0,1], failure probability p G (0,1] and the minimum frequency <t> G [0,1] that will be of interest are selected.

[0103] In sub-step A3, the UI calculates and provides a lower bound for a possible block size B in bytes, that is, the block size.

[0104] In the embodiments of the present disclosure, the block size B satisfies:B > 2 • T • max{dp, d'} (1)

[0105] Here, d is the total depth of the candidate generation sketch, p is the maximum length of the key, d' is the total depth of the verification sketch, and T is a size of bytes in the preset counter.

[0106] Due to the block size satisfying the above formula (1), it may be possible to avoid a case that if the size B of one block in bytes such that the width of one candidate generation sub-sketch or in other words the width of one block of the candidate generation sketch equal to 1 , then all of d majority testers of the candidate generation sub-sketch will become identical and d tries will bring no gain in improving the probability of having small inaccuracy. In other words, it may be possible to avoid a case that the performance of the blocked candidate generation sketch with blocks of width1 would degrade down to the performance of the candidate generation sketch with a single block of width w and depth 1 (and thus, the failure probability is 1 / 2), while occupying d times more space. The dame holds for the verification sketch with block width w'B= 1. If the counters of the size of 8 bytes are used in all data structures, then due to the above formula (1) for the block size, it may be possible to guarantee the width of all sub-sketches to be at least 2, to improve the stability and reliability of the stream processing, and in turn avoid the case of occupying multiple space at the same time.

[0107] The counter can be of a size of 8 bytes or other bytes, which can be set according to the practical scenarios, and details will not be limited in the embodiments of the present disclosure.

[0108] In sub-step A4, the block size is selected according to the lower bound for the provided possible block size B and the desired amount of memory that the system is allowed to occupy in the cache, or according to the lower bound for the possible block size B and a selected faster memory level.

[0109] In the embodiments of the present disclosure, if the provided bound is unacceptable (for the user), then it (the user) can either choose not to use the block structure and occupy only the slower memory level, or to choose a larger failure probability.

[0110] For example, in the embodiments of the present disclosure, the following memory levels listed from the fastest and smallest to the slowest and largest may be considered: LI cache (usually of the size from 2 KB to 64 KB or 128 KB), L2 cache (usually of the size 512 KB), RAM, external memory, and network-attached storage. The following standard scenarios for locating the entire data structure (composed of the candidate generation sketch and the verification sketch) in a "slower memory” and being able to load one of blocks corresponding to the candidate generation sub-sketch to a "faster memory” can be considered.

[0111] If the candidate generation sketch and the verification sketch are stored in RAM, LI or L2 cache can be considered as faster memory levels for one block to be loaded to. In this case, the size of the block needs to match the allowable (user-defined) amount of space in LI or L2 cache (e.g.,2 KB, 64 KB, or 128 KB).

[0112] If the data structure is stored in the external memory (HDD or SSD), RAM can be considered as a faster memory level for one block to be loaded to. In this case, it is desired for one block to match one unit of access (generally 4 KB) of the external memory, so that it can be loaded to RAM in one I / O.

[0113] If the candidate generation sketch and the verification sketch are stored in a distributed manner on multiple nodes in the network, the external memory of the node performing computations can be regarded as a faster memory for loading one block.

[0114] In sub-step A5, the UI provides the total amount M of memory in bytes, which the operator will need as its state operator with the parameters selected above in a slower memory level (RAM).

[0115] In sub-step A6, after the configuration is complete, the operator is added to a logical query plan of a user program.

[0116] In sub-step S2, a total width of the candidate generation sketch, a total depth of the candidate generation sketch, and a number of blocks of the candidate generation sketch are determined based on the set of parameters, to obtain the candidate generation sketch.

[0117] In the embodiments of the present disclosure, the number of blocks of the candidate generation sketch is determined based on the set of parameters, and the number k of blocks of the candidate generation sketch is proportional towhere p is the maximum length of the key, andB is the block size.

[0118] For example, the number k of blocks of the candidate generation sketch may satisfyZc=|8p ~J; here, p is the maximum length of the key, B is the block size; and the embodiments B of the present disclosure do not limit the specific constant, which can be 8 in this formula or other numerical values, which can be set according to the practical scenarios.

[0119] In the embodiments of the present disclosure, in a case where k' = k = 1 the width w and depth d of the candidate generation sketch and width w' and depth d' of the verification sketch can be set due to the following arguments. In order to ensure that a frequent key is missed in the set of candidate keys (and hence not output) with probability at most Q, it is sufficient to choose 2 1 2 w > — <t> and > log °— g<t>. The size of the set of candidate keyJs is at most dw; therefore, when w' > - e and d' — 2d + 1 are chosen, the probability of outputting the key with frequency less than <p — £ is at most:

[0121] Hereis the minimum frequency that will be of interest during the run of the streaming program, g is the failure probability, £ is the inaccuracy.

[0122] In the embodiments of the present disclosure, in a case of k > 2 , determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during the run of the streaming program, the failure probability, and the block size, includes: w is proportional twhere p is the maximum length of the key,is the minimum frequency that will be of interest during the run of the streaming program, Q is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

[0123] For example, in the embodiments of the present disclosure, the correspondence relationship between the total width and the total depth of the candidate generation sketch in a case of k > 2 is determined based on the maximum length of the key, the upper bound on the frequency of most frequent key, the minimum frequency that will be of interest during the run of the streaming program, the inaccuracy, the failure probability, and the block size.

[0124] The corresponding relationship between the total width of the candidate generation sketch and the total depth of the candidate generation sketch in a case of k > 2, includes:where p is the maximum length of the key, (pris the upper bound on the frequency of most frequent key, <t> is the minimum frequency that will be of interest during the run of the streaming program, E is the inaccuracy, Q is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

[0125] In the embodiments of the present disclosure, the process of determining the correspondence between the total width of the candidate generation sketch and the total depth of the candidate generation sketch in case of k > 2, may include: the following content.

[0126] In order to estimate the overall accuracy degradation of a block sketch, the number nrof keys that may be assigned to one block is first estimated via a random hash function h0. The key distribution process may be modeled as a weighted-balls-into-bins problem: N different balls (keys) with weights (frequencies) (pt 6 (0,1],= 1 are uniformly randomly distributed into k bins (blocks), to estimate the number nrof keys that may be assigned to one block via the random hash function h0, and in turn estimate the overall accuracy degradation of the block sketch.

[0127] For the model of weighted-balls-into-bins, for any y > 0, the load of one bin exceeds with probability at most exp{— y2 / 2k^1], where <p±< 1 is the weight of the heaviest ball (i.e.,frequency of the most frequent key) in the embodiments of the present disclosure.

[0128] By plugging y it may be guaranteed that the number nrofkeys in one block exceeds with probability no more than p / 2, resulting in upper boundof the overall inaccuracy of the blocked stream processing method of the embodiments of the present disclosure, that is, upper bound of the overall inaccuracy for the blocked NAGT+.

[0129] The overall inaccuracy for a sketch usually depends on the total number n of keys inserted into the sketch and the width w of the sketch. In the embodiments of the present disclosure, the width is selected to guarantee a small inaccuracy s; for example, for the case of k = k' = 1, it is sufficient to set w' = 2 / s for the width of the verification sketch to guarantee that only keys with count > 4>n — sn will pass the verification sketch, and it is sufficient to set w = 2 / for the width of the candidate generation sketch to guarantee that keys with sufficient frequency exist in the set of candidate keys. But in the case when a sketch is divided into k 6 [2, j] blocks of width w / k, the inaccuracy of each block becomes k times higher. For example, in this case, a block of the verification sketch may then overestimate frequencies of elements by an additional factor of k, while a block of the candidate generation sketch may miss keys with frequency less than 0 / c instead of 0 in the set of candidate keys. On the other hand, each block r has a smaller number nrof keys. However, if only one hash function assigns keys to blocks, then it cannot guarantee an even load for blocks with high probability.

[0130] The probability of missing a frequent key from the output by a candidate generation sketch in case of k > 2 may be determined first. A key s with count <pn > <t»n is sent to a block with candidate sketch r = h0(s) of width wB= w / k and depth d, and the number of keys in block r is set as nr.

[0131] In one aspect, if then r-th block of the candidate sketchmisses a frequent key s with count> <Pn = (— ) • nrwith probability at most 2d; if the width \nr / of the sub-sketch is then the above first inequalityholds; when w satisfies2 , according to the number k of blocks of the candidate generation sketch satisfiesk = |8p as well as < 1 , the above inequality is solved with respect w to obtain the corresponding relationship between the total width and total depth of the candidate generation sketch that includes: w is proportional t

[0132] For example, w

[0133] When 1 > <p1> O, it is determined that (since big-0 hides constants anyway):

[0134] according to the first terms of Taylor series for Vl + x.

[0135] To sum up, if w is selected according to the corresponding relationship between the total width and total depth of the candidate generation sketch, the probability of missing one key s with frequency (fm > n is at most 2d.

[0136] Further, the total depth of the candidate generation sketch may be determined based on the minimum frequency that will be of interest during the run of the streaming program and the failure probability.

[0137] In the embodiments of the present disclosure, d is proportional to log— .

[0138] Here, 0 is the minimum frequency that will be of interest during the run of the streaming program, g is the failure probability, and d is the total depth of the candidate generation sketch.

[0139] Optionally, the probability of missing at least one frequent key of all the frequent keys may be at most 2d / according to the union bound; and due to as formula (3), it maybe guaranteed that the probability of missing at least one frequent key of all the frequent keys is at most Q / 2.

[0140] In another aspect, the probability of is also at most Q / 2.Therefore, the total probability of missing a frequent key in the set of candidate keys is at most Q.

[0141] Further, the total width of the candidate generation sketch may be determined based on the total depth of the candidate generation sketch and the correspondence between the total width of the candidate generation sketch and the total depth of the candidate generation sketch.

[0142] In sub-step S3, a total width of the verification sketch, a total depth of the verification sketch, and a number of blocks k', for the case of k' > 2, of the verification sketch are determined based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

[0143] In the embodiments of the present disclosure, in case of k' > 2 a correspondence relationship between the total width of the verification sketch and the total depth of the verification sketch is determined based on the inaccuracy, the failure probability and the block size.

[0144] In the embodiments of the present disclosure, the number k' of blocks of the verification sketch is proportional to

[0145] For example, the number ofblocks ofthe verification sketch may satisfies k' =and the embodiments of the present disclosure do not limit the specific constant, which can be 8 in this formula or other numerical values, which can be set according to the practical scenarios.

[0146] The correspondence relationship between the total width of the verification sketch and the total depth of the verification sketch in case of k' > 2 includes: w' is proportional twhere £ is the inaccuracy, Q is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0147] Optionally, in the embodiments of the present disclosure, the correspondence relationship between the total width of the verification sketch and the total depth of the verification sketch is determined based on the maximum length of the key, the upper bound on the frequency of most frequent key, the minimum frequency that will be of interest during the run of the streaming program, the inaccuracy, the failure probability, and the block size.

[0148] The correspondence relationship between the total width of the verification sketch and the total depth of the verification sketch in case of k' > 2 includes: . _ _ _ ■, -1 , 2 lie d'ipi , 2 116 d'cpi , 2 _ \ . . ln- + a - - — In- - - — ln- + 2e (4)Q NBQ NBQ J where p is the maximum length of the key,is the upper bound on the frequency of most frequent key,is the minimum frequency that will be of interest during the run of the streaming program, £ is the inaccuracy, g is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0149] Considering that any key s with count (pn < <t>n — en, it will be verified using the counters of a block with verification sub-sketch number rof width -7 and depth d'. nris set to be the number of keys tracked by the counters in block r (i.e., r -th block).

[0150] When nr< -^ ( 1 + 2<p1k' In- ) , if the width w' / k' of the verification sketch satisfies: , then the key s with count dm < 4>n — sn =nr•w('<t>in the r -th block of the verification sketch erroneously passes verification with nrprobability at most 2d>.

[0151] The above inequality on w' holds, if E > +

[0152] since k' > Bw'd' IB .The above inequality is solved with respect w' to obtain the corresponding relationship between the total width and total depth of the verification sketch: w' is proportional t

[0153] For example, the correspondence relationship between the total width of the verification sketch and the total depth of the verification sketch may include:

[0154] That is, O (since big-0 hides constants anyway):

[0155] according to the first terms of Taylor series Vl + x and |x| <1, where 1 > <p1> <t> > £.

[0156] To sum up, w' is selected according to the correspondence relationship of the total width of the verification sketch and the total depth d' of the verification sketch, the probability that the key s with frequency less than (p — £ erroneously passes verification is at most 2d' .

[0157] Further, the total depth of the verification sketch may be determined based on the failure probability, the total width of the candidate generation sketch and the total depth of the candidate generation sketch.

[0158] In the embodiments of the present disclosure, d' is proportional to loghere, Q is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

[0159] Optionally, in the embodiments of the present disclosure, based on the above correspondence relationship, determining the total depth of the verification sketch based on the failure probability, the total width of the candidate generation sketch and the total depth of the candidate generation sketch may include:where g is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

[0160] Since d' > the probability of outputting any key with frequency less than (p —s may be guaranteed to be at most Q / 2.I I 2 i

[0161] In another aspect, the probability that nr> - 1 1 + l2<p1k In - 1 is also at most p / 2.\ \ Q / Therefore, the total probability of outputting any key with frequency less than (p — s is at most Q.

[0162] Further, the total width of the verification sketch may be determined based on the total depth of the verification sketch as well as the correspondence relationship between the total width and total depth of the verification sketch.

[0163] In step 203, a plurality of records in an insertion stream and / or retraction stream are received, and each of the records includes a key and a count that are in correspondence.

[0164] In the embodiments of the present disclosure, a plurality of records (s, c) with keys s and counts c in the insertion stream and / or retraction stream may be received, and the counts may be negative.

[0165] In step 204, an r-th block of the candidate generation sketch is determined as the target block of the candidate generation sketch based on a predefined first function, where r G {1, ... , k}.

[0166] In the embodiments of the present disclosure, it is determined that for a record (s, c) with key s and count c, r = h0(s) based on a hash function h0: U -» {1, ... , k }, the r-th block of the candidate generation sketch being the target block of the candidate generation sketch.

[0167] In step 205, each record is transferred to the target block of the candidate generation sketch.

[0168] After the target block of the candidate generation sketch is determined, the record can be forwarded to the target block of the candidate generation sketch.

[0169] In step 206, an r'-th block of the verification sketch is determined as the target block of the verification sketch based on a predefined second function, where r' E {1, ... , k'}.

[0170] In the embodiments of the present disclosure, it is determined that for a record (s, c) with key s and count c, r' = h'0(s) based on a hash function h'0: U -> {1, .... k'}, the r'-th block of the verification sketch being the target block of the verification sketch.

[0171] In step 207, each record is transferred to the target block of the verification sketch.

[0172] In the embodiments of the present disclosure, after the target block of the verification sketch is determined, the record can be forwarded to the target block of the verification sketch.

[0173] FIG. 4 is a schematic diagram of a record transmitted during the update, in accordance with embodiments of the present disclosure. As shown in FIG. 4, during the update, the record (s, c) with key s and count c in the received input stream is transferred to one target block of the candidate generation sketch r = h0(s) and one target block of the verification sketch r' = h’^s) corresponding to the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is continuously loaded into faster memory and corresponding update processing is carried out in the faster memory.

[0174] In step 208, majority testers corresponding to the record’s key are updated in the target block of the candidate generation sketch.

[0175] FIG. 5 is a schematic diagram of a record further transmitted during the update, in accordance with embodiments of the present disclosure. As shown in FIG. 5, in the target block of the candidate generation sketch, an array of majority testers Mijr, i =j = l, ... , w / k corresponding to the target block r 6 {1, ... , k] of the candidate generation sketch are obtained, and then the majority testers are updated by sending d copies of the corresponding record (s, c) with key s and count c to the array of majority testers Mih.^s)rselected via hash functions h^ U -> {1, ... , w / k), i = 1, ... , d. Each of majority testers is used to provide candidate keys during the later query. Compared with the existing NAGT, in the embodiments of the present disclosure, the majority testers are not used for verification, so that the number of the majority testers needed in the stream processing is smaller, and the 1 smaller number of the majority testers makes the memory consumption of storing the majority testers less, thereby reducing the memory consumption of stream processing, and improving the processing efficiency. Moreover, less majority testers lead to a smaller set of candidate keys they generate during query, which in turn leads to less time for verifying those keys, and hence improves processing efficiency.

[0176] In the embodiments, the array of majority testers supports the following four operations. The first one is initialization, which includes taking a key length p, and initializing a count fs= 0 for each key s E U = {0, ... , 2P— 1}. The second one is update, which includes taking a key s E U and an integer c (possibly negative) and incrementing the count fsof 5 by c. The negative value c represents a decrement. It is assumed that the update maintains the invariant fs> 0. The total count of all keys is represented by n (which may increase or decrease after the update). The third one is total count query, which includes returning the total count n. The fourth one is majority query, which includes returning s if there is a key s E U such that fs> p otherwise returning an arbitrary key.

[0177] In step 209, the counters corresponding to the record’s key are updated in the target block of the verification sketch.

[0178] In the embodiments of the present disclosure, referring to FIG. 5, an array of counters Cijr, i = 1, ... , d' , j = 1, ... , w' / k' corresponding to the target block of the verification sketch r' Gis obtained; and for a record (s, c) with key s and count c, updating the counters selected via hash functions h'l'. U -» {1, ... , w' / k'}, i — 1, ... , d' , by incrementing them by count c.

[0179] In step 210, the k blocks of the candidate generation sketch are queried to determine a set of candidate keys.

[0180] In step 211, the set of candidate keys are divided into k' sub-sets of candidate keys, and the k' sub-sets of candidate keys are in one to one correspondence with the k' blocks of the verification sketch.

[0181] In the embodiments of the present disclosure, since the candidate generation sketch and the verification sketch are processed consecutively, blocks of the two sketches are always aligned during the query. The k blocks of the candidate generation sketch may be consecutively queried one by one to determine the set of candidate keys, and then the set of candidate keys may be divided into k' sub-sets of candidate keys= { s G S | ho(s) = i ], i = 1, ... , k'.

[0182] In step 212, each sub-set of candidate keys is tested by using one block of the verification sketch out of the k' blocks of the verification sketch, so as to obtain the verified current list of frequent keys.

[0183] In the embodiments of the present disclosure, a candidate key is output in a case where the values of all counters, that correspond to this candidate key, in the corresponding block of the verification sketch is greater than or equal to the predefined threshold.

[0184] Each sub-set of candidate keysincludes a plurality of candidate keys all of which should be tested through one corresponding block i of the verification sketch, i = 1, ... , k'.

[0185] In the embodiments of the present disclosure, the preset threshold is threshold is set to be equal to <pn, where (p > <t> is a frequency defined by a user, if he wants to track keys that has frequency at least (p.

[0186] Whenever at least one of the counters, that correspond to a candidate key in a block of the verification sketch is below the present threshold, the candidate key must be skipped and determined to be an infrequent key.

[0187] In case k' = k = 1, the total number of counters used in the stream processing method is:

[0188] The number of counters is constant. That is, the number of counters only depends on the constant parameters corresponding to the parameter set selected by the user, but not on the total number n of all keys in the stream, or the stream length. In turn, in a stream of n keys, each such counter can occupy not more than O(logn) bits. Therefore, the overall memory consumption is at most logarithmic in the stream length, which makes memory consumption feasible for large stream processing.

[0189] However, in practical applications, it is sufficient to use counters of constant with a size of 8 bytes for the most of practical cases, even if the stream is such that there are 230keys (230~ 109) arriving each second, 8-byte counters are also enough to run for 234seconds, which is 512 years.

[0190] In case k' = k = 1, when O(p) time for hashing keys of length p is calculated, the update time is:

[0192] Therefore, in case of k' = k = 1, d > log and d' > which both affect theupdate time and the query time are corresponding relationships related to . In other words, compared with the update time and query time of the existing NAGT, this application replaces - in the existing scheme by which is better than the existing NAGT by orders of magnitude.

[0193] In the stream processing method provided in the present disclosure, the total number of counters used in the block with k > 2 and k' > 2 is (wdp + w'd') , w, d, w', d' are defined according to the formulas (2) - (5), which can be asymptotically estimated as: Since big-0 hides constants, please change it to

[0195] Where in the above formula, B > 16 p log^ by (1) and (pr< 1.

[0196] It should be noted that, although the number of counters in the case of k > 2, k' > 2 is larger than the number of counters in the case of k' = k = 1, it is still a constant. That is, the number of counters only depends on the constant parameters corresponding to the parameter setselected by the user, but not on the total number n of all keys in the stream, or the stream length. In addition, since in a stream of n keys, each such counter will occupy not more than O(logn) bits, the overall memory consumption is at most logarithmical in the length of the stream, which makes memory consumption feasible for large stream processing.

[0197] When O(p) time for hashing keys of length p is calculated, in the case of k, k' > 2, the update time is:

[0198] The query time, in the case of k, k’ > 2, is:

[0199] Where in the above formula,

[0200] In summary, it is ensured that the block size is determined by the block size B, and a corresponding block of the candidate generation sketch and / or verification sketch can be put into faster memory, in which the number of counters, update time and query time are all reduced.

[0201] Due to the trade-off of the inaccuracy, block size, total width and total depth of the candidate generation sketch, and total width and the total depth of the verification sketch, the throughput may be improved, and the quality of service may be ensured.

[0202] In step 213, the current list of frequent keys in the stream with insertions and / or retractions is output.

[0203] In the stream processing method provided in the embodiments of the present disclosure, by obtaining the candidate generation sketch including k blocks and the verification sketch including k' blocks, during the update, the record (s, c) with key s and count c in the received input stream is transferred to one target block of the candidate generation sketch r = hQ(s) and one target block of the verification sketch r' = h'0(s) out of the k blocks of the candidate generation sketch and the k' blocks of the verification sketch, respectively; and thus, the update affects only two blocks, each of which is consecutively loaded into faster memory and corresponding update processing is carried out in the faster memory. Although the entire data structure is stored in a slower, larger memory level, one required block may be fully loaded into faster memory during the update or query; and thus, the update or query may be executed in faster memory and takes less time. During the query each block of candidate sketch is processed sequentially and needs to be processed once, thereafter each block of candidate sketch needs to be processed at most once. Thus, queries as well as updates benefit from loading one required block at a time into faster memory where the execution takes less time. Due to the trade-off of the inaccuracy, block size, total width and total depth of thecandidate generation sketch, and total width and the total depth of the verification sketch, the throughput may be improved and the quality of service may be ensured. Furthermore, due to the improvements of the trade-off, update and query, there is a logarithmic relationship between the overall memory consumption and the stream length, which make the memory consumption feasible for large stream processing.

[0204] FIG. 6 is a schematic diagram showing a structure of a computing device, in accordance with embodiments of the present disclosure. The computing device may be an electronic device with computing power, such as a server, a supercomputer, a mainframe computer, a network device, a terminal device, a personal computer (PC), or a chip or system-on-chip (SoC) in the above-mentioned device. As shown in FIG. 6, the computing device 300 includes at least one processor 302 connected to the memory 301

[0205] The memory 301 is configured to store computer instructions. The computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a candidate generation sketch, the candidate generation sketch including k blocks, k being an integer greater than or equal to 1 ; obtaining a verification sketch, the verification sketch including k' blocks, k' being an integer greater than or equal to 1 ; receiving a plurality of records in stream with insertions and / or retractions, each of the records including a key and a count that are in correspondence; updating the sketches by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k' blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch; querying the k blocks of the candidate generation sketch to determine a set of candidate keys; dividing the set of candidate keys into k' sub-sets of candidate keys, the k' sub-sets of candidate keys being in one-to-one correspondence with the k' blocks of the verification sketch; testing each sub-set of candidate keys by using one block of the verification sketch of the k' blocks of the verification sketch to obtain a verified current list of frequent keys; and outputting the current list of frequent keys in stream with insertions and / or retractions.

[0206] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement:determining an r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on a predefined first function, wherein r Gtransferring each record to the target block of the candidate generation sketch; determining an r'-th block of the verification sketch as the target block of the verification sketch based on a predefined second function, wherein r' G {1, ... , / c'}; transferring each record to the target block of the verification sketch; updating majority testers corresponding to the record’s key in the target block of the candidate generation sketch; and updating the counters corresponding to the keys of records in the target block of the verification sketch.

[0207] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c determining that r = h0(s) based on a hash function h0: U -> {1, ... , k }, the r-th block of the candidate generation sketch being the target block of the candidate generation sketch.

[0208] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c determining that r' = h'0(s) based on a hash function h'o: U -> {1, ... , k'}, the r'-th block of the verification sketch being the target block of the verification sketch.L 0209 ] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c obtaining an array of majority testers Mijr, i = 1,j = 1, ..., w / k corresponding to the target block r G {1, ... , k] of the candidate generation sketch and updating the majority testers by sending d copies of keys corresponding to the records to the array of majority testersselected via hash functions hj\ U -> {1, ... , w / k], i = l, ... , d.

[0210] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c obtaining an array of counterscorresponding to the target block r' G {1, ... , k'}; of the verification sketch andupdating the counters Cih^r, selected via hash functions h'p U -> {1, ... , w' / k'}, t = 1, ... , d' , by incrementing them by count c.

[0211] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a set of parameters; determining a total width and a total depth of the candidate generation sketch, and a number k of blocks of based on the set of parameters; determining a total width and a total depth of the verification sketch, and a number k' of blocks based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

[0212] In one possible implementation, the set of parameters includes at least one or more of: a maximum length of a key, a minimum frequency that will be of interest during run of a streaming program, an inaccuracy, a failure probability, or a block size; and the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during run of the streaming program, the failure probability, and the block size; determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability; and determining the total width of the candidate generation sketch based on the total depth of the candidate generation sketch, and the correspondence relationship between the total width and the total depth of the candidate generation sketch.

[0213] In one possible implementation, the computer instructions, when executed by the at least i , one processor, cause the computing device to implement: w is proportional to • — , where<P2p is the maximum length of the key,is the minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

[0214] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: d is proportional to log — , whereisthe minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, and d is the total depth of the candidate generation sketch.

[0215] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the verification sketch based on the inaccuracy, the failure probability, and the block size; determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch; and determining the total width of the verification sketch based on the total depth of the verification sketch and the correspondence relationship between the total width and the total depth of the verification sketch.

[0216] In one possible implementation, the computer instructions, when executed by the at leastI d? 1 1 one processor, cause the computing device to implement: w' is proportional to J— In- where£ is the inaccuracy, Q is the failure probability, B is the block size, w’ is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0217] In one possible implementation, the computer instructions, when executed by the at leastone processor, cause the computing device to implement: d' is proportional to log where Q is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

[0218] In one possible implementation, each block of the candidate generation sketch has a width of w / k and a depth of d, where w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch; and each block of the verification sketch has a width of w' / k' and a depth of d' , wherein w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

[0219] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: the number k of blocks of the candidate generation sketch is proportional towhere p is the maximum length of the key, and B is theblock size.

[0220] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement: the number k' of blocks of the verification sketch is proportional t

[0221] In one possible implementation, the computer instructions, when executed by the at least one processor, cause the computing device to implement:outputting a candidate key only if the values of all counters, that correspond to this candidate key in the corresponding block of the verification sketch, is greater than or equal to the predefined threshold. Each sub-set of candidate keys including a plurality of candidate keys.

[0222] It should be understood that the computing device 300 according to the embodiments can implement the stream processing method in the embodiments, which will not be repeated here for the sake of brevity.

[0223] As shown in FIG. 6, the computing device 300 includes a processor 302, a bus 303, a storage 301, a communication interface 304 and a memory (namely, a main memory) 305. The processor 302, the storage device 301, the memory 305 and the communication interface 304 are connected through the bus 303.

[0224] In some embodiments, the processor 302 may be a central processing unit (CPU), or other general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or a transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0225] The computing device 300 may also include a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling program execution of the present disclosure.

[0226] The communication interface 304 is used to realize the communication between the computing device 300 and external devices or components. In the embodiments of the present disclosure, when the computing device 300 is used to implement the update or query of the computing device, the communication interface 304 is used to obtain a stream so that the processor 302 can perform the stream processing.

[0227] The bus 303 may include a path for transferring information between the above components (e.g., the processor 302, memory 305, and storage 301). In addition to a data bus, the bus 303 may also include a power bus, a control bus, a state signal bus, etc. However, for the sake of clarity, various buses are marked as "Bus 303" in the figure. The bus 303 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), or a cache coherent interconnect for accelerators (CCIX). The bus 303 may be divided into an address bus, a data bus, a control bus, etc.

[0228] For example, the computing device 300 may include a plurality of processors. The processor may be a multi-core processor (multi-CPU). The processor 302 may refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions).

[0229] It is will be noted that FIG. 6 only shows an example where the computer device 300 includes one processor 302 and one storage 303. Herein, the processor 302 and the storage 303 are respectively used to indicate a type of component or device. In specific embodiments, the number of each type of component or device may be determined according to the needs of the service.

[0230] The memory 305 may be either a volatile memory pool or a non-volatile memory pool, or may include both volatile memory and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable read-only memory (electrically EPROM, EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), which acts as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data date SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) and direct rambus RAM (DR RAM). The memory 305 is used to store the query latency target, the data freshness target, the query time cost, the loading time cost, the query timestamp, the loading timestamp, etc.

[0231] The storage 301 may correspond to the storage medium (e.g., a magnetic disk, such as a mechanical hard disk or a solid state hard disk) used to store computer instructions in the above method embodiments.

[0232] The above computing device 300 may be a general-purpose device or a special-purpose device. For example, the computer device 300 may be an edge device (e.g., a box carrying a chip with processing power). In some embodiments, the computer device 300 may also be a server or other device with computing capabilities.

[0233] Some embodiments of the present disclosure provide a computer program product including instructions that, when executed by a computer, cause the computer to perform the method in any one of the above embodiments of the present disclosure.

[0234] Some embodiments of the present disclosure provide a computer-readable storage medium has stored a computer program or instructions. The computer program or instructions, when executed by a computer, cause the computer to perform the method as described in any one of the above embodiments.

[0235] Some embodiments of the present disclosure provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server or a local server in a local data center. In someembodiments, the computing device may be a terminal device, such as a desktop computer, a laptop computer or a smartphone.

[0236] FIG. 7 shows a structure of the computing device cluster provided in the embodiments of the present disclosure. As shown in FIG. 7, the computing device cluster includes at least one computing device 300. The memory 301 in one or more computing devices 300 in the computer cluster may store the same instructions for executing the stream processing method.

[0237] In some possible implementations, memories 301 in a plurality of computing devices 300 in the computer cluster may each store some of the instructions for executing the stream processing method. In other words, the plurality of computing devices 300 can jointly perform the instructions for executing the stream processing method.

[0238] It will be noted that in the computer cluster, different memories 301 in different computing devices 300 may store different instructions for executing different parts of functions of the stream processing method.

[0239] In some embodiments, one or more computing devices in the computer cluster may be connected via a network, such as a wide area network (WAN) or a local area network (LAN).

[0240] It is to be understood that the processor in the embodiments of the present disclosure may be a CPU, and the processor may also be other general purpose processor, DSP, ASIC, FPGA or other programmable logic device, a discrete gate or a transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0241] It should also be understood that the memory in the embodiments of the present disclosure may be either a volatile memory or a non-volatile memory, or may include both volatile memory and non-volatile memory. The non-volatile memory may be a ROM, a PROM, an EPROM, an EEPROM or a flash memory. The volatile memory may be an RAM, which acts as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as SRAM, DRAM, SDRAM, DDR SDRAM, ESDRAM, SLDRAM and DR RAM.

[0242] The embodiments described above may be implemented, in whole or in part, by software, hardware (e.g., circuitry), firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loading or executing the computer instructions or computer programs on a computer produces, in whole or in part, a process or function in accordance with the embodiments of the present disclosure. The computer may be a general purpose computer, a special-purpose computer, a computer network, or another programmable device. The computer instructions may bestored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g., the computer instructions may be transmitted by wired or wireless (e.g., infrared, Wi-Fi, microwave, etc.) means from one website site, computer, server, or data center to another website site, computer, server, or data center. The computer-readable storage medium may be any usable medium to which a computer is capable of accessing or a data storage device such as a server, data center, etc. that contains a collection of one or more usable media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, and tape), an optical medium (e.g., DVD), or a semiconductor medium, where the semiconductor medium may be a solid state disk.

[0243] It should be understood that the term "and / or" herein is merely a description of the relationship of associated objects, and indicates that three types of relationships may exist, e.g., A and / or B, which may indicate the existence of A alone, the existence of both A and B, and the existence of B alone, where A and B may each be in singular or in plural. In addition, the character " / " herein, generally indicates that associated objects before and after this character are in an "or" relationship, but may also indicate in an "and / or" relationship, specific reference can be made to the preceding and following text for understanding.

[0244] In the embodiments of the present disclosure, "at least one" refers to one or more, and "a plurality of' refers to two or more. The expression "at least one item (one) of the following" or similar expressions refers to any combination of these items, including any combination of singular item (one) or plural items (plurality). For example, at least one item (one) of a, b, or c may be expressed as: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, and c may each be singular or plural.

[0245] In the embodiments of the present disclosure, the words "exemplarily", "for example", and the like are used to denote examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present disclosure should not be construed as being preferred or advantageous over other embodiments or design schemes. Rather, the use of the term "example" is intended to present concepts in a concrete manner.

[0246] It should be understood that in various embodiments of the present disclosure, the magnitude of the serial numbers of the above-described processes does not imply a sequential order of execution, and the order in which the processes are executed should be determined by their function and inherent logic, and should not constitute any limitation on the processes for implementing embodiments of the present disclosure.

[0247] Those of ordinary skill in the art may realize that the units and algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein are capable of being implemented in electronic hardware, or a combination of computer software and electronichardware. Whether these functions are performed in hardware or software depends on the particular application and design constraints of the technical solution. The skilled professional may use different methods to implement the described functions for each particular application, but such implementations should not be considered outside the scope of the present disclosure.

[0248] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the working processes of the systems, devices and modules described above may refer to the corresponding processes in the above embodiments for the method, and details will not be repeated here.

[0249] In the embodiments provided by the present disclosure, it will be understood that the system, device and method disclosed may be implemented in other ways. For example, embodiments of the device described above are merely exemplary. For example, the division of the units is only a logical functional division. In actual implementation, there are another division manners. For example, a plurality of devices or components are combined or integrated into another system, or some features may be ignored or not executed. Moreover, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

[0250] The units described as separate components may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, the component may be located in one place, or may be distributed to multiple different network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the solutions of the embodiments.

[0251] In addition, various functional units in various embodiments of the present disclosure may be integrated in a single processing unit, or each unit may physically exist separately, or two or more units may be integrated in a single unit.

[0252] If the described functions may be stored in a computer-readable storage medium when implemented as a software functional unit and sold or used as a stand-alone product. Based on this understanding, the technical solution of the present disclosure is essentially or contributes to the prior art or parts of the technical solution may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in various embodiments of the present disclosure. The aforementioned storage media includes a USB flash drive, a removable hard disk, an ROM, an RAM, a magnetic disk or optical disk, and other media that can store program codes.

[0253] The foregoing description is only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or replacements that a person skilled in the art could conceive of within the technical scope of the present disclosureshall be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

CLAIMS1. A stream processing method, comprising: obtaining a candidate generation sketch, the candidate generation sketch including k blocks, k being an integer greater than or equal to 1 ; obtaining a verification sketch, the verification sketch including k' blocks, k' being an integer greater than or equal to 1 ; receiving a plurality of records in a stream with insertions and / or retractions, each of the records including a key and a count that are in correspondence; updating the sketches by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k' blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch; querying the k blocks of the candidate generation sketch to determine a set of candidate keys; dividing the set of candidate keys into k' sub-sets of candidate keys, the k' sub-sets of candidate keys being in one-to-one correspondence with the k' blocks of the verification sketch; testing each sub-set of candidate keys by using one block of the verification sketch of the k' blocks of the verification sketch to obtain a verified current list of frequent keys; and outputting the current list of frequent keys in the stream with insertions and / or retractions.

2. The stream processing method of claim 1, wherein updating the key and the count by transferring each record to the target block of the candidate generation sketch and the target block of the verification sketch, includes: for each record (s, c) with key s and count c determining an r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on a predefined first function, wherein r E {1, ... , k}; transferring record (s, c) to the target block of the candidate generation sketch; determining an r'-th block of the verification sketch as the target block of the verification sketch based on a predefined second function, wherein r' E {1, ... , k'}; transferring record (s, c) to the target block of the verification sketch; updating majority testers corresponding to the key s in the target block of the candidate generation sketch; and updating the counters corresponding to the key s in the target block of the verification sketch.

3. The stream processing method of claim 2, wherein determining the r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on the predefined first function, includes: for a record (s, c) with key s and count cdetermining that r = h0(s) based on a hash function h0: U {1, ... , k }, the r-th block of the candidate generation sketch being the target block of the candidate generation sketch; determining the r'-th block of the verification sketch as the target block of the verification sketch based on the predefined second function, includes: determining that r' = h'0(s) based on a hash function{1, / c'}, the r'-th block of the verification sketch being the target block of the verification sketch.

4. The stream processing method of claim 2, wherein updating majority testers corresponding to the key in each record in the target block of the candidate generation sketch, includes: for a record (s, c) with key s and count c obtaining an array of majority testersj = 1, ... , w / k corresponding to the target block of the candidate generation sketch r E {1, ... , k}; and updating the majority testers by sending d copies of the corresponding record to the array of majority testers Mih.^rselected via hash functions h^ U -* {1, ... , w / k], i = 1, ... , d.

5. The stream processing method of claim 2, wherein updating the counters corresponding to the key in each record in the target block of the verification sketch, includes: for a record (s, c) with key s and count c obtaining an array of counters Cijr, i = 1, ... , d', j = 1,corresponding to the target block r' 6 {1, ... , k'} of the verification sketch; and updating the counters Cihf^r, selected via hash functions h't: U -> {1, ... , w' / k'], i = 1, ... , d1, by incrementing them by count c6. The stream processing method of claim 1, wherein obtaining the candidate generation sketch and obtaining the verification sketch includes: obtaining a set of parameters; determining a total width and a total depth of the candidate generation sketch, and a number k of blocks based on the set of parameters; and determining a total width and a total depth of the verification sketch, and a number k1of blocks based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

7. The stream processing method of claim 6, wherein the set of parameters includes at least one or more of: a maximum length of a key, a minimum frequency that will be of interest during run of a streaming program, an inaccuracy, a failure probability, or a block size; determining the total width and the total depth of the candidate generation sketch based on the set of parameters includes:determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during run of the streaming program, the failure probability, and the block size; determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability; and determining the total width of the candidate generation sketch based on the total depth of the candidate generation sketch, and the correspondence relationship between the total width and the total depth of the candidate generation sketch.

8. The stream processing method of claim 7, wherein the correspondence relationship between the total width and the total depth of the candidate generation sketch includes: w is proportional twherein p is the maximum length of the key, <t> is the minimum frequency that will be of interest during run of the streaming program, g is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

9. The stream processing method of claim 7, wherein determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability, includes: d is proportional twherein <t> is the minimum frequency that will be of interest during run of the streaming program, g is the failure probability, and d is the total depth of the candidate generation sketch.

10. The stream processing method of claim 7, wherein determining the total width and the total depth of the verification sketch based on the set of parameters as well as the total width and the total depth of the candidate generation sketch, includes: determining a correspondence relationship between the total width and the total depth of the verification sketch based on the inaccuracy, the failure probability, and the block size; determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch; and determining the total width of the verification sketch based on the total depth of the verification sketch and the correspondence relationship between the total width and the total depth of the verification sketch.

11. The stream processing method of claim 10, wherein the correspondence relationship between the total width and the total depth of the verification sketch includes: w' is proportional twherein E is the inaccuracy, Q is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

12. The stream processing method of claim 10, wherein determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch includes: d' is proportional twherein g is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

13. The stream processing method of any one of claims 6 to 12, wherein each block of the candidate generation sketch has a width of w / k and a depth of d, wherein w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch; each block of the verification sketch has a width of w' / k' and a depth of d' , wherein w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

14. The stream processing method of claim 13, wherein determining the number k of blocks of the candidate generation sketch based on the set of parameters, includes:the number k of blocks of the candidate generation sketch is proportional to wherein pis the maximum length of the key, and B is the block size; wherein determining the number k' of blocks of the verification sketch based on the set of parameters, includes:the number k' of blocks of the verification sketch is proportional to15. The stream processing method of claim 1, wherein testing each sub-set of candidate keys by using one block of the verification sketch out of the k' blocks of the verification sketch to obtain the verified current list of frequent keys, includes: outputting a candidate key only if the values of all counters, that correspond to this candidate key in the corresponding block of the verification sketch, is greater than or equal to the predefined threshold. Each sub-set of candidate keys including a plurality of candidate keys.

16. A computing device, comprising: a memory; and at least one processor connected to the memory; wherein the memory is configured to store computer instructions; and the computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a candidate generation sketch, the candidate generation sketch including k blocks, k being an integer greater than or equal to 1 ;obtaining a verification sketch, the verification sketch including k' blocks, k’ being an integer greater than or equal to 1 ; receiving a plurality of records in a stream with insertions and / or retractions, each of the records including a key and a count that are in correspondence; updating the sketches by transferring each record to one block of the candidate generation sketch of the k blocks of the candidate generation sketch and one block of the verification sketch of the k' blocks of the verification sketch, the one block of the candidate generation sketch being a target block of the candidate generation sketch, and the one block of the verification sketch being a target block of the verification sketch; querying the k blocks of the candidate generation sketches to determine a set of candidate keys; dividing the set of candidate keys into k' sub-sets of candidate keys, the k' sub-sets of candidate keys being in one-to-one correspondence with the k' blocks of the verification sketch; testing each sub-set of candidate keys by using one block of the verification sketch out of the k' blocks of the verification sketch to obtain a verified current list of frequent keys; and outputting the current list of frequent keys in the stream with insertions and / or retractions.

17. The computing device of claim 16, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining an r-th block of the candidate generation sketch as the target block of the candidate generation sketch based on a predefined first function, wherein r G {1, ... , k}; transferring each record to the target block of the candidate generation sketch; determining an r'-th block of the verification sketch as the target block of the verification sketch based on a predefined second function, wherein r' G {1, ... , / c'}; transferring each record to the target block of the verification sketch; updating majority testers corresponding to the key in each record in the target block of the candidate generation sketch; and updating the counters corresponding to the keys of records in the target block of the verification sketch.

18. The computing device of claim 17, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c determining that r = h0(s) based on a hash function h0: U -> {1, ... , k }, the r-th block of the candidate generation sketch being the target block of the candidate generation sketch ; and determining that r' = tints') based on a hash function h'0: U -» {1, ... , k'}, the r'-th block of the verification sketch being the target block of the verification sketch .

19. The computing device of claim 17, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c obtaining an array of majority testers Mijr, i = 1,j = 1, ..., w / k corresponding to the target block r E {1, ... , k] of the candidate generation sketch and updating the majority testers by sending d copies of the corresponding record to the array of majority testers MiflQSyrvia hash functions h^ U -» {1, ... , w / k}, i = 1, ... , d.

20. The computing device of claim 17, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: for a record (s, c) with key s and count c obtaining an array of counters Q7r, I = 1, .... d' , j = 1, ... , w' / k' corresponding to the target block of the verification sketch r' G {1, ... , k'}; and updating the counters Cih>^rrselected via hash functions1, ... , d', by incrementing them by count c.

21. The computing device of claim 16, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: obtaining a set of parameters; determining a total width and a total depth of the candidate generation sketch, and a number k of blocks of the candidate generation sketch based on the set of parameters; determining a total width and a total depth of the verification sketch, and a number k' of blocks of the verification sketch based on the set of parameters as well as the total width and the total depth of the candidate generation sketch to obtain the verification sketch.

22. The computing device of claim 21, wherein the set of parameters includes at least one or more of: a maximum length of a key, a minimum frequency that will be of interest during run of a streaming program, an inaccuracy, a failure probability, or a block size; the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the candidate generation sketch based on the maximum length of the key, the minimum frequency that will be of interest during run of the streaming program, the failure probability, and the block size; determining the total depth of the candidate generation sketch based on the minimum frequency that will be of interest during run of the streaming program and the failure probability; and determining the total width of the candidate generation sketch based on the total depth of the candidate generation sketch, and the correspondence relationship between the total width and the total depth of the candidate generation sketch.

23. The computing device of claim 22, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: w is proportional twherein p is the maximum length of the key, 0 is the minimum frequency that will be of interest during run of the streaming program, g is the failure probability, B is the block size, w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch.

24. The computing device of claim 22, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: d is proportional twherein <t> is the minimum frequency that will be of interest during run of the streaming program, Q is the failure probability, and d is the total depth of the candidate generation sketch.

25. The computing device of claim 22, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: determining a correspondence relationship between the total width and the total depth of the verification sketch based on the inaccuracy, the failure probability, and the block size; determining the total depth of the verification sketch based on the failure probability as well as the total width and the total depth of the candidate generation sketch; and determining the total width of the verification sketch based on the total depth of the verification sketch and the correspondence relationship between the total width and the total depth of the verification sketch.

26. The computing device of claim 25, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: w' is proportional twherein s is the inaccuracy, Q is the failure probability, B is the block size, w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

27. The computing device of claim 25, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: d' is proportional twherein Q is the failure probability, w is the total width of the candidate generation sketch, d is the total depth of the candidate generation sketch, and d' is the total depth of the verification sketch.

28. The computing device of any one of claims 21 to 27, wherein each block of the candidate generation sketch has a width of w / k and a depth of d, wherein w is the total width of the candidate generation sketch, and d is the total depth of the candidate generation sketch; each block of the verification sketch has a width of w' / k' and a depth of d', wherein w' is the total width of the verification sketch, and d' is the total depth of the verification sketch.

29. The computing device of claim 28, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: the number k of blocks of the candidate generation sketch is proportional towherein p is the maximum length of the key, and B is the block size; andthe number k' of blocks of the verification sketch is proportional to30. The computing device of claim 16, wherein the computer instructions, when executed by the at least one processor, cause the computing device to implement: outputting a candidate key only if the values of all counters, that correspond to this candidate key in the corresponding block of the verification sketch, is greater than or equal to the predefined threshold. Each sub-set of candidate keys including a plurality of candidate keys.

31. A computer-readable storage medium having stored computer instructions, wherein when executed by a computer, the computer instructions cause the computer to perform the stream processing method of any one of claims 1 to 15.

32. A computer program product, comprising instructions that, when executed by a computer, cause the computer to perform the stream processing method of any one of claims 1 to 15.

33. A computer cluster, comprising at least one computing device each including a processor and a memory; wherein the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, to cause the computer cluster to perform the stream processing method of any one of claims 1 to 15.

Images