A data query method and system for accelerating cuckoo filter based on local minimum strategy

Abstract

The present invention discloses a data query method and system for accelerating a cuckoo filter based on a local minimum strategy. The method includes assigning a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kickouts that have occurred in the corresponding bucket; when all candidate buckets for the first element to be inserted are full, the bucket with the smallest label is selected from all candidate buckets as the first candidate bucket; for each fingerprint stored in the first candidate bucket, the smallest label among the labels of the remaining candidate buckets except the current candidate bucket is calculated, and the smallest label is used as the label of the fingerprint; the fingerprint with the smallest fingerprint label is selected from the first candidate bucket for kicking out, and the fingerprint of the first element is inserted; the element corresponding to the kicked fingerprint is used as the new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the label of the bucket reaches a predefined threshold. The solution provided by the present invention can achieve higher insertion throughput under the same conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of cuckoo filters, and in particular to a data query method and system for accelerating cuckoo filters based on a local minimum strategy. Background Art

[0002] With the development of big data applications, many approximate data structures are used in databases, caches, and network measurements to support membership queries. It is very important to design a data structure that is both space-saving and computationally efficient for set representation and membership queries. The most classic set representation data structure is the Bloom filter (BF) and its variants. It stores the raw data of the elements in the set in a cell array as Boolean values. However, BF and its variants have limitations in deletion operations. Therefore, Fan et al. proposed the Cuckoo Filter (CF), a lightweight approximate data structure with higher space utilization and faster deletion speed.

[0003] CF is a probabilistic data structure that supports deletion operations and is superior to BF in element query speed. CF consists of m buckets, each bucket has b slots and can store up to b fingerprints. Each element has k candidate buckets, that is, , the fingerprint is stored in the candidate bucket. The core of Cuckoo filter is to use the concept of kick-out-reinsert to resolve hash conflicts. When each element has two candidate buckets, they are represented by the hash function and Calculated. Figure 1 Elements described in storage, CF checks the address of the candidate bucket and tries to find an empty slot to store the fingerprint. Store it. If the element All candidate buckets are full, CF will randomly kick out an element. In this example, it is assumed that the element , and Stored to the corresponding address. Then the element If another candidate bucket is also full, the above-mentioned kick-out-reinsert operation will be repeated, such as randomly kicking out elements. To store Ultimately, if If an idle position is identified, the recursive process will terminate; otherwise, the above reallocation operation will be repeated.

[0004] That is, the cuckoo filter provides two candidate buckets for each element. When both candidate buckets are full, the storage buckets of randomly kicked elements are recursively redistributed until the insertion is successful or the maximum number of kicks is reached. However, when the load of the cuckoo filter exceeds 0.8, the redistribution strategy will immediately lead to a decrease in insertion performance. Intuitively, the original redistribution strategy can be regarded as a random walk in a random graph, which may lead to uneven kicking frequencies between buckets, that is, frequent redistribution of elements from several fixed buckets, affecting the processing analysis of high-speed streams. Summary of the invention

[0005] In order to solve the technical problem that the random walk reallocation strategy will significantly reduce the insertion performance when the load of the cuckoo filter is large, the embodiment of the present invention provides a data query method and system for accelerating the cuckoo filter based on the local minimum strategy.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides a data query method for accelerating a cuckoo filter based on a local minimum strategy, the method comprising:

[0008] Assign a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kicks that have occurred in the corresponding bucket;

[0009] When all candidate buckets for the first element to be inserted are full, select a bucket with the smallest label from all candidate buckets as the first candidate bucket;

[0010] For each fingerprint stored in the first candidate bucket, calculate the minimum label of the fingerprint among the labels of the remaining candidate buckets except the current candidate bucket, and use the minimum label as the label of the fingerprint;

[0011] Select the fingerprint with the smallest fingerprint label from the first candidate bucket to kick it out, and insert the fingerprint of the first element;

[0012] The element corresponding to the kicked-out fingerprint is used as the new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the bucket label reaches the predefined threshold.

[0013] In one embodiment, the predefined threshold is , m is the number of buckets in the cuckoo filter.

[0014] In one embodiment, the predefined threshold is determined by:

[0015] Modeling cuckoo filters based on directed graphs;

[0016] Based on the modeling of the cuckoo filter, four lemmas and two theorems are determined;

[0017] According to the four lemmas and two theorems, a predefined threshold is determined.

[0018] In one embodiment, a cuckoo filter is modeled based on a directed graph, including:

[0019] Through the directed graph For each bucket, slots, each element has The cuckoo filter of candidate buckets is modeled; the candidate bucket corresponding to a vertex stores elements, then the out-degree of the corresponding vertex is ; The vertex The set of neighbor vertices of ,for , using directed edges Represents an element Stored in the vertex middle, Is an element Alternative candidate buckets;

[0020] Insert The total number of insertion or removal operations required to perform the process of elements is recorded as , in a directed graph , express Step after vertex Label, The corresponding unfilled bucket set and edge set at step are , ,make Represents from the vertex arrive The shortest distance, that is ;for For any vertex in , we have: .

[0021] In one embodiment, 4 lemmas and 2 theorems include:

[0022] Lemma 1: ;

[0023] Lemma 2: , The tags have the following relationship: ;

[0024] Lemma 3: For a graph Any , Make Existence , ,right satisfy ;

[0025] Lemma 4: In probability Next, for any and , bucket collection Can completely contain the collection of elements Must meet ;

[0026] Theorem 1: Let represents a bipartite graph, Represents the collection of elements to be stored, Represents a slot set, both exist Relationship, then have arrive A perfect match is a perfect match if and only if for every subset ,inequality Established, among which, yes The set of slots that are adjacent to at least one element in ;

[0027] Theorem 2: With probability , the maximum label of any vertex is .

[0028] The embodiment of the present invention further provides a data query system for accelerating a cuckoo filter based on a local minimum strategy, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein when the processor is used to run the computer program, the following steps are performed:

[0029] Assign a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kicks that have occurred in the corresponding bucket;

[0030] When all candidate buckets for the first element to be inserted are full, select a bucket with the smallest label from all candidate buckets as the first candidate bucket;

[0031] For each fingerprint stored in the first candidate bucket, calculate the minimum label of the fingerprint among the labels of the remaining candidate buckets except the current candidate bucket, and use the minimum label as the label of the fingerprint;

[0032] Select the fingerprint with the smallest fingerprint label from the first candidate bucket to kick it out, and insert the fingerprint of the first element;

[0033] The element corresponding to the kicked-out fingerprint is used as the new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the bucket label reaches the predefined threshold.

[0034] In one embodiment, the predefined threshold is , m is the number of buckets in the cuckoo filter.

[0035] In one embodiment, the predefined threshold is determined by:

[0036] Modeling cuckoo filters based on directed graphs;

[0037] Based on the modeling of the cuckoo filter, four lemmas and two theorems are determined;

[0038] According to the four lemmas and two theorems, a predefined threshold is determined.

[0039] In one embodiment, a cuckoo filter is modeled based on a directed graph, including:

[0040] Through the directed graph For each bucket, slots, each element has The cuckoo filter of candidate buckets is modeled; the candidate bucket corresponding to a vertex stores elements, then the out-degree of the corresponding vertex is ; The vertex The set of neighbor vertices of ,for , using directed edges Represents an element Stored in the vertex middle, Is an element Alternative candidate buckets;

[0041] Insert The total number of insertion or removal operations required to perform the process of elements is recorded as , in a directed graph , express Step after vertex Label, The corresponding unfilled bucket set and edge set at step are , ,make Represents from the vertex arrive The shortest distance, that is ;for For any vertex in , we have: .

[0042] In one embodiment, 4 lemmas and 2 theorems include:

[0043] Lemma 1: ;

[0044] Lemma 2: , The tags have the following relationship: ;

[0045] Lemma 3: For a graph Any , Make Existence , ,right satisfy ;

[0046] Lemma 4: In probability Next, for any and , bucket collection Can completely contain the collection of elements Must meet ;

[0047] Theorem 1: Let represents a bipartite graph, Represents the collection of elements to be stored, Represents a slot set, both exist Relationship, then have arrive A perfect match is a perfect match if and only if for every subset ,inequality Established, among which, yes The set of slots that are adjacent to at least one element in ;

[0048] Theorem 2: With probability , the maximum label of any vertex is .

[0049] This embodiment has the following beneficial effects:

[0050] This embodiment uses the number of buckets kicked out as prior knowledge and uses the matching relationship between elements and buckets in graph theory to optimize the storage of elements with as few kick-out paths as possible. This embodiment not only considers the relationship between buckets and between buckets and elements, but also achieves higher insertion throughput under the same conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1Schematic diagram of the insertion process of the original CF;

[0052] Figure 2 It is a flow chart of a method for accelerating a data query of a cuckoo filter based on a local minimum strategy according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of an example of a local minimum strategy according to an embodiment of the present invention;

[0054] FIG4 (a) shows an embodiment of the present invention when the parameters are and Schematic diagram of the results of the experiment in CF;

[0055] FIG4( b ) shows an embodiment of the present invention when the parameters are and Schematic diagram of the results of the experiment in CF;

[0056] FIG4(c) shows an embodiment of the present invention when the parameters are and Schematic diagram of the results of the experiment in CF;

[0057] FIG5 (a) is a schematic diagram of a comparison result of the insertion time of evaluation indicators according to an embodiment of the present invention;

[0058] FIG5( b ) is a schematic diagram of comparison results of evaluation indicator insertion throughput according to an embodiment of the present invention;

[0059] FIG5( c ) is a schematic diagram of comparison results of the number of kickouts of evaluation indicators according to an embodiment of the present invention;

[0060] FIG6 (a) is a schematic diagram showing the effect of bucket size on insertion time according to an embodiment of the present invention;

[0061] FIG6 (b) is a schematic diagram showing the effect of the threshold on the insertion time according to an embodiment of the present invention;

[0062] Figure 7 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0064] The embodiment of the present invention provides a data query method for accelerating a cuckoo filter based on a local minimum strategy, such as Figure 2 As shown, the method includes:

[0065] Step 101: assign a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kicks that have occurred in the corresponding bucket;

[0066] Step 102: When all candidate buckets for the first element to be inserted are full, select a bucket with the smallest label from all candidate buckets as the first candidate bucket;

[0067] Step 103: for each fingerprint stored in the first candidate bucket, calculate the minimum label of the fingerprint among the labels of the remaining candidate buckets except the current candidate bucket, and use the minimum label as the label of the fingerprint;

[0068] Step 104: Select the fingerprint with the smallest fingerprint label from the first candidate bucket to kick it out, and insert the fingerprint of the first element;

[0069] Step 105: The element corresponding to the kicked-out fingerprint is used as a new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the label of the bucket reaches a predefined threshold.

[0070] This embodiment proposes a new cuckoo filter reallocation strategy. This strategy combines the local minimum of the candidate bucket counter label with the local minimum of the element virtual label to significantly reduce the number of kicks and the time required for element insertion. Specifically, this embodiment assigns a counter as a label to each bucket to record the number of kicks that have occurred. Lower labels are associated with lower kick costs, which means that they have a higher probability of having a shorter kick path. When each reallocation operation is activated, the bucket with the smallest label is selected, and the element with the smallest virtual label is selected from the bucket as the kick object. The virtual label of the element is dynamically calculated by the minimum label of its remaining k-1 candidate buckets. This strategy achieves higher insertion throughput at the lowest label cost, and has significant application value for scenarios such as large-scale data processing or high-speed network flow monitoring. In theory, this new kick strategy in this embodiment can eventually ensure successful insertion within the upper limit of the kick operation.

[0071] Specifically, this embodiment first introduces the development motivation of this solution, and then introduces the algorithm of this embodiment in detail.

[0072] 1. Observation and motivation

[0073] In this embodiment, a directed graph For each bucket, slots, each element has We model the CF of candidate buckets and analyze their redistribution operations. If a vertex corresponds to a candidate bucket with elements, then the vertex will be considered to be a fully stored vertex, and the corresponding vertex out-degree is In this embodiment, the vertex The set of neighbor vertices of , it should be noted that the vertex out-degree is less than Corresponding to the vertices that are not fully stored. , this embodiment uses directed edges Represents an element Stored in the vertex middle, Is an element In addition, the directed graph exist If it is greater than 2, it is converted into a hypergraph .

[0074] When the load factor of the cuckoo filter increases, all available buckets may be occupied. If an element y is removed, at least one candidate bucket is full. In this case, there are two possible cases to consider. First, another candidate bucket has available capacity and y is successfully inserted. Otherwise, it needs to kick out another element. When the conflict occurs again and the element When in the optional kick list of the new element, The probability of being kicked out again should be less than that of other elements that have not been kicked out. In addition, if the candidate bucket of an element is full or has been kicked out many times, the probability of the element being kicked out should be lower than that of other elements. However, due to the inherent randomness of the random walk strategy (RW), the probability of element y being kicked out is the same as that of other elements. This will lead to inevitable infinite loops or repeated reallocations, which will eventually affect the overall performance of the structure. On the other hand, the kicking frequency of each bucket when using RW is uneven, which conforms to the long-tail distribution. That is, some candidate buckets experience frequent kicking operations, while others do not. The number of reallocations directly determines the insertion time of each element. Inspired by the long-tail theory, the cumulative time cost of the kicking frequency at the tail is comparable to or even greater than the kicking frequency at the head.

[0075] Redistribution during element insertion is similar to determining a feasible path that starts from any subset of vertices and terminates when the output degree of the vertices is not less than 0. Obviously, similar problems can be solved by graph traversal algorithms. The traditional cuckoo filter can be equivalent to the random walk (RW) method in a directed graph. RW is effective, but it may mislead reallocation decisions to some extent and fall into cycles. Another classic graph traversal method is breadth-first search (BFS), which can be executed in parallel to check all candidate buckets and determine the best path to perform iterative reallocation operations. Although BFS can effectively reduce the number of kicks and avoid the appearance of cycles, its search space expands exponentially with the increase of load. In addition, excessive use of temporary caches will further affect insertion performance.

[0076] When comparing the RW strategy and the BFS strategy, it is observed that the difference in kickout frequency gradually increases as the load intensity increases. If the kickout of each bucket is as balanced as possible, the length of the reallocation can be significantly minimized. This embodiment proposes a reallocation algorithm that significantly reduces the frequency of reallocation operations by using auxiliary space, thereby improving the insertion efficiency of CF in a cache-friendly environment.

[0077] 2. Algorithm Design

[0078] In this section, this embodiment first proposes a local minimum strategy, which is a deterministic strategy for selecting a candidate bucket and kicking out an element when all candidate buckets for inserting an element are full. Then, this embodiment theoretically analyzes the rationality of the algorithm and proposes a method for calculating the upper threshold of the label in each bucket. Finally, this embodiment compares the time and space costs of the algorithm with those of the RW algorithm.

[0079] 2.1. Local Minimum Strategy

[0080] This embodiment assigns a counter as a label to each bucket in the CF and systematically updates this counter in each element kick instance. This approach helps to dynamically evaluate the operational costs associated with reallocation. These counters are the main determinant of the allocation priority during the kick process. At each reallocation, the bucket with the smallest label is selected and the victim (i.e., the kicked object) is selected from this bucket. The victim's label is not actually stored, but is determined by its remaining The minimum label of the candidate is determined dynamically.

[0081] This embodiment uses an example to illustrate the algorithm implementation process of this embodiment. Assume If both candidate buckets are fully occupied, the reallocation operation will be activated. The labels of the two candidate buckets are 3 and 1, indicating the number of times they have been kicked out. Figure 3 As shown, this strategy The candidate buckets are selected from the bucket with the smallest label to determine the element to be kicked out. element fingerprint, so the process of selecting elements to be kicked out in a bucket is still controlled by the minimum label rule. For each fingerprint in a given bucket, the remaining Candidate buckets and their associated labels. The label of each fingerprint is defined as The minimum label identified in the candidate bucket. In this case, the minimum label of the fingerprint in the victim bucket is zero, corresponding to the fingerprint . Then, Replace with By repeating the above steps, the victim will be reallocated to its The reallocation process will continue until there are no more evictees or the bucket label reaches a predefined threshold. Once the bucket label reaches the threshold, the cuckoo filter is considered full and no more elements can be inserted, resulting in insertion failure.

[0082] Table 1 below provides pseudo code for implementing reallocation using a local minimum strategy when the candidate bucket for inserting elements in the cuckoo filter is full in this embodiment. express The bucket with the smallest label among the candidate buckets The stored fingerprint collection, is the threshold for the label in each bucket. Buckets with the smallest labels are associated with lower eviction costs, meaning that element insertion can be completed with a higher probability from a shorter eviction path.

[0083] Table 1

[0084]

[0085] 2.2. Label Threshold

[0086] Based on the above cuckoo filter modeling, this embodiment will insert The total number of insertion or removal operations required to perform the process of elements is recorded as , indicating that CF can be Step insert or kick operation is successfully stored elements. In a directed graph , express Step after vertex Label, The corresponding unfilled bucket set and edge set at step are , .make Represents from the vertex arrive The shortest distance, that is .for For any vertex in , we have:

[0087]

[0088] Therefore, the algorithm in this embodiment can obtain the following lemma:

[0089] Lemma 1:

[0090] Proof: Each time a kick operation is performed, from a set The vertex with the smallest label is selected from the vertices to kick out an element, thereby changing the edge connections in the graph. vertices that were previously pointed to by the selected vertex will now point to it. Furthermore, the new label of the selected vertex will not exceed the label of the newly pointed vertex. Obviously, the most extreme case occurs when the number of times a vertex is kicked out reaches times, this means that at most one of its adjacent vertices has a label of 0. By induction, we can prove that the label of a vertex does not exceed the value of the labels of its adjacent vertices by more than .

[0091] Obviously, Lemma 1 is valid in the initial step, since all buckets are empty at the beginning and the associated vertex labels are equal to 0. Assume that this condition is valid in step After that, it also holds true. In the step, this embodiment inserts the element The candidate bucket is represented as and . Insert not greater than Tags Therefore, there is .

[0092] Therefore, consider , so that In .according to , it is deduced that:

[0093]

[0094] because The neighbor pointed to remains unchanged, so the equation is This also holds, thus proving Lemma 1.

[0095] Lemma 2: , The tags have the following relationship:

[0096] Proof: Consider the vertex arrive The non-full vertices in Need to traverse vertices, so that By iteratively applying Lemma 1, we can conclude that:

[0097]

[0098] Combined with formula 1 ,get , and finally prove Lemma 2.

[0099] Lemma 2 means that buckets with smaller labels are more likely to achieve element insertion with fewer reallocations. Obviously, the upper bound of the label is determined by the shortest distance from any vertex to the unfilled vertex set. The upper bound of the label of the cuckoo filter can be determined by its underlying graph structure. Specifically, this embodiment analyzes the relationship between elements and their associated slots (further expanded to buckets), which can be represented as a bipartite graph.

[0100] In a bipartite graph, one set of vertices represents the elements to be inserted, while another set of vertices corresponds to the available slots within the filter. The edges connecting these vertices represent the hash functions that assign elements to their potential slots, with each element adjacent to slot, where .

[0101] Assume that the hash function used in the cuckoo filter follows a simple uniform distribution, that is, each element is hashed into any bucket with the same probability. Using the concepts of graph theory, we can gain insight into the insertion process and analyze the upper bound of the labels in the cuckoo filter. This abstraction provides Hall's theorem for this embodiment.

[0102] Theorem 1: (Hall's Theorem) Let represents a bipartite graph, Represents the collection of elements to be stored, Represents a slot set, both exist Relationship. have arrive A perfect match is a perfect match if and only if for every subset ,inequality Established. Among them, yes The set of slots in that are adjacent to at least one element.

[0103] make express The number of buckets included, i.e. In addition, let Indicates that it has been successfully stored Abstract diagram of a cuckoo filter with elements such that ,in Therefore, it can be considered that in matching, the relationship between any subset of elements and the corresponding number of adjacent buckets is:

[0104]

[0105] Then we can get the following lemma:

[0106] Lemma 3: For a graph Any , Make Existence , ,right satisfy .

[0107] This lemma states that most vertices are within a constant bounded distance to the empty vertex set. Since the above equation holds for any vertex set in G and its corresponding bucket set, there exists For any have . It is important to note that the following inequality holds true in any case:

[0108]

[0109] Therefore, there is Similarly, this embodiment can use the vertex peeling method to prove the lemma. Let express All non-full vertices in express All vertices in which It is through The graph after wheel vertex peeling. Since the following equation holds:

[0110]

[0111]

[0112] This embodiment can prove that the above is true after any number of vertex peeling. ,because ,have:

[0113]

[0114] About The sub-vertex stripped graph, i.e. ,have .when Sometimes, there is .for Make , Any vertex in arrive The distance is Inside, and , Lemma 3 is proved.

[0115] make Indicates departure The farthest vertex set If this embodiment can prove Any vertex to The distance from any vertex to The maximum distance can be proved indirectly by the following lemma. This example defines a random variable To indicate the size The expected number of vertices of the edge set, that is, the number of vertices that can be completely covered by these edge sets vertices are included. This shows that The hash result of elements must have at least different buckets. Based on the previous assumption that the hash function is uniform, we have:

[0116]

[0117] Combining Hall's theorem and observation, this embodiment provides the relationship between the number of buckets and successfully stored elements with high probability.

[0118] Lemma 4: In probability Next, for any and , bucket collection Can completely contain the collection of elements Must meet .

[0119] Proof: It should be noted that exist is established because each Elements have corresponding storage locations. Proving the above theorem is equivalent to proving For a given It is highly unlikely. Substituting into the above formula, we can get:

[0120]

[0121] for , meeting the conditions and The expected number of pairs is:

[0122]

[0123] Therefore, this embodiment can be asserted that The probability that satisfy hour, The relationship does not hold, and Lemma 4 is proved.

[0124] For any integer , Represents a set of The maximum distance is The vertex set of Since each vertex can store at most elements, so The set of elements contained in is of times. The size of the fully contained edge set is . This example can be concluded that for any , the probability that the following statement is true is :

[0125]

[0126] Theorem 2: With probability , the maximum label of any vertex is .

[0127] Proof: For , from the above formula we have .Pick Make ,at this time The maximum distance between any two vertices in is .make Indicates from To any subset The shortest distance, , . Then there will be:

[0128]

[0129] Therefore, this embodiment has , which means is bounded, where . Combined with Lemma 3, we can get The farthest vertex The distance is:

[0130]

[0131] Considering Lemma 2, we have Thus, the threshold of the tag is proved, which shows that the maximum value of the tag is related to the number of buckets.

[0132] The above analysis provides a relatively wide range for the vertex label threshold. This example will further narrow this threshold range through experimental methods. The false positive rate of the experimental process is set to Under different threshold settings, the maximum load factor is measured when element insertion exceeds the threshold and the insertion operation is stopped.

[0133] In this embodiment, the parameters are and As shown in Figure 4(a), under the parameter Using LM in CF can be achieved Compared with RW with the kick threshold set to 500, the load increases As shown in Figure 4(b), when When In contrast, RW has only As shown in Figure 4(c), when When the load increases to , and RW is only In addition, when the label threshold is set to The maximum load factor is obtained when . After this point, increasing the threshold of the tag will not have any effect on the maximum load.

[0134] 2.3. Time and space costs

[0135] In the worst case of inserting elements in the cuckoo filter, the time complexity is significantly affected due to repeated kicking. In the algorithm of this embodiment, the total time of kicking is quantified by the label and, expressed as According to Theorem 2, this embodiment can deduce that the upper bound of the kickout in the worst case is When the traditional method sets the threshold to 500, the algorithm of this embodiment shows significantly fewer kickouts compared to RW when the number of inserted elements is the same, which also means that the overall insertion time of this embodiment will be shorter. The results of the performance evaluation also verify this feature.

[0136] The most critical point of the LM strategy is the increase in the space overhead associated with the label in each bucket. Since the underlying structure of the cuckoo filter remains unchanged in the design of this embodiment, only an additional counter space is added, where the maximum value of each counter is . Note that experiments have shown that the threshold can be For simplicity, this example takes . Similar to the original cuckoo filter, the average space cost is:

[0137]

[0138] The average space cost of the cuckoo filter applying the LM strategy is:

[0139]

[0140] Due to the load factor Greater than , which allows the cuckoo filter of the same size to store more elements. In the case of a high false alarm rate requirement, the algorithm of this embodiment can save more space. For example, when and hour, , the above shows that the bits allocated to each element in the CF applying the RW strategy are , the bits allocated to each element of CF using the LM strategy are The method of allocating less than 1 bit to each element in this embodiment improves the insertion time, and when the false alarm rate reaches When , the bits occupied by the tag will not affect the space performance. Obviously, when the false alarm rate is low, the algorithm of this embodiment shows greater advantages.

[0141] 3. Performance evaluation

[0142] In this section, this example applies the local minimum strategy (LM) to the cuckoo filter and systematically compares its performance with the random walk strategy (RW), CostCounter (CC), and MinCounter (MC). In addition, this example also conducts experiments to quantify the impact of various parameters in LM.

[0143] 3.1. Evaluation Setting

[0144] This example uses a randomly generated string as input and uses the FNV hash function to generate a 64-bit unsigned integer for each element. All experiments were conducted on a machine with an Intel Core i7 processor and 16GB DRAM. In addition, the cuckoo filter in the experiment has Buckets, each with 4 slots. Set the false positive rate to , and set the maximum kick threshold of RW and MC to 500. The evaluation indicators include average insertion time, instantaneous insertion throughput, average kick times, and cumulative kick times. This embodiment also studies the impact of different bucket sizes or thresholds on time consumption and space utilization. All experimental results are the average of 100 repeated experiments.

[0145] 3.2 Experimental results

[0146] The overall storage speed is evaluated by the total time required to insert a certain number of elements in a fixed-size filter, as shown in Figure 5(a). The results show that the average insertion time of LM is lower than that of other strategies. At a load factor of 0.95, the strategy of this embodiment reduces the insertion time by 11% compared to RW. Before reaching a load factor of 0.9, the insertion time of CC is close to that of LM. The root cause is that both LM and CC must go through traversal comparison to identify the kicked elements, and the traversal operation masks the reduced insertion time due to less reallocation. However, after reaching a load of 0.96, LM can reduce the insertion time of RW by at least 12%, saving more time compared to CC and MC.

[0147] This embodiment analyzes the instantaneous insertion throughput during the entire storage process, as shown in Figure 5 (b). When the load exceeds 0.94, the insertion throughput of LM increases by more than 100% compared to RW. For loads from 0.8 to 0.95, the strategy of this embodiment increases the insertion throughput by 10% to 154% compared to RW. At the same time, this embodiment observes an increase of 5% to 10% compared to CC and 8% to 92% compared to MC. It should be noted that the label of each fingerprint in CC consumes more storage space than the strategy of this embodiment, so the LM strategy is more advantageous.

[0148] The above phenomenon can be attributed to the fact that, as the load increases, the LM strategy helps insert new elements while minimizing the frequency of kick operations. Figure 5(c) compares the average number of kicks and the cumulative number of kicks under different load factors. The results of the average number of kicks show that LM can significantly reduce the number of reallocation operations compared with RW. It is worth noting that for the cases with loads ranging from 0.8 to 0.95, LM reduces the number of reallocations by 40% to 80% compared with RW. Meanwhile, the average number of kicks of LM is 22% to 45% lower than that of CC and 37% to 75% lower than that of MC. In terms of the cumulative number of kicks, LM has an advantage over other strategies at various load factors. When the load is low, no matter which strategy is adopted, each inserted element can be stored with one or two reallocation operations. When the load reaches 0.95, LM reduces the cumulative number of kicks by 67% compared with RW. It is worth noting that after the load reaches 0.96, the number of reallocations of LM is reduced by more than 73% compared with RW, and by 36% and 66% compared with CC and MC, respectively.

[0149] Table 2 Maximum load factors achieved under different bucket sizes

[0150]

[0151] Figure 6(a) compares the total insertion time consumed by RW and LM with different bucket sizes. Specifically, this embodiment fixes the number of elements to be processed to , and the bucket size are set to 2 and 4 respectively. The results show that no matter how the bucket size changes, the time required for LM to store these elements is always less than RW. In addition, the CDF of the insertion time under different thresholds is shown in Figure 6(b). When the label threshold of LM is set to When , the average insertion time is 10s; and the tag threshold is set to When RW sets the kick-out threshold to 500, the average insertion time is 27s; and when the kick-out threshold is set to 1000, the average insertion time is 43s. This phenomenon shows that while ensuring space utilization, using a smaller threshold can significantly shorten the insertion time, because the elements that cannot be successfully stored will terminate the reallocation process in time, avoiding inefficient cyclic reallocation operations. This embodiment further compares the maximum load capacity of different strategies in Table 2. The results show that the LM strategy optimizes space utilization under different bucket sizes.

[0152] This example analyzes the number of reallocations in the Cuckoo filter and finds that when the load factor is high enough, the insertion throughput will immediately drop due to frequent element reallocation. This example proposes an effective strategy to solve this problem. This strategy uses the implicit meaning of the number of kicks per bucket to represent the minimum kick cost incurred when inserting an element. This example uses graph theory to prove that each bucket has a The kick-out threshold can indicate whether the storage of the element is successful. In addition, this embodiment proves through experiments that the threshold is The experimental results show that when the load factor reaches 0.95, the strategy of this embodiment reduces the insertion delay by 11% and the cumulative number of reallocations by 67%. The insertion throughput increases when From the perspective of space utilization, compared with other strategies, the strategy of this embodiment also improves the maximum load factor.

[0153] In summary, this embodiment has achieved the following contributions:

[0154] (1) This embodiment analyzes the problem of degradation of insertion operation performance in the cuckoo filter, which is caused by unnecessary and insufficient kicking elements of RW.

[0155] (2) This embodiment proposes a new and more effective element redistribution strategy, called the local minimum strategy (LM). It takes the number of buckets to be kicked out as prior knowledge and uses the matching relationship between elements and buckets in graph theory to optimize the storage of elements with as few kicked out paths as possible. This embodiment strategy not only considers the relationship between buckets and between buckets and elements, but also achieves higher insertion throughput under the same conditions.

[0156] (3) This example theoretically proves that the upper boundary of this label is , and experimentally compress the upper boundary to Moreover, this embodiment verifies the effectiveness of the local minimum strategy from three dimensions: time consumption, number of kickouts, and space utilization, through comprehensive experiments.

[0157] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a data query system for accelerating the cuckoo filter based on the local minimum strategy, including: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the steps of the above-mentioned method.

[0158] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0159] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0160] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0161] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] The device provided by an embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0171] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0172] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data query method for accelerating cuckoo filter based on local minimum strategy, characterized in that: The method comprises: The data to be inserted is obtained, and the data to be inserted is used as the first element to be queried and stored in a cuckoo filter in the following manner, wherein the cuckoo filter is a data structure used in databases, caches, and network measurements to support data query; the cuckoo filter includes a plurality of buckets, each bucket includes a plurality of slots, and each slot is used to store a fingerprint of an element; each element corresponds to a plurality of candidate buckets in the cuckoo filter; the fingerprint of each element can only be stored in the corresponding candidate bucket: Assign a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kicks that have occurred in the corresponding bucket; When all candidate buckets for the first element to be inserted are full, select a bucket with the smallest label from all candidate buckets as the first candidate bucket; For each fingerprint stored in the first candidate bucket, calculate the minimum label of the fingerprint among the labels of the remaining candidate buckets except the current candidate bucket, and use the minimum label as the label of the fingerprint; Select the fingerprint with the smallest fingerprint label from the first candidate bucket to kick it out, and insert the fingerprint of the first element; The element corresponding to the kicked-out fingerprint is used as the new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the bucket label reaches the predefined threshold; The method for determining the predefined threshold includes: Modeling cuckoo filters based on directed graphs; Based on the modeling of the cuckoo filter, four lemmas and two theorems are determined; According to the four lemmas and the two theorems, a predefined threshold is determined; Among them, 4 lemmas and 2 theorems include: Lemma 1: ; Lemma 2: , The tags have the following relationship: ; Lemma 3: For a graph Any , Make Existence , ,right satisfy ; m is the number of buckets in the cuckoo filter; Lemma 4: In probability Next, for any and , bucket collection Can completely contain the collection of elements Must meet ; Theorem 1: Let represents a bipartite graph, Represents the collection of elements to be stored, Represents a slot set, both exist Relationship, then have arrive A perfect match is a perfect match if and only if for every subset ,inequality Established, among which, yes The set of slots that are adjacent to at least one element in ; Theorem 2: In probability The maximum label of any vertex is .

2. The data query method based on the local minimum strategy to accelerate the cuckoo filter according to claim 1 is characterized in that: The predefined threshold is , m is the number of buckets in the cuckoo filter.

3. The data query method based on the local minimum strategy to accelerate the cuckoo filter according to claim 1 is characterized in that: Modeling the cuckoo filter based on a directed graph includes: Through the directed graph For each bucket, slots, each element has The cuckoo filter of candidate buckets is modeled; the candidate bucket corresponding to a vertex stores elements, then the out-degree of the corresponding vertex is ; The vertex The set of neighbor vertices of ,for , using directed edges Represents an element Stored in the vertex middle, Is an element Alternative candidate buckets; Insert The total number of insertion or removal operations required to perform the process of elements is recorded as , in a directed graph , express Vertex after step Label, The corresponding unfilled bucket set and edge set at step are , ,make Represents from the vertex arrive The shortest distance, that is ;for For any vertex in , we have: .

4. A data query system based on a local minimum strategy to accelerate the cuckoo filter, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein the processor, when used to run the computer program, performs the following steps: The data to be inserted is obtained, and the data to be inserted is used as the first element to be queried and stored in a cuckoo filter in the following manner, wherein the cuckoo filter is a data structure used in databases, caches, and network measurements to support data query; the cuckoo filter includes a plurality of buckets, each bucket includes a plurality of slots, and each slot is used to store a fingerprint of an element; each element corresponds to a plurality of candidate buckets in the cuckoo filter; the fingerprint of each element can only be stored in the corresponding candidate bucket: Assign a counter as a label to each bucket in the cuckoo filter; the counter is used to record the number of kicks that have occurred in the corresponding bucket; When all candidate buckets for the first element to be inserted are full, select a bucket with the smallest label from all candidate buckets as the first candidate bucket; For each fingerprint stored in the first candidate bucket, calculate the minimum label of the fingerprint among the labels of the remaining candidate buckets except the current candidate bucket, and use the minimum label as the label of the fingerprint; Select the fingerprint with the smallest fingerprint label from the first candidate bucket to kick it out, and insert the fingerprint of the first element; The element corresponding to the kicked-out fingerprint is used as the new element to be inserted, and the above operation is repeated until there is no new element to be inserted or the bucket label reaches the predefined threshold; The method for determining the predefined threshold includes: Modeling cuckoo filters based on directed graphs; Based on the modeling of the cuckoo filter, four lemmas and two theorems are determined; According to the four lemmas and the two theorems, a predefined threshold is determined; Among them, 4 lemmas and 2 theorems include: Lemma 1: ; Lemma 2: , The tags have the following relationship: ; Lemma 3: For a graph Any , Make Existence , ,right satisfy ; m is the number of buckets in the cuckoo filter; Lemma 4: In probability Next, for any and , bucket collection Can completely contain the collection of elements Must meet ; Theorem 1: Let represents a bipartite graph, Represents the collection of elements to be stored, Represents a slot set, both exist Relationship, then have arrive A perfect match is a perfect match if and only if for every subset ,inequality Established, among which, yes The set of slots that are adjacent to at least one element in ; Theorem 2: In probability The maximum label of any vertex is .

5. The data query system based on the local minimum strategy to accelerate the cuckoo filter according to claim 4 is characterized in that: The predefined threshold is , m is the number of buckets in the cuckoo filter.

6. The data query system based on the local minimum strategy to accelerate the cuckoo filter according to claim 4 is characterized in that: Modeling the cuckoo filter based on a directed graph includes: Through the directed graph For each bucket, slots, each element has The cuckoo filter of candidate buckets is modeled; the candidate bucket corresponding to a vertex stores elements, then the out-degree of the corresponding vertex is ; The vertex The set of neighbor vertices of ,for , using directed edges Represents an element Stored in the vertex middle, Is an element Alternative candidate buckets; Insert The total number of insertion or removal operations required to perform the process of elements is recorded as , in a directed graph , express Step after vertex Label, The corresponding unfilled bucket set and edge set at step are , ,make Represents from the vertex arrive The shortest distance, that is ;for For any vertex in , we have: .

Images