Cuckoo Filter and Methods for Data Insertion, Query, and Deletion

Abstract

The present invention proposes a bracket filter and methods for data insertion, query, and deletion, and relates to the technical field of computer information presentation and search. The bracket filter is composed of a fingerprint record table and a position flag table. The fingerprint record table is composed of m buckets, each having b slots for storing data fingerprints, and is used for storing the inserted data fingerprints. The position flag table is created for each bucket and consists of m vectors for recording the insertion position information of the data fingerprints in the fingerprint record table. Each data fingerprint corresponds to two candidate buckets, and one slot of the bucket for storing the data fingerprint is selected from them. And only when it is finally stored in the second candidate bucket, the value of the slot position symbol is recorded in the vector corresponding to the second candidate bucket. The present invention solves the technical problem that the calculation efficiency of data fingerprints is higher, the false positive rate of data query is lower, and there are query false positives in the bracket filter.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This invention claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China on June 15, 2023, with the application number 202310712462.5 and the invention title "Bracket Filter and Methods for Inserting, Querying, and Deleting Data". All of its content is incorporated into this invention by reference and is part of this invention and is used for all purposes.

[0002] This invention belongs to the technical field of computer information presentation and retrieval, and particularly relates to bracket filters and methods for inserting, querying, and deleting data.

Background Art

[0003] The description of this part only provides information on the background art related to this invention and does not necessarily constitute prior art.

[0004] An approximate membership query data structure (AMQ) stores a probabilistic representation of a set of keys S in a data zone U in a compact format, supports data insertion and query operations, and some AMQs support data deletion operations. For queries on existing elements in a set, queries on set membership can be efficiently completed. However, for queries on elements outside the set, there is a controllable false positive rate (hereinafter referred to as the false positive rate), that is, when querying an element that does not exist in the set, there is a probability of returning that the element exists in the set. The most significant feature of AMQ is its efficient space efficiency. With an acceptable false positive rate, AMQ can operate on devices with limited memory resources, such as network routers, switches, or IoT devices.

[0005] The Bloom Filter (BF) is a typical example of AMQ. It supports insertion and query operations for a set S of a group of keys. When querying a key that exists in the set S, the Bloom Filter can complete the query quickly. However, when querying a key outside the set S, due to its probabilistic structure, the probability of returning "does not exist" after querying is at least 1 - ε. Here, [Number] where n is the number of elements already added, k is the number of hash functions used, m represents the length of the Bloom Filter, indicating that BF provides a controllable false positive rate ε, and a trade-off mechanism between space efficiency and query accuracy is provided. That is, the length of the Bloom Filter directly affects the false positive rate. The longer the Bloom Filter, the higher its false positive rate. Also, the number of hash functions also requires a trade-off. The more the number, the lower the efficiency of the Bloom Filter, but if it is too small, the false positive rate will be high. In recent years, BF has been widely applied to packet classification, inspection of the effective payload of deep packet inspection (DPI), reduction of magnetic disk I / O, avoidance of cache penetration in databases, and issues such as distributed connection and Huff connection, indexing, auxiliary metadata, and query processing in data services on mobile terminals and IoT devices. Its main application advantage is that under the premise of limited space, it can store and access many classification rules in a very compact form in dedicated hardware such as FPGAs. In the past decade, when the hardware storage space was limited or high latency occurred due to frequent access to external memory, the Bloom Filter has become a common solution.

[0006] Compared with a general hash table or binary tree, the main advantage of BF is that its size is constant and the query and insertion efficiency is constant regardless of the number of elements in the structure. The main disadvantage of BF is that it does not support data deletion operations. The Counting Bloom Filter (CBF) solves the problem that BF does not support data deletion, but it requires three to four times more space than BF to maintain the same false positive rate as BF. Also, once the required storage space becomes larger than the RAM, the performance of the filter will decrease significantly. This is because BF uses random read and write and cannot be effectively extended to external memory, such as flash memory. Although the current false positive rate of the filter has already decreased to a very low level, there is still a probability of query false alarms.

[0007] Recently, some scholars have proposed a BF with a False Positive Free Zone (FPFZ). By using the mapping of elements to positions with special attributes in the filter, when the number of elements inserted into the filter is less than a predetermined threshold, the false positive rate within a predetermined zone can be completely removed. However, in any case, the FPFZ is very small, and the supported zone and data volume are limited. Therefore, the applicable range of this technology is greatly restricted.

[0008] Compared with the BF, the Cuckoo Filter (CF) supports dynamic deletion of data. In terms of space efficiency, CF calculates and stores the fingerprint of the initial set data rather than the initial data using one hash function, which guarantees a low false positive rate and smaller occupied space. In terms of time efficiency, CF calculates the element insertion position by cuckoo hashing, but there is a probability that data needs to be reset due to the existence of hash collisions during the element insertion process. Through a large amount of research work, the insertion performance and query performance of the cuckoo filter are further optimized, and the occurrence probability of resetting during the element insertion process and the memory occupancy of the filter itself are reduced. Due to the structure of the cuckoo filter, there is a contradictory relationship between the false positive rate and the space efficiency itself, which is a trade-off. Therefore, for the optimization of the cuckoo filter, it is necessary to comprehensively consider the false positive rate of its structure and the required memory space.

[0009] Currently, many experts and scholars are improving the structure and algorithm of CF for different application scenarios. In terms of balancing the storage space and false positive rate of the cuckoo filter, according to what has been grasped, there is no work to remove the false positive rate of the cuckoo filter, and there are serious efficiency problems in the variants of the cuckoo filter proposed in most works. Therefore, it is still necessary to further study and optimize the structure of CF.

Summary of the Invention

[0010] To overcome the deficiencies of the above prior art, the present invention provides a cuckoo filter and methods for inserting, querying, and deleting data. The configured cuckoo filter is composed of a Fingerprint Record Table (FRT) for storing the fingerprints of the inserted data and a Position Flag Table (PFT) for recording the insertion position information of the data fingerprints in the fingerprint table, thereby solving the technical problem that there are query false positives in the cuckoo filter.

[0011] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions.

[0012] In a first aspect of the present invention, a cuckoo filter is provided.

[0013] It is composed of a fingerprint record table and a position flag table. The fingerprint record table is composed of m buckets, each having b slots for storing data fingerprints, and is for storing the inserted data fingerprints. The position flag table is created for each bucket and consists of m vectors for recording the insertion position information of the data fingerprints in the fingerprint record table. Each data fingerprint corresponds to two candidate buckets, and one slot of the bucket for storing the data fingerprint is selected from them. And only when it is finally stored in the second candidate bucket, the value of the slot position symbol is recorded in the vector corresponding to the second candidate bucket. It is a cuckoo filter.

[0014] Furthermore, the number m of the buckets is a power of 2.

[0015] Furthermore, the data fingerprint is calculated as follows. Obtain the data to be inserted. Calculate the digest data of the data to be inserted by a hash function and a modulo operation. dig x =h a (x)mod 2 n In the formula, n is the length of the digest data, and h a (·) represents a hash function with a fixed-length output. Based on the data to be inserted and the digest data, the lower part of the digest data is directly used as the data fingerprint of the data to be inserted.

[0016] Furthermore, the two candidate buckets are specifically [Number] calculated by wherein [Number] represents the second candidate bucket, x represents the data to be inserted, dig x represents the digest data, f x represents the data fingerprint, l represents the length of the fingerprint, h b (·) represents a hash function with a fixed-length output, and m represents the number of buckets.

[0017] In a second aspect of the present invention, a data insertion method is provided.

[0018] A data insertion method based on the cuckoo filter provided in the first aspect, when inserting data x, first, the digest data dig x corresponding to x, the data fingerprint f x and two candidate buckets [Number] are calculated, and then the insertion is performed as follows: Select an empty slot e j in one bucket from the two candidate buckets to store the data fingerprint. If there are no empty slots in both of the two candidate buckets, select one slot e j and evict the existing data fingerprint f j ' in slot e x by the eviction process, and store the data fingerprint f x in the emptied slot e j . When the data fingerprint f x is finally stored in the second candidate bucket, slot e jStore the value j of the position symbol in the vector corresponding to the bucket [Number] This is a data insertion method.

[0019] Furthermore, selecting one empty slot e of one bucket from the two candidate buckets to store the data fingerprint is specifically divided into the following two cases: j Specifically, it is divided into the following two cases: (1) When there are empty slots in both of the two candidate buckets, randomly select one bucket and store the data fingerprint in any empty slot e j , j ∈ [0, b), where j is the value of the position symbol of slot e j and (2) When there is an empty slot in only one of the two candidate buckets, store the data fingerprint in any empty slot e of that bucket j , j ∈ [0, b), where j is the value of the position symbol of slot e j and

[0020] Furthermore, when there are no empty slots in both of the two candidate buckets, the eviction process is specifically performed as follows: Randomly select one bucket from the two candidate buckets, and randomly evict the data fingerprint f j ' at any slot e, j ∈ [0, b) in the bucket, store f x in e x and update the value of the position symbol of the bucket j [Number] and For the evicted element f x ', calculate the dual position of the current insertion position. If there is an empty slot in the bucket, f x ​Insert ' into the empty slot, update the value of the position symbol of the corresponding vector, and if there is no empty slot, repeat the eviction process until all items are stored in the filter.

[0021] In a third aspect of the present invention, a data query method is provided.

[0022] A data query method based on the cuckoo filter provided in the first aspect, when querying data y, first, the digest data dig corresponding to y x , the data fingerprint f y and two candidate buckets

Number

Number

Number

Number

Number

[0023] In a fourth aspect of the present invention, a data deletion method is provided.

[0024] A data deletion method based on the cuckoo filter provided in the first aspect, when deleting data Z, query the data fingerprint of the data Z to be deleted, and perform subsequent operations in two cases. If the query is successful, delete the data fingerprint from the corresponding position in the fingerprint record table, and delete the position flag in the corresponding vector, and return deletion success. If the query fails, it indicates that the element does not exist in the filter, and return deletion failure. This is the data deletion method.

[0025] Furthermore, the deletion operation in the case of successful query is specifically performed as follows. If the data fingerprint to be deleted exists in the first candidate bucket, delete the data fingerprint from the first candidate bucket. If the data fingerprint to be deleted exists in the second candidate bucket, delete the data fingerprint from the second candidate bucket and delete the position flag in the vector corresponding to the second candidate bucket.

[0026] One or more of the above technical solutions have the following beneficial effects.

[0027] The present invention has higher data fingerprint calculation efficiency. In the data insertion stage, when obtaining the data fingerprint, by directly taking the lower part of the digest data, the hash operation is reduced and the data insertion delay is decreased.

[0028] The present invention has a lower false positive rate for data queries. By a method of recording the value of the position symbol only when stored in the second candidate bucket, the symbols of the array are skillfully incorporated into the data fingerprint, significantly increasing the fingerprint matching length without increasing the length of the fingerprint, reducing the false positive rate of the cuckoo filter by m times (m is the number of buckets in the fingerprint recording table), and when the length of the fingerprint is 8 bits, in actual operation, the probability that the false positive rate is 0 is on average 93.1%. When the length of the fingerprint is 12 bits, compared with 0.04% of CF, in this filter, the probability that the false positive rate is 0 is 99.3%.

[0029] The present invention has a smaller space cost at the same false positive rate. On the premise that the false positive rate is 0, the length of the fingerprint required for this filter is only 8.76993 bits, while CF requires 34.3597 bits.

[0030] Some of the advantages of further aspects of the present invention are shown in the following description, some are revealed from the following description, or are grasped by the implementation of the present invention.

[0031] The drawings in the specification constituting a part of the present invention are for further understanding of the present invention, and the exemplary embodiments and their descriptions of the present invention are for interpreting the present invention and do not unduly limit the present invention.

Brief Description of the Drawings

[0032]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0033] It should be noted that all of the following detailed descriptions are exemplary and for the purpose of further explaining the present application. Unless otherwise specified, all technical terms and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art.

[0034] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. For example, unless otherwise specified in the context, the singular forms used in this specification are intended to include the plural forms, and when the terms "comprising" and / or "including" are used in this specification, it is also understood that the presence of features, steps, operations, devices, assemblies and / or combinations thereof is indicated.

[0035] Embodiment 1 In one or more embodiments, a cuckoo filter based on a data fingerprint position flag is disclosed. As shown in FIG. 1 which is a structural schematic diagram of the cuckoo filter, it is composed of a fingerprint record table (Fingerprint Record Table, FRT) and a position flag table (Position Flag Table, PFT). The fingerprint record table FCT is for storing the fingerprints of the inserted data, and the position flag table PFT is for recording the insertion position information of the data fingerprints in the fingerprint table.

[0036] (1) Fingerprint Record Table As shown in FIG. 2 which is a structural schematic diagram of a fingerprint recording table, the fingerprint recording table FCT is composed of m arrays (hereinafter referred to as buckets), where m must be a power of 2. The merit of this requirement is that when performing an exclusive OR operation, it can be guaranteed that the calculated symbol is always in the array. Each bucket has b storage units e (hereinafter referred to as slots) that can store the data fingerprint f.

[0037] Before inserting, querying, or deleting data for the cuckoo filter, it is necessary to calculate the digest data, data fingerprint, and two candidate buckets corresponding to all data. Taking the data x to be inserted as an example, first, calculate the digest data of the data x to be inserted as dig x =h a (x) mod 2 n (1) by where n is the length of the digest data, and h a (x) is a hash function that outputs a binary string of a specific length.

[0038] Then, based on the data to be inserted and the digest data, the lower part of the digest data is directly used as the data fingerprint of the data to be inserted. The formula is f x =dig x mod 2 l (2) as shown, where l is the length of the fingerprint. By directly using the lower part of the digest data as the data fingerprint of the data to be inserted, the hash operation is reduced, the data insertion delay is decreased, and the data fingerprint calculation efficiency is higher.

[0039] Finally, determine the positions of two different candidate buckets of the data x to be inserted as follows:

Equation

[0040] (2) Position flag table PFT The position flag table PFT in this embodiment does not record the storage position information of the data fingerprint in the fingerprint recording table FCT by means of the normal matrix F consisting of 0 and 1. Instead, by the method of recording the value of the position symbol only when it is stored in the second candidate bucket, the fingerprint matching length is significantly increased without increasing the length of the fingerprint, and the false positive rate of the data query is reduced. m×b In the normal concept, 1 bit is used to indicate the position of the bucket where each data fingerprint is inserted, 0 and 1 indicate the p1 position and the p2 position respectively, and the position flag table PFT is the matrix F consisting of 0 and 1

[0041] which occupies b bits per row, and mb bits of memory space are required to store the position flag table PFT. Since the hash function may collide, the load of the filter cannot reach 100%. Therefore, in the fingerprint recording table FRT, there are a very large number of empty slots. Also, since F is a zero matrix at initialization, bit 0 further indicates that the corresponding position in the fingerprint recording table FRT is empty, which means that a very large number of invalid position information is stored in F. m×b

[0042] By observing and analyzing the data insertion characteristics of the conventional cuckoo filter, it is found that the data fingerprints are mostly inserted into their p1 positions. Figure 3 shows the occupancy rate of 1 in all elements of the matrix F when the number of slots b = 2, 4, 8 and the number of buckets m = 2 15 20 , 2 25 , 2 25 which means that the number of 0s in the matrix F is much larger than 1, and in the extreme case, the matrix F approximates a sparse matrix.

[0043] Based on the above conclusions, only the symbols in the bucket of the data fingerprint inserted at the second candidate bucket p2 position in each bucket are stored, and the position flag table PFT is constructed using m vectors. Each vector stores the position symbol of the data fingerprint inserted at the p2 position in the corresponding bucket of the fingerprint record table FRT. The size of each symbol is logb bits. FIG. 4 is a structural schematic diagram of the position flag table.

[0044] Depending on the position information of the position flag table PFT, during the data query process, if a matching fingerprint is found in the fingerprint record table FRT, as shown in FIG. 4, it is further necessary to confirm whether the two fingerprints are from the same bucket position in the corresponding vector of the position flag table PFT, that is, whether the symbols in the buckets of these two fingerprints can be queried in the corresponding vector. If the fingerprint of the query target data y is hashed to bucket[i + 1] and bucket[i + 1] is the p2 position of y, assuming that the fingerprint of x in the position flag table FRT matches y, and at this time, if the position symbol in the bucket of x is found in vector[i + 1], it indicates that the current bucket is also the p2 position of x, which means that the entire digest data of x and y is the same. Therefore, a query success is returned.

[0045] Embodiment 2 In one or more embodiments, a cuckoo filter-based data insertion method is disclosed. When adopting the cuckoo filter based on the data fingerprint position flag provided in Embodiment 1 and inserting the data x, first, the corresponding digest data dig x of x is calculated, and then the fingerprint f x of x and two candidate buckets

Number

[0046] (1) For both of the two candidate buckets

Number

Number

Number

[0047] (2) If there is only one empty slot left in one of the two candidate buckets

Number

Number

[0048] in it.

Number

Number

Number

Number

Number

[0049] For the evicted element, calculate the dual position of the current insertion position

Number

[0050]

Table 1

[0051] Example 3 In one or more embodiments, a cuckoo filter-based data query method is disclosed. The cuckoo filter based on the data fingerprint position flag provided in Embodiment 1 is adopted, and the data query procedure is as shown in FIG. 6, and the query algorithm is as shown in Table 2. When querying data y, first, calculate the digest data dig x of y and the data fingerprint f y , and then calculate the two insertion candidate bucket positions

Number

Number

Number

[0052] bucket

Number

Number

[0053]

Table 2

[0054] Example 4 In one or more embodiments, a data deletion method based on a cuckoo filter is disclosed. The cuckoo filter based on the data fingerprint position flag provided in Example 1 is adopted, and the deletion procedure is as shown in FIG. 7, and the deletion algorithm is as shown in Table 3. When deleting data Z, first, the query method provided in Example 3 is used to query the element Z to be deleted in the filter, and the following corresponding operations are performed based on the query result.

[0055] (1) If a query success is returned, the fingerprint is deleted from the corresponding position in the FRT, and the position flag in the corresponding vector of the PFT is deleted, and a deletion success is returned. Specifically, it is performed as follows. If the data fingerprint to be deleted exists in the first candidate bucket, the data fingerprint is deleted from the first candidate bucket. If the data fingerprint to be deleted exists in the second candidate bucket, the data fingerprint is deleted from the second candidate bucket, and the position flag in the vector corresponding to the second candidate bucket is deleted.

[0056] (2) If the query fails, it indicates that the element does not exist in the filter, and a deletion failure is returned.

[0057] [Table 3]

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention. For those skilled in the art, various modifications and changes are possible to the present invention. Modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall all be included within the protection scope of the present invention.

Claims

1. A cuckoo filter based on data fingerprint position flags, which is composed of a fingerprint record table and a position flag table, The fingerprint record table is composed of m buckets, each having b slots for storing data fingerprints, and is for storing the inserted data fingerprints, The position flag table is created for each bucket and consists of m vectors for recording the insertion position information of the data fingerprints in the fingerprint record table, Each data fingerprint corresponds to two candidate buckets, and the two candidate buckets are specifically, 【30 numbers】 calculated by, In the formula, 【Number 31】 represents the first candidate bucket, 【Number 32】 represents the second candidate bucket, x represents the data to be inserted, and dig ｘ represents the digest data, and f ｘ represents the data fingerprint, l represents the length of the fingerprint, and h ｂ () represents a hash function with a fixed-length output, m represents the number of buckets, and only when a slot of a bucket for storing the data fingerprint is selected from them and finally stored in the second candidate bucket, the value of the position symbol of the slot is recorded in the vector corresponding to the second candidate bucket. A cuckoo filter based on a data fingerprint position flag, characterized in that.

2. The cuckoo filter based on data fingerprint position flags according to Claim 1, wherein the number of buckets m is a power of 2.

3. The data fingerprint is calculated as follows, Obtain the data to be inserted, Calculate the digest data of the data to be inserted by a hash function and a modulo operation, dig ｘ = h ａ (x) mod 2 ｎ where n is the length of the digest data, and h ａ (·) represents a hash function with a fixed-length output, Based on the data to be inserted and the digest data, the lower part of the digest data is directly used as the data fingerprint of the data to be inserted. The cuckoo filter based on data fingerprint position flags according to Claim 1.

4. A data insertion method based on a cuckoo filter based on a data fingerprint position flag according to any one of claims 1 to 3. When inserting data x, first, the digest data dig corresponding to x ｘ , the data fingerprint f ｘ and two candidate buckets 【Number 33】 Calculate, and perform the insertion as follows, Select one empty slot e of one bucket from two candidate buckets and store the data fingerprint. If there are no empty slots in both of the two candidate buckets, select one slot e ｊ and evict the existing data fingerprint f ｊ in slot e ｊ by the eviction process, and store the data fingerprint f ｘ ' in the emptied slot e ｘ and ｊ store it in the emptied slot e Data fingerprint f ｘ If ’ is finally stored in the second candidate bucket, the position symbol value j of slot e ｊ in the bucket 【Number 34】 Store it in the vector corresponding to. A data insertion method based on a cuckoo filter, characterized in that.

5. One empty slot e of one bucket is selected from the two candidate buckets described above to store the data fingerprint. ｊ Specifically, selecting one empty slot e of one bucket from the two candidate buckets described above to store the data fingerprint can be divided into the following two cases: (1) If there are empty slots in both of the two candidate buckets, randomly select one bucket and store the data fingerprint in any empty slot e ｊ , j ∈ [0, b), where j is the value of the position symbol of slot e ｊ and If there is an empty slot in only one of the two candidate buckets, store the data fingerprint in any empty slot e of the bucket ｊ , where j ∈ [0, b), and j is the value of the position symbol of slot e ｊ A data insertion method based on the cuckoo filter according to claim 4, characterized in that.

6. When there are no empty slots in both of the two candidate buckets, the eviction process is specifically performed as follows, Randomly select one bucket from two candidate buckets, and eject the data fingerprint f ｊ ' at any slot e ｘ , j ∈ [0, b) randomly, and store f ｘ in e ｊ and store it in the bucket 【Number 35】 Update the value of the position symbol of, Evicted element f ｘ For ’, calculate the dual position of the current insertion position. If there is an empty slot in the bucket, f ｘ ’ is inserted into the empty slot, the value of the position symbol of the corresponding vector is updated. If there is no empty slot, the eviction process is repeated until all items are stored in the filter. A data insertion method based on the cuckoo filter according to claim 4, characterized in that.

7. A data query method based on a cuckoo filter based on a data fingerprint position flag according to any one of claims 1 to 3. When querying data y, first, the digest data dig corresponding to y ｘ , data fingerprint f ｙ and two candidate buckets 【No. 36】 Calculate it, and then, as follows, the data fingerprint f ｙ is compared with all fingerprints in these two buckets, Query j in the first candidate bucket, if it does not exist, return query success, otherwise, return query failure, 【No. 37】 slot e ｊ If the fingerprint stored at j ∈ [0, b) matches f ｙ then 【Number 38】 Query k in the second candidate bucket, if it exists, return query success, otherwise, return query failure. A data query method based on a cuckoo filter, characterized in that. The second candidate bucket 【Number 39】 slot e ｋ If the fingerprint stored in k ∈ [0, b) is f ｙ matches f, 【Number 40】 Query k in, if it exists, return query success, otherwise, return query failure. A data query method based on a cuckoo filter, characterized in that.

8. A data deletion method based on the cuckoo filter based on the data fingerprint position flag according to any one of claims 1 to 3, wherein when deleting data Z, query the data fingerprint of the data Z to be deleted, and perform subsequent operations in two cases: If the query is successful, delete the data fingerprint from the corresponding position in the fingerprint record table, delete the position flag in the corresponding vector, and return a successful deletion. If the query fails, it indicates that the data Z to be deleted does not exist in the filter, and the method returns a deletion failure, which is a data deletion method based on the cuckoo filter.

9. The deletion operation in the case of successful query is specifically performed as follows: If the data fingerprint to be deleted exists in the first candidate bucket, delete the data fingerprint from the first candidate bucket. If the data fingerprint to be deleted exists in the second candidate bucket, delete the data fingerprint from the second candidate bucket and delete the position flag in the vector corresponding to the second candidate bucket, which is a data deletion method based on the cuckoo filter according to claim 8.

Images