Cuckoo filter program and how to insert, query, and delete data

Abstract

The present invention proposes a bracket filter and methods for data insertion, query, and deletion, and relates to the technical field of computer information presentation and search. The bracket filter is composed of a fingerprint record table and a position flag table. The fingerprint record table is composed of m buckets, each having b slots for storing data fingerprints, and is used for storing the inserted data fingerprints. The position flag table is created for each bucket and consists of m vectors for recording the insertion position information of the data fingerprints in the fingerprint record table. Each data fingerprint corresponds to two candidate buckets, and one slot of the bucket for storing the data fingerprint is selected from them. And only when it is finally stored in the second candidate bucket, the value of the slot position symbol is recorded in the vector corresponding to the second candidate bucket. The present invention solves the technical problem that the calculation efficiency of data fingerprints is higher, the false positive rate of data query is lower, and there are query false positives in the bracket filter.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This invention claims priority to a Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on June 15, 2023, bearing application number 202310712462.5 and entitled "Cuckoo filter and method for inserting, querying and deleting data," the entire contents of which are incorporated herein by reference and made a part of the present invention for all purposes.

[0002] The present invention is in the field of computer information presentation and retrieval, and more particularly relates to Cuckoo filters and methods for inserting, querying, and deleting data. [Background technology]

[0003] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0004] An approximate membership query data structure (AMQ) stores a probabilistic representation of a key set S in a data zone U in a compact format. It supports data insertion and query operations, and some AMQs also support data deletion operations. It can efficiently complete queries for set membership for queries on elements existing in a set, but has a controllable false positive rate for queries on elements outside the set. That is, when querying an element that is not in the set, there is a probability that the element is in the set. The most significant feature of AMQ is its efficient space efficiency, and with an acceptable false positive rate, AMQ can operate on devices with limited memory resources, such as network routers, switches, or IoT equipment.

[0005] A Bloom Filter (BF) is a typical example of AMQ. It supports insert and query operations on a set S of keys. When querying for keys that exist in S, the Bloom Filter can quickly complete the query. However, when querying for keys outside S, the Bloom Filter has a probabilistic structure, so the probability of the query returning "does not exist" is at least 1-ε, where:

number

[0006] Compared to common hash tables or binary trees, the main advantage of BF is that it has a constant size and consistent query and insertion efficiency regardless of the number of elements in the structure. The main disadvantage of BF is that it does not support data deletion operations. Counting Bloom filters (CBFs) solve the problem of BF not supporting data deletion, but they require three to four times more space than BF to maintain the same false positive rate. Furthermore, once the required storage space exceeds RAM, the filter's performance significantly degrades. This is because BFs use random reads and writes and cannot be effectively expanded to external memory, such as flash memory. Although the false positive rate of current filters has already been reduced to a very low level, there is still a possibility of query false alarms.

[0007] Recently, some scholars have proposed BFs with false positive free zones (FPFZs). By using a mapping of elements to positions with special attributes in the filter, they can completely eliminate the false positive rate within a given zone when the number of elements inserted into the filter is less than a certain threshold. However, the FPFZ is very small in any case, and the supported zones and data volume are also limited, which significantly limits the scope of application of this technique.

[0008] Compared to BF, Cuckoo Filter (CF) supports dynamic data deletion. In terms of space efficiency, CF uses a single hash function to calculate and store fingerprints of the initial set of data, rather than the initial data, ensuring a low false positive rate and occupying less space. In terms of time efficiency, CF calculates element insertion locations using Cuckoo hashing, but there is a possibility that data resetting will be required during the element insertion process due to hash collisions. Extensive research has further optimized the insertion and query performance of Cuckoo Filter, reducing the probability of resetting during the element insertion process and the memory usage of the filter itself. Due to the structure of the Cuckoo Filter, there is a trade-off between the false positive rate and space efficiency. Therefore, optimizing the Cuckoo Filter requires comprehensive consideration of the false positive rate of the structure and the required memory space.

[0009] At present, many experts and scholars have improved the structure and algorithm of CF for different application scenarios. However, in terms of balancing the storage space and false positive rate of the Cuckoo filter, it has been found that there is no work to eliminate the false positive rate of the Cuckoo filter. In addition, the Cuckoo filter variants proposed in most works have serious efficiency problems. Therefore, there is still a need for further research and optimization of the CF structure. Summary of the Invention

[0010] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a Cuckoo filter and a method for inserting, querying, and deleting data. The constructed Cuckoo filter comprises a Fingerprint Record Table (FRT) for storing fingerprints of inserted data, and a Position Flag Table (PFT) for recording insertion position information of the data fingerprint in the Fingerprint Table, thereby solving the technical problem of the existence of query false positives in the Cuckoo filter.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] In a first aspect of the present invention, a Cuckoo filter is provided.

[0013] It is composed of a fingerprint record table and a location flag table, the fingerprint record table is configured with m buckets, each having b slots for storing data fingerprints, for storing inserted data fingerprints; the position flag table is created for each bucket and includes m vectors for recording insertion position information of a data fingerprint in a fingerprint recording table; Each data fingerprint corresponds to two candidate buckets, and one bucket slot is selected from among them to store the data fingerprint. Only if the data fingerprint is finally stored in the second candidate bucket, is the Cuckoo filter recording the value of the slot's position symbol in the vector corresponding to the second candidate bucket.

[0014] Furthermore, the number of buckets m is an exponential power of two.

[0015] Furthermore, the data fingerprint is calculated by: Get the data to be inserted, Calculate digest data of the data to be inserted using a hash function and a modulo operation; dig x =h a (x)mod 2 n where n is the length of the digest data and h a (·) represents a hash function with a fixed output length, Based on the data to be inserted and the digest data, a lower portion of the digest data is used as a data fingerprint of the data to be inserted directly.

[0016] Furthermore, the two candidate buckets are specifically:

number

number

[0017] In a second aspect of the present invention, a data insertion method is provided.

[0018] In a first aspect of the present invention, there is provided a data insertion method based on a Cuckoo filter, in which when data x is to be inserted, digest data dig corresponding to x is first obtained. x , data fingerprint f x and two candidate buckets

number

number

[0019] Furthermore, one empty slot e in one of the two candidate buckets is selected. j Selecting and storing data fingerprints can be divided into the following two cases: (1) If both of the two candidate buckets have free slots, randomly select one bucket and store the data fingerprint in one of the free slots e j ,j∈[0,b), where j is the slot e j is the value of the position symbol of (2) If only one of the two candidate buckets has an empty slot, the data fingerprint is stored in one of the empty slots e j ,j∈[0,b), where j is the slot e j is the value of the position symbol.

[0020] Furthermore, if both of the two candidate buckets already have no free slots, the eviction process is specifically carried out as follows: Randomly select one bucket from the two candidate buckets and select slot e of one of the buckets. j , j∈[0,b) x ' is randomly kicked out, f x e j Store in a bucket

number

[0021] In a third aspect of the present invention, a data query method is provided.

[0022] A data query method based on a Cuckoo filter is provided in a first aspect, wherein when querying data y, first, digest data dig corresponding to y is obtained. x , data fingerprint f y and two candidate buckets

number

number

number

number

number

[0023] In a fourth aspect of the present invention, there is provided a data deletion method.

[0024] A data deletion method based on a Cuckoo filter provided in a first aspect, wherein when deleting data Z, the data fingerprint of the data Z to be deleted is queried, and subsequent operations are performed in two cases; If the query is successful, remove the data fingerprint from the corresponding position in the fingerprint record table and remove the position flag in the corresponding vector, and return a successful removal; If the query fails, it indicates that the element does not exist in the filter and returns a deletion failure. This is a data deletion method.

[0025] Furthermore, the deletion operation when the query is successful is specifically performed as follows: If the data fingerprint to be deleted is present in the first candidate bucket, delete the data fingerprint from the first candidate bucket; If the data fingerprint to be deleted is present in the second candidate bucket, the data fingerprint is deleted from the second candidate bucket, and the position flag in the vector corresponding to the second candidate bucket is deleted.

[0026] The above one or more technical solutions have the following beneficial effects:

[0027] The present invention has higher data fingerprint calculation efficiency. In the data insertion stage, when obtaining the data fingerprint, the lower part of the digest data is directly taken, thereby reducing the hash operation and lowering the data insertion delay.

[0028] The present invention has a lower false positive rate for data queries. By cleverly incorporating sequence symbols into the data fingerprint by recording the value of the location symbol only if it is stored in the second candidate bucket, the fingerprint matching length is significantly increased without increasing the fingerprint length, reducing the false positive rate of the Cuckoo filter by m times (m is the number of buckets in the fingerprint recording table). When the fingerprint length is 8 bits, in actual operation, the false positive rate of this filter is 93.1% on average. When the fingerprint length is 12 bits, the false positive rate of this filter is 99.3%, compared to CF's 0.04%.

[0029] Our invention has a smaller space cost at the same false positive rate. Under the assumption of a false positive rate of 0, our filter requires a fingerprint length of only 8.76993 bits, while CF requires 34.3597 bits.

[0030] Advantages of additional aspects of the invention will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the invention.

[0031] The drawings in the specification that form a part of this invention are intended to provide a further understanding of the invention, and the illustrative embodiments of the invention and their descriptions are intended to interpret the invention and are not intended to unduly limit the invention. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a structural schematic diagram of a Cuckoo filter according to a first embodiment. [Figure 2] FIG. 2 is a structural schematic diagram of a fingerprint record table in the first embodiment. [Figure 3] 10 is a diagram showing the occupancy rate of 1 in all elements of matrix F when the number of slots b=2, 4, 8 and the number of buckets m=215, 220, 225 in the first embodiment. FIG. [Figure 4]FIG. 4 is a structural schematic diagram of a position flag table in the first embodiment. [Figure 5] 10 is a flowchart of the method of Example 2. [Figure 6] 10 is a flowchart of the method of Example 3. [Figure 7] 10 is a flowchart of the method of Example 4. DETAILED DESCRIPTION OF THE INVENTION

[0033] It should be noted that the following detailed description is merely illustrative and is intended to further explain the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art.

[0034] It should be noted that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the exemplary embodiments according to the present application. For example, unless the context clearly indicates otherwise, the singular forms used herein are intended to include the plural forms, and the use of the terms "comprises" and / or "includes" herein is not to be understood as indicating the presence of features, steps, operations, devices, assemblies, and / or combinations thereof.

[0035] Example 1 In one or more embodiments, a Cuckoo filter based on a data fingerprint position flag is disclosed. As shown in Figure 1, which is a structural diagram of a Cuckoo filter, the Cuckoo filter is composed of a Fingerprint Record Table (FRT) and a Position Flag Table (PFT), where the Fingerprint Record Table FCT is for storing the fingerprint of inserted data, and the Position Flag Table PFT is for recording the insertion position information of the data fingerprint in the Fingerprint Table.

[0036] (1) Fingerprint Record Table As shown in Figure 2, which is a schematic diagram of the structure of the fingerprint record table, the fingerprint record table FCT consists of m arrays (hereinafter referred to as buckets), where m must be an exponential power of 2. The advantage of this requirement is that when performing an exclusive OR operation, it can be guaranteed that the calculated symbol is always in the array, and each bucket has b storage units e (hereinafter referred to as slots) that can store a data fingerprint f.

[0037] Before inserting, querying, or deleting data in the Cuckoo filter, it is necessary to calculate the digest data, data fingerprint, and two candidate buckets corresponding to all the data. For example, take data x to be inserted as follows: dig x =h a (x)mod 2 n (1) Calculated by where n is the length of the digest data and h a (x) is a hash function that outputs a binary string of a specific length.

[0038] Then, based on the insertion target data and the digest data, the lower part of the digest data is directly used as the data fingerprint of the insertion target data, and the formula is: f x =dig x mod 2 l (2) is shown as In the formula, l is the length of the fingerprint. By directly using the lower part of the digest data as the data fingerprint of the data to be inserted, the hash operation is reduced, the data insertion delay is lowered, and the data fingerprint calculation efficiency is higher.

[0039] Finally, determine the locations of two different candidate buckets for the data x to be inserted as follows:

number

[0040] (2) Position flag table PFT The position flag table PFT in this embodiment is a matrix F consisting of normal 0s and 1s. m×b By not recording the storage location information of the data fingerprint in the fingerprint record table FCT by the above method, but by recording the value of the location symbol only when it is stored in the second candidate bucket, the fingerprint matching length is significantly increased without increasing the fingerprint length, and the false positive rate of data queries is reduced.

[0041] In the conventional way, one bit is used to indicate the location of the bucket where each data fingerprint is inserted, with 0 and 1 indicating the p1 and p2 locations, respectively, and the location flag table PFT is a matrix F consisting of 0s and 1s. m×b where each row occupies b bits, and storing the location flag table PFT requires mb bits of memory space. Because hash function collisions may occur, the filter load cannot reach 100%, so there are a large number of free slots in the fingerprint record table FRT. Furthermore, since F is a zero matrix at initialization, a bit 0 also indicates that the corresponding position in the fingerprint record table FRT is free, which means that a large number of invalid location information is stored in F.

[0042] By observing and analyzing the data insertion characteristics of the conventional Cuckoo filter, it was found that data fingerprints are mostly inserted at their p1 position. Figure 3 shows the results for the cases where the number of slots b = 2, 4, 8 and the number of buckets m = 2. 15 ,2 20 ,2 25 This means that the number of 0s in matrix F is much greater than the number of 1s, and in the extreme case, matrix F approximates a sparse matrix.

[0043] Based on the above conclusions, only the symbol in the bucket of the data fingerprint inserted at the second candidate bucket p2 position in each bucket is stored, and a position flag table PFT is constructed using m vectors, each vector stores the position symbol of the data fingerprint inserted at the p2 position in the corresponding bucket of the fingerprint record table FRT, and the size of each symbol is logb bits. Figure 4 is a schematic diagram of the structure of the position flag table.

[0044] Depending on the location information in the Location Flag Table (PFT), if a matching fingerprint is found in the Fingerprint Record Table (FRT) during the data query process, it is necessary to further check whether the two fingerprints are from the same bucket location in the corresponding vector in the Location Flag Table (PFT), that is, whether the bucket symbols of these two fingerprints can be queried in the corresponding vector, as shown in Figure 4. Suppose the fingerprint of the query target data y is hashed to bucket[i+1], and bucket[i+1] is the p2 location of y, and the fingerprint of x in the Location Flag Table (FRT) matches y. In this case, if the location symbol in the bucket of x is found in vector[i+1], it indicates that the current bucket is also the p2 location of x, which means that the entire digest data of x and y are the same, so the query returns successful.

[0045] Example 2 In one or more embodiments, a data insertion method based on a Cuckoo filter is disclosed. The Cuckoo filter based on the data fingerprint location flag provided in the first embodiment is adopted. When inserting data x, first, the digest data dig corresponding to x is x , and calculate the fingerprint f of x. x and two candidate buckets

number

[0046] (1) Two candidate buckets

number

number

number

[0047] (2) One of the two candidate buckets

number

number

[0048] (3) If both of the two candidate buckets already have no free slots,

number

number

number

number

number

[0049] The opposite position of the current insertion position for the evicted element

number

[0050] [Table 1]

[0051] Example 3 In one or more embodiments, a data query method based on a Cuckoo filter is disclosed. The Cuckoo filter based on the data fingerprint location flag provided in the first embodiment is adopted. The data query procedure is as shown in Figure 6, and the query algorithm is as shown in Table 2. When querying data y, first, the digest data dig of y is x and the data fingerprint f y Calculate the two candidate bucket positions for insertion of y.

number

number

number

[0052] bucket

number

number

[0053] [Table 2]

[0054] Example 4 In one or more embodiments, a data deletion method based on a Cuckoo filter is disclosed. The Cuckoo filter based on the data fingerprint location flag provided in Example 1 is adopted, the deletion procedure is as shown in Figure 7, and the deletion algorithm is as shown in Table 3. When deleting data Z, first, query the element Z to be deleted in the filter according to the query method provided in Example 3, and then perform the following corresponding operations based on the query result:

[0055] (1) If the query returns success, delete the fingerprint from the corresponding position in the FRT, and delete the position flag in the corresponding vector of the PFT, and return a deletion success. Specifically, this is done as follows: If the data fingerprint to be deleted is present in the first candidate bucket, delete the data fingerprint from the first candidate bucket; If the data fingerprint to be deleted is present in the second candidate bucket, the data fingerprint is deleted from the second candidate bucket, and the position flag in the vector corresponding to the second candidate bucket is deleted.

[0056] (2) If the query fails, it indicates that the element is not in the filter and returns a deletion failure.

[0057] [Table 3]

[0058] The above description is merely a preferred embodiment of the present invention, and is not intended to limit the present invention. Those skilled in the art can make various modifications and variations to the present invention. All modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A cuckoo filter program based on data fingerprint location flags, comprising a fingerprint record table and a location flag table, the fingerprint record table is configured with m buckets, each having b slots for storing data fingerprints, for storing inserted data fingerprints; the position flag table is created for each bucket and includes m vectors for recording insertion position information of a data fingerprint in a fingerprint recording table; Each data fingerprint corresponds to two candidate buckets, and the two candidate buckets are specifically: [Equation 30] is calculated by During the ceremony, [Equation 31] represents the first candidate bucket, [Equation 32] represents the second candidate bucket, x represents the data to be inserted, and dig ｘ represents the digest data, and f ｘ represents the data fingerprint, l represents the length of the fingerprint, and h ｂ (·) represents a hash function with a fixed output length, m represents the number of buckets, one slot of the bucket is selected to store the data fingerprint, and only if the data fingerprint is finally stored in the second candidate bucket, the value of the position symbol of the slot is recorded in the vector corresponding to the second candidate bucket; When querying data y, first, the digest data dig y corresponding to y, the data fingerprint f y and two candidate buckets are obtained. [Equation 33] and then match the data fingerprint f y against all fingerprints in the two candidate buckets as follows: First candidate bucket [Equation 34] If the fingerprint stored in slot e j , j∈[0, b) of matches f y , then [Equation 35] If j does not exist, the query returns success. If j exists, the query returns failure. Second candidate bucket [Equation 36] If the fingerprint stored in slot e k , kε[0, b) of matches f y , then [Equation 37] A cuckoo filter program based on data fingerprint location flags that causes a computer to query k in and, if it exists, return a query success, and conversely, if it does not exist, return a query failure.

2. 2. The Cuckoo filter program based on data fingerprint location flags according to claim 1, wherein the number of buckets m is an exponential power of two.

3. The data fingerprint is calculated as follows: Get the data to be inserted, Calculate digest data of the data to be inserted using a hash function and a modulo operation; dig ｘ =h ａ (x) mod 2 ｎ where n is the length of the digest data, and h ａ (·) represents a hash function with a fixed output length, 2. The Cuckoo filter program based on the data fingerprint position flag according to claim 1, wherein a lower portion of the digest data is directly used as the data fingerprint of the data to be inserted based on the data to be inserted and the digest data.

4. 4. A data insertion method executed by a computer using a Cuckoo filter program based on a data fingerprint location flag according to claim 1, wherein when data x is inserted, first, digest data dig corresponding to x is extracted. ｘ , data fingerprint f ｘ and two candidate buckets [Number 38] and then perform the insertion as follows: One free slot e in one bucket from two candidate buckets ｊ If there are no free slots in either of the two candidate buckets, one slot e ｊ and then eject it from slot e ｊ The existing data fingerprint f ｘ ' and data fingerprint f ｘ The vacant slot e ｊ Store in Data Fingerprint ｘ ' is finally stored in the second candidate bucket, slot e ｊ The value j of the location symbol of the bucket [0.39] A data insertion method based on a Cuckoo filter program, characterized in that:

5. One empty slot e of one bucket from the two candidate buckets ｊ Selecting and storing the data fingerprint can be divided into the following two cases: (1) If both of the two candidate buckets have free slots, randomly select one bucket and transfer the data fingerprint to one of the free slots e ｊ , j∈[0, b), where j is the slot e ｊ is the value of the position symbol of (2) If only one of the two candidate buckets has an empty slot, the data fingerprint is stored in one of the empty slots e ｊ , j∈[0, b), where j is the slot e ｊ 5. The data insertion method based on the Cuckoo filter program according to claim 4, wherein the position symbol value is:

6. If both of the two candidate buckets already have no free slots, the eviction process is specifically performed as follows: Randomly select one bucket from the two candidate buckets, and select slot e ｊ , j∈[0, b) ｘ ' is randomly expelled, and f ｘ wo e ｊ Store in a bucket [Equation 40] Update the value of the position symbol of The expelled element f ｘ Calculate the opposite position of the current insertion position for ', and if there is an empty slot in the bucket, ｘ 5. The data insertion method based on the Cuckoo filter program of claim 4, characterized in that: ' is inserted into the empty slot, and the value of the position symbol of the corresponding vector is updated; if there are no empty slots, the eviction process is repeated until all items are stored in the filter.

7. A computer implemented data query method using a Cuckoo filter program based on data fingerprint location flags according to any one of claims 1 to 3.

8. 4. A data deletion method executed by a computer using a Cuckoo filter program based on a data fingerprint location flag according to claim 1, wherein when deleting data Z, the data fingerprint of the data Z to be deleted is queried, and subsequent operations are performed in two cases: If the query is successful, remove the data fingerprint from the corresponding position in the fingerprint record table and remove the position flag in the corresponding vector, and return a successful removal; A data deletion method based on a Cuckoo filter program, characterized in that if the query fails, it indicates that the data Z to be deleted does not exist in the filter, and returns a deletion failure.

9. Specifically, the deletion operation in the case of successful query is performed as follows: If the data fingerprint to be deleted is present in the first candidate bucket, delete the data fingerprint from the first candidate bucket; 9. The data deletion method based on the Cuckoo filter program of claim 8, wherein if the data fingerprint to be deleted is present in the second candidate bucket, the data fingerprint is deleted from the second candidate bucket and the position flag in the vector corresponding to the second candidate bucket is deleted.

Images