Data filtering method and device

Abstract

The present invention provides a data filtering method and device, which is applied to a filtering system, wherein the filtering system includes m cells, each cell includes b storage slots, each storage slot includes a first domain and a second domain, the first domain is used to record fingerprint information of the element fingerprint of the data element, and the second domain is used to match the identification identifier corresponding to the fingerprint information stored in the first domain. The data filtering method includes: determining the element fingerprint of the data element to be processed in response to the obtained target instruction; performing a target operation on the data element based on at least the element fingerprint and the storage information of the first domain and the second domain in each storage slot, wherein the target operation includes at least one or more of insertion, query, and deletion operations.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data filtering method and device. Background Art

[0002] Currently, the basic characteristics of big data are extremely large volume and low data value density. These characteristics bring huge challenges to the storage, transmission, processing and query of large-scale data. To this end, a variety of space-efficient data summary structures have been proposed to represent large-scale data sets and support query functions with constant time complexity. Typical data summary structures include Bloom filtering, cuckoo filtering, quotient filtering, Count-Min and their variants. These data summary structures usually use hash functions to map data elements to one or more positions in a hash table or bit vector. Subsequently, information related to the data elements will be recorded by the bits, counters or fingerprints corresponding to these positions. For example, membership query technology based on data summaries (answering the question of whether a given data element belongs to a given data set) has been widely used in caching, routing, privacy protection, data synchronization and other fields.

[0003] Data summary structures may implement diverse functionality in different ways. Bloom filtering uses h independent hash functions to map data elements to h positions in a bit vector initialized to 0. Subsequently, the bits at these positions are set to 1 to represent the membership information of the data element. Subsequent interpolations only require checking these bits. In contrast, cuckoo filtering directly stores data element fingerprints using a hash table consisting of m cells, each with b storage slots. By providing two candidate cells for each data element, cuckoo filtering can achieve high space utilization. In addition to membership queries, more fields are introduced into cuckoo filtering and Bloom filtering (such as counters, checksums, flags, etc.) to improve performance and enable diverse functionality.

[0004] Cuckoo filtering is considered to be more efficient than Bloom filtering. However, the pursuit of space efficiency, accuracy, and high throughput in data summarization has never stopped. For any data element x, when both candidate cells are fully occupied, the cuckoo filter will randomly propose one of the two candidate cells that has stored fingerprints, and put the fingerprint of x into the emptied storage slot. The proposed data element fingerprint ηx will be attempted to be stored in its other candidate cell. The above kick-out and replay operations will continue max times to achieve high space utilization. The candidate cell index value of the data element x can be calculated as follows: h1(x) = hash(x), Each data element fingerprint redistribution in the cuckoo filter requires a hash calculation and an XOR operation. This computational overhead undoubtedly has a negative impact on insertion throughput, especially when the cuckoo filter space is already highly utilized. Therefore, a lightweight data fingerprint redistribution strategy is crucial to further improve the throughput of the cuckoo filter.

[0005] In addition, the length of the cuckoo filter (the number of cells m) must be strictly set to m = 2 c and c>1 is an integer. Otherwise, the XOR operation may overflow. This lack of design flexibility makes the cuckoo filter either in a low-utilization state in actual use or unable to represent a given number of data elements. For example, given b=4, if 520 data elements are to be represented, a cuckoo filter length of 256 will result in a significant waste of space; if the length is set to 128, there will not be enough storage slots to represent this data. Therefore, in order to achieve more reasonable space utilization, a flexible design mechanism is urgently needed in which the storage capacity is proportional to the data volume.

[0006] Finally, complex big data analysis tasks usually require data summaries to support other functions in addition to membership queries. Typical functions include dynamic representation (capacity elasticity), cardinality estimation, frequency estimation, top-k query, etc. However, existing data summaries cannot achieve such diverse functions, and there is a lack of relevant discussion and generalization work. For example, Bloom filter, counting Bloom filter and dynamic Bloom filter can achieve membership query, element insertion and dynamic representation, but they cannot support the above functions at the same time and there is still a lack of data structures that can achieve the above functions at the same time. This problem also exists in cuckoo filter and dynamic cuckoo filter. In other words, studying a data summary structure that takes into account diverse functions at the same time is a practical need to support big data analysis. Summary of the Invention

[0007] An embodiment of the present invention provides a data filtering method, which is applied to a filtering system. The filtering system includes m cells, each of which includes b storage slots, each of which includes a first field and a second field. The first field is used to record fingerprint information of an element fingerprint of a data element, and the second field is used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering method includes:

[0008] In response to the obtained target instruction, determining an element fingerprint of a data element to be processed;

[0009] A target operation is performed on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots, where the target operation includes at least one or more of an insert, query, and delete operation.

[0010] As an optional embodiment, the fingerprint information is a quotient value and a remainder value calculated based on a filter length of the filter system, and the quotient value and remainder value corresponding to the same data element are respectively stored in different candidate cells.

[0011] As an optional embodiment, the target operation is an insert operation;

[0012] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0013] Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length;

[0014] When there is an empty storage slot in the filtering system, directly storing the data element in the empty storage slot, and updating the corresponding first field and second field based on the quotient value and the remainder value; or

[0015] When there is no empty storage slot in the filtering system, randomly kicking out data in the storage slot to store the data element in the empty storage slot;

[0016] Redirecting the kicked-out data element to a new storage slot and storing the kicked-out data element, while dynamically adjusting the values ​​of the corresponding first field and second field, wherein the new storage slot is formed by kicking out the data in another storage slot;

[0017] Repeat the above steps until all data is stored to the maximum extent possible.

[0018] As an optional embodiment, the target operation is a query operation;

[0019] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0020] Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length;

[0021] Based on the quotient value and the remainder value, querying the first domain and the second domain corresponding to the quotient value and the remainder value in the filtering system;

[0022] The candidate cell storing the data element is determined based on the query result.

[0023] As an optional embodiment, the target operation is a delete operation;

[0024] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0025] After the candidate cells storing the data element are determined, the data stored in the candidate cells are deleted.

[0026] As an optional embodiment, the target operation further includes performing filter compression or filter merging on the two isomorphic filter systems;

[0027] The method further comprises:

[0028] determining the storage capacity of the two isomorphic filtering systems;

[0029] The data in the filtering system with a smaller storage capacity is represented in the storage form of the filtering system with a larger storage capacity, and is stored accordingly.

[0030] As an optional embodiment, the target operation further includes a filtering deduplication operation for deduplicating identical storage elements in two isomorphic filtering systems, and a filtering intersection operation for obtaining the data intersection of two complete or partial filtering systems.

[0031] As an optional embodiment, the present invention further includes:

[0032] Pre-determining the future storage capacity of the filtering system;

[0033] The number of storage slots in some or all of the cells is increased based at least on the storage amount.

[0034] As an optional embodiment, each of the storage slots further includes a third field, and the third field is used to record the frequency of occurrence of the corresponding data element;

[0035] The method further comprises:

[0036] Based on the type of the target operation, the value of the third field is adjusted accordingly.

[0037] Another embodiment of the present invention also provides a data filtering device, which is applied to a filtering system. The filtering system includes m cells, each of which includes b storage slots, each of which includes a first field and a second field. The first field is used to record fingerprint information of an element fingerprint of a data element, and the second field is used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering device includes:

[0038] a determination module, configured to determine an element fingerprint of a data element to be processed in response to the obtained target instruction;

[0039] An execution module is used to perform a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each storage slot, where the target operation includes at least one or more of an insertion, a query, and a deletion operation.

[0040] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0041] The technical solution of the present application is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present application and do not constitute a limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 Flowchart of a data filtering method according to an embodiment of the present invention.

[0044] Figure 2 Schematic diagram of the framework of the filtering system in an embodiment of the present invention.

[0045] Figure 3 FIG. 4 is another schematic diagram of a filtering system in an embodiment of the present invention.

[0046] Figure 4 FIG. 4 is another schematic diagram of a filtering system in an embodiment of the present invention.

[0047] Figure 5 This is an example diagram of the sorting performed by the filtering system in an embodiment of the present invention.

[0048] Figure 6 2 is a structural block diagram of a data filtering device in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0050] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0051] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0052] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0053] It should also be understood that although the invention has been described with reference to certain specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics recited in the claims and are therefore within the scope of protection defined thereby.

[0054] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0055] Specific embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant detail. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but rather serve merely as a basis and representative basis for the claims to teach those skilled in the art to variously employ the present disclosure with substantially any suitable detailed structure.

[0056] This description may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," each of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0057] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0058] As shown in the figure, an embodiment of the present invention provides a data filtering method, which is applied to a filtering system. The filtering system includes m cells, each cell includes b storage slots, each storage slot includes a first field and a second field, the first field is used to record fingerprint information of an element fingerprint of a data element, and the second field is used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering method includes:

[0059] In response to the obtained target instruction, determining an element fingerprint of a data element to be processed;

[0060] A target operation is performed on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each storage slot, where the target operation includes at least one or more of an insert, query, and delete operation.

[0061] The fingerprint information is the quotient value and the remainder value calculated based on the filter length of the filter system, and the quotient value and the remainder value corresponding to the same data element are stored in different candidate cells respectively.

[0062] For example, the filtering system is composed of m cells, each of which contains b storage slots. Each storage slot includes but is not limited to two domains, namely the first domain and the second domain. In this embodiment, the first domain and the second domain are respectively referred to as the Carry domain and the Flag domain. The Carry domain is responsible for recording the fingerprint information of the element fingerprint of the data element, such as recording the quotient value, or recording the remainder value. The fingerprint information is the quotient value and the remainder value calculated based on the filtering length of the filtering system. For example, the length of m cells is the filtering length, which can be recorded as m. The Flag domain can be a bit to explicitly mark whether the storage slot records the quotient part (let the Flag bit be 1) or the remainder part (keep the Flag bit as 0), that is, to characterize the specific category of the recorded fingerprint information based on different numerical identifiers. For any empty storage slot, its Carry domain is set to -1, and its Flag domain is set to 0. For any data element x, a hash function h(x) can be used to generate its element fingerprint η x , and the value range of element fingerprint is [0, m(m-1)].

[0063] The filtering method of the filtering system in this embodiment is to store only part of the element fingerprint information, which is either its quotient value or its remainder value. Secondly, the filtering method of this embodiment is to determine the candidate cell index of the data element by calculating the quotient and remainder values. Specifically, the values ​​of the quotient and remainder (respectively denoted as Q and R) are calculated as follows:

[0064] Q x =η x / m

[0065] R x =η x %m

[0066] Furthermore, when the target operation is an insert operation, the target operation is performed on the data element based on at least the element fingerprint and the storage information of the first field and the second field in each storage slot, including:

[0067] Determine the quotient value and remainder value of the data element based on the element fingerprint and the filter length;

[0068] When there is an empty storage slot in the filtering system, directly store the data element in the empty storage slot, and update the corresponding first field and second field based on the quotient value and the remainder value; or

[0069] When there is no empty storage slot in the filtering system, randomly kick out data in a storage slot to store the data element in the empty storage slot;

[0070] Redirecting the kicked data element to a new storage slot and storing it, while dynamically adjusting the values ​​of the corresponding first field and second field. The new storage slot is formed by kicking the data in another storage slot;

[0071] Repeat the above steps until all data is stored to the maximum extent possible.

[0072] For example, the insertion operation of any data element x needs to follow two basic principles: 1) the information of element x can only be stored in the storage slot of its candidate cell; 2) the filtering system is implemented by the cell AF[R x Q x Or in AF[Q x ] to store R x The specific algorithm includes: the filtering system first calculates the fingerprint η of the data element x x , and the quotient Q under the existing parameter settings x Sum remainder value R x If AF[R x ] or AF[Q x ] has an empty storage slot, then Q x or R x Will be stored in it, and the Carry field and Flag field in the storage slot will also be assigned corresponding values. Otherwise, the filtering system will select the two candidate cells AF[R x ] and AF[Q x ] randomly kicks out the quotient value (if the Flag field is 0) or remainder value (if the Flag field is 1) of the element stored in a storage slot to provide space for storing the information of element x. The kicked-out element will be redirected to another candidate cell. If the candidate cell has an empty storage slot, the corresponding remainder value or quotient value will be stored successfully, and the algorithm will return true; otherwise, a data element stored in the candidate cell will be kicked out, and the data element kicked out last time will be stored in it. The above kick-out replay process continues to explore more candidate cells in order to eventually find an empty storage slot and not generate new victims, that is, kicked-out data. The above process is executed at most max times. In this process, the Flag field in the storage slot needs to be dynamically updated according to whether the Carry field stores the quotient value or the remainder value. It is worth noting that if the filtering system returns information that the insertion failed, it means that the filtering system is full and cannot continue to insert new data elements. In this embodiment, the time complexity of inserting a data element is O(b×max), because at most max reallocations are performed, and each reallocation queries at most b storage slots.

[0073] That is, in the filtering system of this embodiment, if the remainder value Rx corresponding to a data element is indexed by Q x The cell is successfully stored, or its quotient value Q x The index value is R x The storage is successful if any cell is successfully stored. In order to solve hash collisions and improve space utilization, the filtering system of this embodiment still adopts the kick-out replay strategy. Specifically, if there is an empty storage slot in any of the two candidate cells, the information of the element x will be stored therein. However, if both candidate cells are fully occupied, the Ark filter will randomly kick out a stored quotient value (if the storage slot Flag field is 0) or remainder value (if the storage slot Flag field is 1). The vacated slot will be used to store the newly added data element, and the proposed element will be marked as a victim and try to search for an empty storage slot in another candidate cell. If the above kick-out replay process can achieve no new victims, the storage is considered successful, otherwise the kick-out replay will continue until the preset max number of times is reached.

[0074] The filtering system of this embodiment essentially leverages the above framework to reuse element fingerprint information and the cell index information of the stored elements. Based on this approach, only one hash calculation is required for each data insertion. Furthermore, the data element fingerprint value range is fixed within [0, m(m-1)] to ensure that the calculated quotient and remainder values ​​do not overflow.

[0075] Furthermore, when the target operation is a query operation, the target operation is performed on the data element based on at least the element fingerprint and the storage information of the first field and the second field in each storage slot, including:

[0076] Determine the quotient value and remainder value of the data element based on the element fingerprint and the filter length;

[0077] Based on the quotient value and the remainder value, querying the first domain and the second domain in which the quotient value and the remainder value are stored in the filtering system;

[0078] Candidate cells for storing the data element are determined based on the query results.

[0079] For example, when querying the membership of a data element x, the filtering system first calculates its element fingerprint η x , and the indices Q of its two candidate cells x and R x , that is, the quotient and remainder values ​​of the corresponding element fingerprint. After that, the filtering system only needs to query the two candidate cells that store the quotient and remainder values. Specifically, in a candidate cell AF[Q x ], if Rx If the element x exists in any slot and the Flag field of the slot is exactly 1, the filter system considers the element x to belong to the target set and returns True. Or, if the filter system finds that Q x There is a cell AF[R x ] in the Carry field of a slot in the , and the Flag field of that slot is 0, then Ark filtering also returns True. Unlike previous methods that directly verify fingerprint elements, the filtering system in this embodiment checks whether the quotient value is stored in a candidate cell. The cell is indexed by the remainder value and must ensure that the Flag field is the corresponding value to be determined, and vice versa. The time complexity of membership queries remains constant because only two candidate cells are visited and checked.

[0080] Furthermore, when the target operation is a delete operation, the target operation is performed on the data element based on at least the element fingerprint and the storage information of the first field and the second field in each storage slot, including:

[0081] After the candidate cells storing the data elements are determined, the data stored in the candidate cells are deleted.

[0082] For example, to delete a data element x, the filtering system must ensure that x has been stored before. Specifically, after determining that it has been stored, the filtering system first calculates the fingerprint, quotient, and remainder value of the data element x, and then tries to remove the data element x from the index that is exactly R. x Delete the quotient information Q in the cell x , and vice versa. Before deletion, the corresponding Flag field must be checked to ensure that the Ark filter is deleting the correct element. If the deletion is successful, the filter returns True; otherwise, it returns False, indicating that the data element x has not been stored before. Element deletion also takes constant time.

[0083] Furthermore, the target operation in this embodiment also includes performing filter compression or filter merging on two isomorphic filter systems;

[0084] The method in this embodiment also includes:

[0085] Determine the storage capacity of two isomorphic filtering systems;

[0086] The data in the filtering system with a smaller storage capacity is represented in the storage form of the filtering system with a larger storage capacity, and stored accordingly.

[0087] For example, a homogeneous filtering system can be given two homogeneous filtering systems with the same parameters m and b, which can be denoted as AF A and AF BFilter compression refers to merging two filter systems into one filter system to represent all fingerprint elements. Specifically, assuming AF A The elements stored are larger than AF B For compression, you can try using AF A Indicates AF B If all fingerprints in AFB can be successfully inserted into AF A If the filter system returns True, otherwise it returns False. A All elements in the union A∪B are recorded. Compression is performed only when |A| + |B| is less than the capacity of AFA. Furthermore, when maintaining multiple filter systems to represent a dynamic set where elements dynamically arrive and disappear, compression algorithms can be used to recycle sparser filter systems. This capability is crucial when space for filter systems is scarce. The time complexity of the compression operation is O(m × b).

[0088] Furthermore, the target operation in this embodiment also includes a filtering deduplication operation for deduplicating identical storage elements in two isomorphic filtering systems, and a filtering intersection operation for obtaining the data intersection of two complete or partial filtering systems.

[0089] For example, in filtering deduplication, given two isomorphic filtering systems AF A and AF B , AF B Subtract AF A Means from AF B Remove the common elements shared by both. A Any storage slot in AF A [i][j], if its Carry field is not -1, then this slot is not empty and stores an element. Then from AF B Search for this element in AF B Find AF in [i] A [i][j].Carry, and the Flag fields of these two slots are equal, or they can be in AF B [AF A [i][j].Carry], and the Flag field of the slot is The element is considered a public element and is removed from the AF A Removed. When traversing the entire AF A After clearing the necessary slots, AF A Contains only the elements in AB. Accordingly, by B Subtract AF A , AF BOnly elements in BA will be stored. The time complexity of the filtering deduplication operation is also O(m×b).

[0090] For example, filter intersection, that is, finding the intersection, is also a basic operation between two given sets, just like the operation between isomorphic filter systems. A and AF B , finding the intersection of the two means to deduce the common elements / fingerprints recorded by the two. For this reason, this embodiment only checks the filter system that stores fewer fingerprint elements, such as AF A If AF A A fingerprint η in t Not present in AF B In the t From AF A The algorithm then returns the updated AF A As an intersection, it only stores the elements in A∩B. The time complexity of this operation is O(m×b).

[0091] The filtering system can also support filtering level operations between heterogeneous filtering systems, as long as these filtering systems use the same fingerprint size and hash function. In this scenario, the filtering system needs to deduce the initial fingerprint of an element in a certain representation by comprehensively considering the Carry field, Flag field, and the index value of the cell where it is located. The specific implementation is as follows:

[0092]

[0093] According to the above formula, when querying η in another filtering system t (initial fingerprint), we can use η t Directly locate the candidate cells. Therefore, the above-mentioned filter compression, filter deduplication and filter intersection can also be implemented between heterogeneous filter systems.

[0094] Furthermore, when determining candidate cells, the filtering system of this embodiment relies on only one hash function to derive the fingerprint and derive the candidate cells based on the fingerprint. The randomness of the fingerprint ensures the randomness of the candidate cells. For an element x, there is Q x =R x In this case, x can only be stored in this cell. For a cell with index i, if and only if η x This special case occurs when =i×m+i. Therefore, for the entire data structure, the probability that N elements have 2 different candidate cells is:

[0095]

[0096] In addition, in this embodiment, when b=2, the space utilization rate of the data structure is likely to reach about 90%. Therefore, in the case of a given space, maintaining b=2 in the filtering system can ensure a larger m (the length of the filtering system), thereby ensuring a lower false positive rate. Accordingly, when n is known, the length of the Ark filter is calculated to be m=(b×n) / T, where b=2 and T≈1.80, in order to achieve the purpose of saving space while ensuring that elements can be successfully represented with a high probability. The length of the filtering system in this embodiment can be set to any integer as needed, with greater flexibility and adaptability.

[0097] Furthermore, when maintaining multiple isomorphic data structures / filtering systems, the index of an element in the first data structure is also applicable to the other data structures, that is, the index is the same. Due to the dependency between the element fingerprint and the data structure length / filtering system length (m), if the two data structures maintain m1 and m2 (m1≠m2) cells respectively, in order to check whether the element x exists, the filtering system must access the element content of x and calculate the element fingerprint of x in each of the two data structures. In contrast, if m1=m2, only accessing the original content of the element once can achieve information reuse of the fingerprint and candidate cell index. Therefore, the filtering system in this embodiment is more inclined to maintain m unchanged while adding Δb empty slots to the cell.

[0098] Specifically, the method in this embodiment further includes:

[0099] Pre-judge the future storage capacity of the filtering system;

[0100] Increase the number of storage slots in some or all cells based at least on the storage capacity.

[0101] For example, the core concept of the filtering system in this embodiment is to expand capacity by adaptively increasing b. The number of empty slots Δb added is determined by the expected number of future elements. Properly setting Δb ensures that all upcoming elements can be represented in the filtering system with high probability. Figure 3 A simple example of a filtering system is shown. The initial filtering system has a three-slot array (b0=3) and is expanded to a four-slot array (b1=4) to accommodate more elements. Note that the element to be inserted will be processed by the activated slot (i.e. the newly added slot). To query x, the filtering system must check all storage slots in its candidate cells: if the corresponding element fingerprint η is not found in both cells x , the filter system will return False, indicating that x is not a member of the set; otherwise, the filter system will return True. The deletion operation of x will also check the corresponding two candidate cells and then try to delete x.

[0102] When the filter system is sparse enough, it is necessary to release the idle space to improve the space efficiency. The filter system in this embodiment uses a global counter C to record the number of elements it stores. in When The threshold value is then reduced to (mean of b). To this end, the filtering system selects the one with the fewest storage elements. The filtering system does not need to access the actual content of any element for the reinsertion of any element, because the index of the candidate cell is applicable to all slot arrays. Specifically, for a number of stored fingerprints not exceeding The fingerprint of the removed slot in the cell will be directly stored in its otherwise, some fingerprints in this cell must be migrated to their alternative candidate cells.

[0103] Furthermore, given a set of elements, frequency estimation can indicate the number of times element x appears in the set. For example, in network measurement, a specific router needs to estimate the amount of traffic passing through it. Further analysis can then support tasks such as large flow detection, large flow conversion, and entropy estimation. For recommendation systems, recording or estimating the number of times users click on a specific product is essential; in other words, recording the number of occurrences (frequency) of the data is essential.

[0104] In this embodiment, each storage slot further includes a third field, which is used to record the frequency of occurrence of the corresponding data element;

[0105] The method further comprises:

[0106] Based on the type of target operation, adjust the value of the third field accordingly.

[0107] For example, the filtering system in this embodiment can further expand the filtering system by introducing a counting field (third field) in each storage slot to accurately record the frequency information of the storage element. Figure 4 As shown in the figure, through this design, each slot in the filtering system has three fields, namely the Carry field for storing fingerprint information, the Flag field for marking the fingerprint storage part, and the counter field for recording the frequency of the represented element.

[0108] It's worth noting that the operations within the filtering system may differ slightly from those in the initial version of Ark Filtering. When inserting an element x, the filtering system still follows the "kick-and-reassign" strategy. However, the filtering system additionally checks whether x is already stored in one of its candidate cells. If so, it simply increments the counter field in the corresponding slot by 1; otherwise, all three fields in the selected slot are updated. To query the frequency of x, the filtering system traverses its two candidate cells to locate the slot it is in and returns the value of the counter field for that slot. If x is not found, the filtering system returns False. Optionally, the filtering system in this embodiment provides two deletion operations: deleting a copy of x or clearing x. To delete a copy of x, the filtering system locates the location of x's fingerprint and decrements the corresponding counter field by 1. If the counter field drops to 0, the Carry field and Flag field are updated to -1 and 0, respectively, indicating that x has been removed from the data structure. In contrast, to clear x, the filtering system locates the element's storage location, sets the Carry field of the corresponding slot to -1, and resets both the Counter field and Flag field to zero. Furthermore, the filtering system in this embodiment incurs an additional overhead in element insertion because it must check whether the element has already been stored before storing it. This overhead is used to determine whether the data element has been successfully inserted. However, the overhead for query and delete operations in the filtering system remains constant.

[0109] For operations at the data structure level, the filtering system in this embodiment still retains support for two filtering systems, such as CAF A and CAF B However, while performing these operations, CAF must handle the count field, i.e., the value of the third field, accordingly. For any element x, assuming CAF A and CAF B f contains data element x A (x) and f B (x) copies. After the compression operation, the value of the count field is updated to f A (x)+f B (x). When from CAF A Minus CAF B , the resulting filtering system needs to record the f of x A (x)-f B (x) copies. Instead, the resulting filter system after performing the intersection operation only retains the min{f A (x), f B(x)} copies. Of course, according to actual needs, users can customize the definition of these data structure-level operations and perform different calculations at the count domain level. This design flexibility makes the filtering system universal in many cases. For example, max{f A (x), f B (x)} as the counter value of x in the compression result. That is, the recording rule of the value in the count field can be user-defined.

[0110] In addition to membership queries, the filtering system in this embodiment can also provide accurate frequency information after the elements are correctly represented. The number of bits required for the counter is often difficult to determine. This is because a counter that is too large will waste space, while a counter that is too small will lead to overflow risks. Basically, the length of the counter is directly determined by the element with the highest frequency. The counting domain can be optimized by setting the length of the counters in the storage cells to be uneven, so that the slots with long counters can store elements with higher frequencies. In other words, each domain is provided with a counter, and the counting capacity of the counters can be the same or different. The cell where the counter with a long counting capacity is located can store elements with higher frequencies.

[0111] Optionally, the filtering system in this embodiment also supports:

[0112] Top-k query:

[0113] Top-k queries can quickly and efficiently return the top k elements. By filtering the values ​​of the counter fields in the system, they can be used for ranking and querying. To address this problem, there are two different approaches: sorting at query time and sorting before querying. The sorting at query time strategy sorts all counters in the data structure to derive the top k data elements during the query. This approach guarantees accurate query results without incurring additional space overhead, but at a high time cost. Current state-of-the-art sorting algorithms, such as TimSort and Quicksort, require O(mb × log(mb)) time overhead to sort all mb counters in the data structure. In contrast, the sorting before query strategy maintains an additional max-(min-) heap that dynamically sorts the elements according to their counter values. This max-(min-) heap only returns the first (last) k elements to answer the query. This scheme naturally achieves accurate and constant-time top-k queries; however, the space overhead it introduces contradicts the design philosophy of data summarization techniques. Therefore, these approaches are either time-consuming or have excessive space overhead. Therefore, based on this, this embodiment provides a filtering system sorting solution, that is, sorting the counters in the cells, and deriving the top k elements when querying based on the sorted cells.

[0114] Specifically, when an element is inserted or updated, the position where it resides in the cell will be rearranged according to its frequency. The filtering system stores the element fingerprints in the candidate cells in ascending or descending order. In the descending order filtering system, the first slot of each cell stores the element with the largest counter value, the second cell stores the second largest counter value, and so on. Conversely, in the ascending order filtering system, the element with the largest counter value is stored in the last slot. Without loss of generality, Figure 5 This article shows a simple example of a sorted filter system. In this example, each cell has three slots, and the elements in each cell are sorted and stored in ascending order of frequency. Whenever the counter value changes, the storage cell reorders all the elements it records. For a top-k query, the sorted filter system must sort the last slots of all m cells to deduce the top k elements based on the first counter value.

[0115] To enhance the accuracy of the sort, this embodiment searches more slots in the data structure to compare more elements before listing the results. Generally, let θ represent the number of storage slots searched in a single storage cell, and then the accuracy of the query can be guaranteed based on the following theorem.

[0116] Theorem 1: In a filtering system with m storage cells and b slots per cell, for top-k queries, when k ≤ θ, the data structure returns the correct answer.

[0117] Whether the filtering system can return the correct query results depends on the distribution of the top-k elements in the data structure. When k ≤ θ, even if all the top-k elements are stored in one cell, they can still be successfully retrieved by the data structure.

[0118] Theorem 2: When the data structure is inserted with top-k elements, the approximate maximum expected number of elements stored in a single cell is:

[0119]

[0120] In addition, when the filter system is used to represent k elements, let λ be an integer variable and represent the number of elements stored in a cell, then the probability that λ is greater than or equal to i∈[0,k] has the following upper bound:

[0121]

[0122] Based on the above theorem, when θ∈[1, b-1], the probability p that the filtering system returns the correct top-k query results can be calculated as:

[0123] p=1-Pr(max(λ)≥θ+1)

[0124]

[0125] When θ=b, p=1, all slots will be traversed and checked to find the top k elements.

[0126] In summary, the filtering system proposed based on the above embodiments has higher throughput, better flexibility and more functions. The core idea of ​​the filtering system in this embodiment is to rely on the fingerprint of each element to index its candidate cells. This design avoids hash calculations or XOR operations during fingerprint redistribution, reduces processing load and improves efficiency. For any data element x, the filtering system calculates the quotient and remainder values ​​of the element relative to its length to index the candidate cells of the element. In addition, the filtering system in this embodiment can also perform a variety of different types of functions, enriching the execution functions of the filtering system. Theoretical analysis shows that under the same false positive rate, the Bpe of the filtering system is not higher than the existing filtering method. In addition, comprehensive experiments show that the performance of the filtering system is significantly better than that of the existing filtering method. In addition, the filtering system can well support dynamic set representation, frequency estimation and top-k query, and the performance has been greatly improved.

[0127] like Figure 6 As shown, another embodiment of the present invention also provides a data filtering device, which is applied to a filtering system. The filtering system includes m cells, each of which includes b storage slots, each of which includes a first field and a second field, the first field is used to record fingerprint information of an element fingerprint of a data element, and the second field is used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering device includes:

[0128] a determination module, configured to determine an element fingerprint of a data element to be processed in response to the obtained target instruction;

[0129] An execution module is used to perform a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each storage slot, where the target operation includes at least one or more of an insertion, a query, and a deletion operation.

[0130] As an optional embodiment, the fingerprint information is a quotient value and a remainder value calculated based on a filter length of the filter system, and the quotient value and remainder value corresponding to the same data element are respectively stored in different candidate cells.

[0131] As an optional embodiment, the target operation is an insert operation;

[0132] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0133] Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length;

[0134] When there is an empty storage slot in the filtering system, directly storing the data element in the empty storage slot, and updating the corresponding first field and second field based on the quotient value and the remainder value; or

[0135] When there is no empty storage slot in the filtering system, randomly kicking out data in the storage slot to store the data element in the empty storage slot;

[0136] Redirecting the kicked-out data element to a new storage slot and storing the kicked-out data element, while dynamically adjusting the values ​​of the corresponding first field and second field, wherein the new storage slot is formed by kicking out the data in another storage slot;

[0137] Repeat the above steps until all data is stored to the maximum extent possible.

[0138] As an optional embodiment, the target operation is a query operation;

[0139] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0140] Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length;

[0141] Based on the quotient value and the remainder value, querying the first domain and the second domain corresponding to the quotient value and the remainder value in the filtering system;

[0142] The candidate cell storing the data element is determined based on the query result.

[0143] As an optional embodiment, the target operation is a delete operation;

[0144] The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes:

[0145] After the candidate cells storing the data element are determined, the data stored in the candidate cells are deleted.

[0146] As an optional embodiment, the target operation further includes performing filter compression or filter merging on the two isomorphic filter systems;

[0147] The method further comprises:

[0148] determining the storage capacity of the two isomorphic filtering systems;

[0149] The data in the filtering system with a smaller storage capacity is represented in the storage form of the filtering system with a larger storage capacity, and is stored accordingly.

[0150] As an optional embodiment, the target operation further includes a filtering deduplication operation for deduplicating identical storage elements in two isomorphic filtering systems, and a filtering intersection operation for obtaining the data intersection of two complete or partial filtering systems.

[0151] As an optional embodiment, the device further includes:

[0152] A pre-judgment module, used for pre-judging the future storage capacity of the filtering system;

[0153] An increasing module is used to increase the number of storage slots in some or all of the cells based at least on the storage capacity.

[0154] As an optional embodiment, each of the storage slots further includes a third field, and the third field is used to record the frequency of occurrence of the corresponding data element;

[0155] The device further comprises:

[0156] An adjustment module is configured to adjust the value of the third field accordingly based on the type of the target operation.

[0157] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the scope of the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the spirit and scope of protection of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present invention.

Claims

1. A data filtering method, applied to a filtering system, wherein the filtering system comprises m cells, each cell comprising b storage slots, each storage slot comprising a first field and a second field, the first field being used to record fingerprint information of an element fingerprint of a data element, and the second field being used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering method comprises: In response to the obtained target instruction, determining an element fingerprint of a data element to be processed; performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots, the target operation comprising at least one or more of an insert, query, and delete operation; The fingerprint information is a quotient value and a remainder value calculated based on the filter length of the filter system. The quotient value and remainder value corresponding to the same data element are stored in different candidate cells, wherein the quotient value is stored in the first field of the candidate cell and the remainder value is stored in the second field of the candidate cell.

2. The method according to claim 1, wherein The target operation is an insert operation; The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes: Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length; When there is an empty storage slot in the filtering system, directly storing the data element in the empty storage slot, and updating the corresponding first field and second field based on the quotient value and the remainder value; or When there is no empty storage slot in the filtering system, randomly kicking out data in the storage slot to store the data element in the empty storage slot; Redirecting the kicked-out data element to a new storage slot and storing the kicked-out data element, while dynamically adjusting the values ​​of the corresponding first field and second field, wherein the new storage slot is formed by kicking out the data in another storage slot; Repeat the above steps until all data is stored to the maximum extent possible.

3. The method according to claim 1, wherein The target operation is a query operation; The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes: Determining a quotient value and a remainder value of the data element based on the element fingerprint and the filter length; Based on the quotient value and the remainder value, querying the first domain and the second domain corresponding to the quotient value and the remainder value in the filtering system; The candidate cell storing the data element is determined based on the query result.

4. The method according to claim 3, wherein: The target operation is a delete operation; The performing a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots includes: After the candidate cells storing the data element are determined, the data stored in the candidate cells are deleted.

5. The method according to claim 1, wherein The target operation also includes performing filter compression or filter merging on two isomorphic filter systems; The method further comprises: determining the storage capacity of the two isomorphic filtering systems; The data in the filtering system with a smaller storage capacity is represented in the storage form of the filtering system with a larger storage capacity, and is stored accordingly.

6. The method according to claim 1, wherein The target operation also includes a filtering deduplication operation for deduplicating identical storage elements in two isomorphic filtering systems, and a filtering intersection operation for obtaining the data intersection of two complete or partial filtering systems.

7. The method according to claim 1, wherein Also includes: Pre-determining the future storage capacity of the filtering system; The number of storage slots in some or all of the cells is increased based at least on the storage amount.

8. The method according to claim 1, wherein Each of the storage slots further includes a third field, the third field being used to record the frequency of occurrence of the corresponding data element; The method further comprises: Based on the type of the target operation, the value of the third field is adjusted accordingly.

9. A data filtering device, applied to a filtering system, the filtering system comprising m cells, each cell comprising b storage slots, each storage slot comprising a first field and a second field, the first field being used to record fingerprint information of an element fingerprint of a data element, and the second field being used to match and record an identification identifier corresponding to the fingerprint information stored in the first field. The data filtering device comprises: a determination module, configured to determine an element fingerprint of a data element to be processed in response to the obtained target instruction; an execution module, configured to perform a target operation on the data element based at least on the element fingerprint and the storage information of the first field and the second field in each of the storage slots, the target operation comprising at least one or more of an insert, query, and delete operation; The fingerprint information is a quotient value and a remainder value calculated based on the filter length of the filter system. The quotient value and remainder value corresponding to the same data element are stored in different candidate cells, wherein the quotient value is stored in the first field of the candidate cell and the remainder value is stored in the second field of the candidate cell.

Images