A multi-set lookup insertion and lookup optimization method based on multi-path parallel processing

Abstract

The application discloses a multi-set search insertion and search optimization method based on multi-path parallel processing, which firstly initializes multi-set search structure operation, sets parameters of the multi-set search structure, and allocates corresponding memory space. Secondly, element data is inserted into the initialized multi-set search structure, and the format of the inserted data in the multi-set search structure is a key-value pair, wherein the 'value' of different elements must be different, but the 'key' can be the same value. Finally, the multi-set search structure receives to-be-queried data, the to-be-queried data is organized into a queue before entering the structure, then enters the structure for multi-path query processing, and the processed data is output. The application improves the query speed of the multi-set search, reduces the backlog of to-be-queried data in a large data scene, improves the compactness of the overall structure, and maintains high precision.

Description

Technical Field

[0001] This invention belongs to the field of multi-set search, specifically relating to a multi-set search insertion and search optimization method based on multi-path parallel processing. Background Technology

[0002] Approximate membership lookup is a common method in the field of big data. Classic membership lookup methods include Bloom Filter (BF), Cuckoo Filter (CF), and Quotient Filter (QF). This structure can answer whether the element being checked belongs to a specific set. BF is simple and fast, but cannot delete elements; CF has low memory cost, but has capacity limitations; QF further improves space efficiency, however, QF has high update complexity. To address the issues of BF's inability to delete elements, CF's capacity and load, and QF's query speed, many variations such as Counting Bloom Filter, Consistent Cuckoo Filter, and Mixed-counters Quotient Filter have emerged.

[0003] As data complexity increases in areas such as network packet classification and MAC address table lookups—for example, classic MAC table scenarios require distinguishing relationships between 10,000 entries and dozens of ports—simply confirming an element's membership in a single set becomes insufficient. Therefore, multi-set search has gained significant attention. Multi-set search is a common algorithm in big data, capable of finding the set to which data belonging to different subsets belong. Assuming k disjoint sets S1, ..., S2, ... k One of the sets contains element e, and multi-set search is responsible for determining which set element e belongs to.

[0004] Currently popular multi-set search methods include Indexed Set Tables (ISet), Difference Bloom Filters (DBF), and Balanced Multi-hash Color Tables (BMCT). ISet innovatively combines a Bloom Bloom-based Index Filter (IF) with a hash table. It uses IF to quickly calculate and store the index, and the hash table to store the fingerprint of the key. It also incorporates a checksum mechanism to ensure a sufficiently low false positive rate. DBF utilizes the characteristics of Bloom Bloom Filters and, with appropriate modifications, enables multi-set search capabilities. While its query speed is fast, it suffers from an increased false positive rate. BMCT is a data structure that uses hash tables and Bloom Bloom Filters for multi-set search. It uses a graph algorithm to batch the insertion process, significantly improving the structure's space efficiency and insertion speed. However, BMCT does not provide a method for batch processing of query elements. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies in batch processing of query elements in BMCT by designing and implementing a multi-set search and insertion optimization method based on multi-path parallel processing. Structurally, this invention consists of a hash table, a batch function (BF), and multiple computing resources, which can be a thread or a computer device. The data to be searched is queued, and a hash table and its corresponding index in the BF are calculated for it. The query process is divided into multiple parallel processing tasks, which independently query and output the corresponding results. Collisions that may occur during the process are resolved by hashing the elements to multiple index positions using a hash function, with each index position having multiple slots for storing data.

[0006] This invention provides a multi-set search and insertion optimization method based on multi-path parallel processing, comprising the following steps:

[0007] S1, Initialize the multi-set search structure operation. Set the various parameters of the multi-set search, including the number of hash functions used in the multi-set search structure, the number of buckets in each sub-table of the hash table, and the number of slots in each bucket, and then allocate the corresponding memory space for it.

[0008] S2 inserts element data into the initialized multi-set search structure. The format of the data inserted into the multi-set search structure is key-value pairs, where the "value" of different elements must be different, but the "key" can be the same value.

[0009] S3, the multi-set search structure receives the data to be queried. The data to be queried is organized into a queue before entering the multi-set search structure, and then enters the multi-set search structure for multi-way query processing. The processed data will be output.

[0010] Furthermore, the multi-set lookup structure in step S1 is organized by a hash table and a BF (Browser Function). The hash table has the same number of sub-tables as the number of hash functions set, so as to enable parallel operations. A slot is the basic unit for storing the data corresponding to an element, while a bucket is the smallest indexable unit. Each bucket contains at least one slot, and each sub-table of the hash table is composed of buckets.

[0011] Furthermore, step S2 includes the following steps:

[0012] S21, input the value of the key-value pair to be inserted into the hash function group in the multi-set lookup structure to obtain the position index information of each hash sub-table. The hash function group contains the same number of hash functions as the hash sub-tables in the multi-set lookup structure, and each hash function is independent of the others.

[0013] S22, input the location index information into each hash sub-table to obtain the load status of the corresponding bucket. The load status here corresponds to the percentage of slots containing data in the bucket.

[0014] S23. Select the bucket with the lowest load among all hash sub-tables (if the load is the same at the corresponding position in all sub-tables, randomly select the bucket for insertion), and insert the key from the key-value pair data. During the key insertion process, first check if the slot in the bucket that already contains data has the same key. If so, do not store it again; otherwise, select an empty slot in the bucket to store the key information.

[0015] S24, insert “value|key” into BF, where “|” is the concatenation operator.

[0016] S25 returns the insertion result. If none of the corresponding buckets in all hash sub-tables have empty slots, the insertion fails; otherwise, the insertion succeeds.

[0017] Furthermore, the data to be queried in step S3 is organized as a single value, rather than a key-value pair.

[0018] Furthermore, step S3 includes the following steps:

[0019] S31 organizes the computing resources participating in multi-way parallel search into an array for management, and organizes the data to be queried into a queue in chronological order.

[0020] S32, the main computing resource inputs the data to be queried into the hash function group in the multi-set lookup structure to obtain the position index of each corresponding sub-table.

[0021] S33: The main computing resource iterates through the computing resource array, searching for an idle computing resource. It submits the data to be queried and its position index to the corresponding computing resource, then exits the computing resource array. The main computing resource re-enters the array at the position following the last idle computing resource found by the main computing resource.

[0022] S34: After obtaining the data to be queried and the location index information, the computational resources access the corresponding position in the hash sub-table, retrieve all data stored at that position, and organize all data into a candidate set. Then, the "|" operation is performed on each element in the candidate set and the data to be queried, where "|" is the concatenation operator, and the membership of the result of this operation in BF is verified. If the result is a member of BF, it indicates that the element is the key of the data to be queried, and it is output as the result; otherwise, it is output that the element does not exist in the multi-set search structure.

[0023] S35, repeat S31 to S34 until the queue storing the data to be queried is cleared or the multi-set search structure is closed.

[0024] The beneficial effects of this invention are as follows: This invention combines a hash table, a batch file (BF), and multiple computing resources to queue the data to be searched and calculate the hash table and corresponding index in the BF for it. The query process is divided into multiple parallel processing tasks, which independently query and output the corresponding results, further improving the query speed of multi-set searches and reducing the backlog of data to be queried in big data scenarios. This invention also uses load balancing to improve the compactness of the overall structure while maintaining high accuracy. Attached Figure Description

[0025] Figure 1 : Bloom filter schematic diagram;

[0026] Figure 2 : A schematic diagram of the hash table structure in a multi-set search structure; Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings. A multi-set search and insertion and search optimization method based on multi-path parallel processing includes the following detailed steps:

[0028] Step 1: Structure Initialization

[0029] Configure various parameters for multi-set lookup, including the number of hash functions used in the multi-set lookup structure, the number of buckets in each sub-table of the hash table, and the number of slots in each bucket. A slot is the basic unit for storing the data corresponding to an element, while a bucket is the smallest indexable unit. Each bucket contains at least one slot, and each sub-table of the hash table is composed of buckets.

[0030] The server sets an appropriate capacity for the multi-set lookup structure based on requirements, and then allocates corresponding hash tables and Bloom filter (BF) memory space according to the capacity. It's important to note that the precision of the BF decreases with increasing load. This is due to the characteristics of the BF; to check an element, the BF needs to check if all bits at the corresponding position are 1 based on the position index information obtained from its hash function group. As the BF load increases, a large number of bits within the BF will be set to "1". Therefore, the probability of checking an uninserted element with all bits corresponding to its position index being equal to "1" increases. Thus, the BF size needs to be set appropriately according to requirements. Simultaneously, the form of computing resources is specified based on the settings and requirements. These resources can be a computer or a thread. The working principle of a Bloom filter is as follows... Figure 1 As shown.

[0031] Step 2: Insert member data

[0032] Member data is organized as key-value pairs. These key-value pairs are input into a hash function group in a multi-set lookup structure to obtain the position index corresponding to each sub-table. The load of the corresponding position in each sub-table is then obtained through the position index. Finally, the position with the lowest load is selected to hold the key in the key-value pair. The structure of the hash table is as follows: Figure 2 As shown. During the insertion process, it is necessary to check whether the key has already been inserted. This is done by traversing each bucket corresponding to the position index to see if there is a slot containing the same key. If so, the key has already been inserted and does not need to be inserted again. This approach improves the compactness of the multi-set search structure. The specific process is as follows:

[0033] S21, input the value of the key-value pair to be inserted into the hash function group in the multi-set lookup structure to obtain the position index information of each hash sub-table. The hash function group contains the same number of hash functions as the hash sub-tables in the multi-set lookup structure, and each hash function is independent of the others.

[0034] S22, input the location index information into each hash sub-table to obtain the load status of the corresponding bucket. The load status here corresponds to the percentage of slots containing data in the bucket.

[0035] S23. Select the bucket with the lowest load among all hash sub-tables (if the load is the same at the corresponding position in all sub-tables, randomly select the bucket for insertion), and insert the key from the key-value pair data. During the key insertion process, first check if the slot in the bucket that already contains data has the same key. If so, do not store it again; otherwise, select an empty slot in the bucket to store the key information.

[0036] S24, insert “value|key” into BF, where “|” is the concatenation operator.

[0037] S25 returns the insertion result. If none of the corresponding buckets in all hash sub-tables have empty slots, the insertion fails; otherwise, the insertion succeeds.

[0038] Step 3: The multi-set search structure receives data and performs a multi-set search.

[0039] A multi-set lookup structure receives data, checks its membership, and outputs the corresponding key. If the data is a member of a set-set lookup structure (BF), the output is the key; otherwise, it outputs `false` to indicate that the key is not a member of BF. The resources used to process subtasks in a multi-set lookup structure are computational resources, specifically a thread or a computer. The specific process is as follows:

[0040] S31 organizes the computing resources participating in multi-way parallel search into an array for management, and organizes the data to be queried into a queue in chronological order.

[0041] S32, the main computing resource inputs the data to be queried into the hash function group in the multi-set lookup structure to obtain the position index of each corresponding sub-table.

[0042] S33: The main computing resource iterates through the computing resource array, searching for an idle computing resource. It submits the data to be queried and its position index to the corresponding computing resource, then exits the computing resource array. The main computing resource re-enters the array at the position following the last idle computing resource found by the main computing resource.

[0043] S34: After obtaining the data to be queried and the location index information from the computing resources, the corresponding position in the hash sub-table is accessed to retrieve all data stored at that position, and all data is organized into a candidate set. Then, a "|" operation is performed on each element in the candidate set and the data to be queried, and the membership of the result of this operation in BF is verified. If the result is a member of BF, it indicates that the element is the key of the data to be queried, and it is output as the result; otherwise, it is output that the element does not exist in the multi-set search structure.

[0044] The process of BF verifying membership is as follows: The result to be verified is input into BF. BF uses its own hash function group to process the data and obtains the same number of position indices as the hash function group. Finally, BF uses these position indices to check the corresponding bits in the BF structure. If these bits are all equal to "1", then the result is a member of BF; otherwise, the result is not a member of BF.

[0045] S35, repeat S31 to S34 until the queue storing the data to be queried is cleared or the multi-set search structure is closed.

[0046] Step 4: Repeat step 3 until the multi-set lookup structure is turned off or the deployment platform malfunctions.

Claims

1. A multi-set search, insertion, and search optimization method based on multi-path parallel processing, characterized in that, Includes the following steps: S1, Initialize the multi-set search structure: Set the various parameters of the multi-set search and then allocate the corresponding memory space for it; S2, insert element data into the initialized multi-set search structure. The format of the data inserted into the multi-set search structure is key-value pairs, where the "value" of different elements must be different. S3, Multi-set Search Structure Receives Query Data: Before entering the multi-set search structure, the query data is organized into a queue, then enters the multi-set search structure for multi-way query processing. The processed data will be output. The query data is organized into a single value. The specific process is as follows: S31 organizes the computing resources participating in multi-way parallel search into an array for management, and organizes the data to be queried into a queue in the order of query. S32, the main computing resource inputs the data to be queried into the hash function group in the multi-set search structure to obtain the position index of each corresponding sub-table; S33, the main computing resource circulates through the computing resource array to find an idle computing resource, and submits the data to be queried and its location index information to the corresponding computing resource, and then exits the computing resource array; The position of the main computing resource in the array is the position after the last time the main computing resource found an available computing resource; S34, After the computing resources obtain the data to be queried and the location index information, access the corresponding position in the hash sub-table, retrieve all the data stored in that position, and organize all the data into a candidate set; Then, the "|" operation is performed on each element in the query data and the candidate set, and the membership of the result of the operation in BF is verified. If the result of the operation is a member of BF, it means that the element is the key of the query data and is output as the result; otherwise, it is output that the element does not exist in the multi-set search structure. The process of BF verifying membership is as follows: The verification result is input into BF. BF uses its own hash function group to process the data and obtains the same number of position indices as the hash function group. Finally, BF uses these position indices to check the corresponding bits in the BF structure. If these bits are all equal to "1", then the result is a member of BF; otherwise, the result is not a member of BF. S35, repeat S31 to S34 until the queue storing the data to be queried is cleared or the multi-set search structure is closed.

2. The multi-set search, insertion, and search optimization method based on multi-path parallel processing according to claim 1, characterized in that, In S1, the parameters include the number of hash functions used in the multi-set lookup structure, the number of buckets in each sub-table of the hash table, and the number of slots in each bucket.

3. The multi-set search, insertion, and search optimization method based on multi-path parallel processing according to claim 2, characterized in that, In S1, the multi-set lookup structure consists of a hash table and a Bloom filter (BF). The hash table has the same number of sub-tables as the number of hash functions set. A slot is the basic unit for storing the data corresponding to an element; a bucket is the smallest indexable unit. Each bucket contains at least one slot, and each sub-table of a hash table is composed of buckets.

4. The multi-set search, insertion, and search optimization method based on multi-path parallel processing according to claim 3, characterized in that, The specific process of step S2 is as follows: S21, input the value in the key-value pair to be inserted into the hash function group in the multi-set search structure to obtain the position index information of each hash sub-table. The hash function group contains the same number of hash functions as the hash sub-table in the multi-set search structure, and each hash function is independent of the others. S22, input the location index information into each hash sub-table respectively, and obtain the load status of the corresponding bucket; S23, select the bucket with the least load among all hash sub-tables and insert the key into the key-value pair data; during the key insertion process, first check whether the slot in the bucket that already contains data contains the same key. If so, do not store it again. Otherwise, select an empty slot in the bucket to store the key information. S24, insert "value|key" into BF, where "|" is the concatenation operator; S25 returns the insertion result. If none of the buckets at the corresponding positions in all hash sub-tables have empty slots, the insertion fails; otherwise, the insertion succeeds.

5. The multi-set search, insertion, and search optimization method based on multi-path parallel processing according to claim 4, characterized in that, In S22, the load condition refers to the percentage of slots containing data in the bucket.

6. The multi-set search, insertion, and search optimization method based on multi-path parallel processing according to claim 4, characterized in that, S23 also includes the option to randomly select the bucket to insert when the load at the corresponding position of all sub-tables is the same.

Images