An efficient filtering method for accelerating the query of the LSM tree in the cloud platform database

Abstract

The present invention discloses an efficient filtering method for accelerating the query of the LSM tree in a cloud platform database, including the following steps: dividing the written data into mutually independent dataset sub-blocks according to their own characteristics, and separately constructing a cache-line-sized block Bloom filter for each dataset sub-block; combining the data that is missing in the current data block but has been frequently queried historically, adaptively adjusting the hash function set of the written data and storing it in a block hash expresser; jointly constructing a block hash adaptive filter with the block Bloom filter and the block hash mapping table, and deploying it into the system. When judging whether data is written, the single instruction multiple data stream technology is used to simultaneously detect multiple bit positions in a block. The present invention divides the filter into blocks according to the cache line size, and parallelly detects the bit data in the blocks, greatly improving the query efficiency, and combining the hash adaptive technology to effectively avoid the problem of reduced accuracy caused by block division.

Description

Technical Field

[0001] The present invention relates to the field of cloud platform databases, and particularly to an efficient filtering method for accelerating the query of the LSM tree in a cloud platform database. Background Art

[0002] The Log Structured Merge (LSM) tree is a data structure widely used in database storage engines. Its basic idea is to store data in layers. The top layer uses memory to store data, and the remaining layers use disks. Write operations only write data into memory. When the data capacity stored in a layer reaches the threshold, the data in that layer is merged and written into the next layer in sorted order. Query operations start from the top-level memory and query each layer downwards until the data is found. Using the LSM tree can bring extremely high write speeds, but the read speed is relatively slow. In extreme cases, if the data to be queried has not been written, the read operation needs to scan all the data.

[0003] Since the I / O cost caused by the query failure of the LSM tree is very high, it seriously affects the query performance of the database. A common optimization method is to use a Bloom filter to record the data that has been written. Before performing a query operation, first judge whether the data to be queried has been written through the Bloom filter, and then decide whether to perform the query. However, there are false positives in the Bloom filter, that is, data that has not been written is judged to have been written, which leads to a performance decline. Summary of the Invention

[0004] Object of the Invention: To solve the problem of low query performance of the LSM tree, the present invention proposes an efficient filtering method for accelerating the query of the LSM tree in a cloud platform database. The core is to make full use of the cache. First, a new type of Bloom filter is proposed: the block hash adaptive filter, and the core is to divide the Bloom filter into blocks. Secondly, a data writing judgment system based on the block hash adaptive filter is proposed. Technical Solution: The present invention provides an efficient filtering method for accelerating the query of the LSM tree in a cloud platform database, including the following steps:

[0005] (1) Initialization stage of the block hash adaptive filter. The specific steps include: First, all data is divided according to whether it has been written into the LSM tree, that is, the written data and the unwritten data. Secondly, the written data is divided into mutually independent data set sub-blocks according to its own characteristics. Finally, a block Bloom filter is constructed for each data set sub-block.

[0006] (2) Adaptive stage of the block hash adaptive filter. The specific steps include: First, divide the unwritten data into a block Bloom filter. According to whether it is misjudged, the unwritten data is divided into two sets, namely the misjudged unwritten data and the non-misjudged unwritten data. Second, establish two block hash mapping tables based on the written data and the non-misjudged unwritten data respectively. Third, adjust the hash function set of the written data in order to reduce the number of misjudged unwritten data. Finally, re-insert each written data into the block Bloom filter according to its adjusted hash function set, and at the same time store the hash function set into the block hash expressor.

[0007] (3) Deployment and application stage of the block hash adaptive filter. The block Bloom filter and the two block hash mapping tables together constitute the block hash adaptive filter and are deployed into the system. When performing queries, the single instruction multiple data stream technology is used to improve the comparison speed of the bits in the block Bloom filter.

[0008] Specific measures taken to optimize the above technical solution also include:

[0009] Further, in step (1), in stage 1, the unwritten data refers to the data that is missing in the current data block but is frequently queried historically, and is obtained by collecting logs.

[0010] Further, in step (1), the initialization construction process of the block Bloom filter includes the following steps: 1. Apply for a bit array with a length of m bits and divide it into n equal-length parts, each part is called a block; 2. For each inserted written data, use the hash function f to map it to a block of the bit array; 3. For the written data that has been mapped to the block, obtain k bits in the block by using k hash functions, and set these k bits to 1.

[0011] Further, in step (1), the initialization construction process of the block Bloom filter includes the following steps: 1. Apply for a bit array with a length of m bits and divide it into n equal-length parts, each part is called a block; 2. For each inserted written data, use the hash function f to map it to a block of the bit array; 3. For the written data that has been mapped to the block, obtain k bits in the block by using k hash functions, and set these k bits to 1.

[0012] Further, in step (1), the k hash functions used in the initialization construction process of the block Bloom filter are a globally unified default hash function set.

[0013] Further, in step (2), the situation where unwritten data is misjudged as written data specifically means that for unwritten data, a block where it may be inserted is selected through the hash function f. k bits are obtained in this block using k globally unified default hash functions. If all these k bits are 1, it is considered that this unwritten data is misjudged as written data; otherwise, as long as there is one bit that is 0, there is no misjudgment situation.

[0014] Further, in step (2), the specific definition of the block hash mapping table is as follows: 1. The data structure of the block hash mapping table consists of two levels. The outer layer contains multiple block structures, where the block structures correspond one-to-one with the blocks in the block Bloom filter. Each block structure contains multiple bucket structures, where the bucket structures correspond one-to-one with each bit in the Bloom filter. 2. The bucket structure is used to record the data identifier mapped to its corresponding bit. 3. Block hash mapping tables V and T are constructed respectively for the written data and the unwritten data that are not misjudged, and their internal blocks are represented by Vi and Ti respectively.

[0015] Further, in step (2), the steps to adjust the hash function set of the written data are as follows: 1. Traverse the unwritten data eck that is misjudged, and map it using the hash function f to obtain its corresponding block Bloom filter and the block hash mapping table Vi of the written data. 2. Use the default hash function set to obtain k buckets of eck in the block hash mapping table Vi of the written data, and take any written data es corresponding to a bucket. 3. Replace the hash function hi that maps es to this bucket and map es to other buckets. 4. According to the new bit position where es is mapped, obtain the bucket corresponding to the same position in the block hash mapping table Ti of the unwritten data that is not misjudged. 5. Determine whether the unwritten data that is not misjudged and mapped to this position will be misjudged due to the replacement of the hash function. As long as there is one bucket that does not result in misjudgment, the hash function replacement is successful.

[0016] Further, in step (2), the hash function set of each written data will be stored in the block hash expressor. The block hash expressor contains multiple block structures, and each block structure contains multiple tuples. The position of the block corresponds one-to-one with the block in the block Bloom filter, and the position of the tuple corresponds one-to-one with the position of the bit in the Bloom filter. The specific operation of inserting the hash function into the block hash expressor is as follows: 1. Use the hash function f to obtain the block in the hash expressor corresponding to the data es. 2. Use a unified hash function to map es to a tuple in the block and randomly insert one of the hash functions in the hash function set into this tuple. 3. Continue to use the inserted hash function to remap es to another tuple within this block. 4. Repeat the above process until the entire hash function set is inserted.

[0017] Further, in step (3), the deployment and application process of the block hash adaptive filter includes the following steps: 1. Receive a request for user query data. 2. Determine whether the data has been written through the block Bloom filter. If it has been written, allow the LSM tree to execute the query operation and end the process; otherwise, proceed to the next step. 3. Use the set of hash functions corresponding to the data given by the hash expresser to make another determination. If it has been written, allow the LSM tree to execute the query operation; otherwise, the LSM tree cannot execute the query operation.

[0018] Further, in step (3), the SIMD technology is adopted to improve the comparison speed of bit positions in the block Bloom filter. SIMD (Single Instruction Multiple Data), that is, single instruction multiple data streams, can perform multiple data operations simultaneously within the execution cycle of one instruction, thereby improving the execution efficiency of the program. Specifically, a group of patterns are predefined, where a pattern refers to the positions of random k bits within a block. When preparing to insert or query the block Bloom filter, only one hash function is used to select a pattern, and then SIMD is used to insert or detect the pattern into the block. Since the size of the block is the same as the cache line size, by reasonably using SIMD, multiple bit positions within a block can be detected simultaneously, thereby further improving the detection speed.

[0019] Beneficial effects: The cache-friendly block hash adaptive Bloom filter proposed by the present invention fully considers the overhead caused by the discrete memory access of the Bloom filter. The filter is divided into blocks according to the cache line size, so that all the required data can be obtained through one memory access, greatly reducing the query latency of the filter; the method of the present invention uses hash adaptive technology to reduce the false positive rate of queries, effectively avoiding the problem of reduced accuracy caused by block division of the filter; the method of the present invention introduces SIMD to parallelly detect the bit data within the block, further reducing the query latency of the filter. Description of the Drawings

[0020] Figure 1 It is the overall system architecture diagram of the method of the present invention;

[0021] Figure 2 It is a schematic diagram of the method of the present invention for establishing two block hash mapping tables for the written data and the unwritten data that have not been misjudged respectively;

[0022] Figure 3 It is a schematic diagram of the method of the present invention for storing the set of hash functions of each written data into the block hash expresser;

[0023] Figure 4 It is the flowchart for judging whether the data is written in the method of the present invention;

[0024] Figure 5It is a performance comparison diagram of false positive rate between the method of the present invention and a hash adaptive filter (HABF), a Bloom filter (BF) and a block Bloom filter (BBF);

[0025] Figure 6 The figure is a schematic diagram showing the performance comparison of query delay between the method of the present invention and a hash adaptive filter (HABF), a Bloom filter (BF) and a block Bloom filter (BBF). DETAILED DESCRIPTION

[0026] The present invention is further illustrated below in conjunction with the accompanying drawings and specific implementation examples. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the appended claims of the present application.

[0027] The present invention provides an efficient filtering method for accelerating LSM tree query of cloud platform database. The system architecture is as follows: Figure 1 As shown in the figure, the method mainly includes the following three stages:

[0028] Phase 1: Block hash adaptive filter initialization phase.

[0029] Step 1.1: All data in the system are divided into written data and unwritten data according to whether they have been written into the LSM tree. Unwritten data refers to data that is missing in the current data block but has been frequently queried in the past. This data is obtained by collecting logs. When a user queries written data, the system will allow the LSM tree to perform query operations. When a user requests unwritten data, the system will deny the LSM tree from performing query operations.

[0030] Step 1.2, divide the written data into independent data set sub-blocks through a hash function.

[0031] Step 1.3, construct a cache line-sized block Bloom filter for each data set sub-block separately. The specific construction process is as follows: 1. Apply for a bit array of length m bits and divide it into n equal-length parts, each of which is called a block; 2. For each inserted written data, use the hash function f to map it to a block in the bit array; 3. For the written data that has been mapped to the block, use k hash functions to get k bits in the block, and set these k bits to 1. At this point, the block Bloom filter that stores all the written data is built.

[0032] Phase 2: Block Hash Adaptive Filter Adaptation Phase.

[0033] Step 2.1, select a block in the partitioned Bloom filter where the data not yet written may be inserted through the hash function f. Obtain k bits in this block using k globally unified default hash functions. If all these k bits are 1, it is considered that the data not yet written is misjudged as data already written; otherwise, as long as there is one bit that is 0, there is no misjudgment. Divide the data not yet written into two sets according to whether there is a misjudgment, namely the misjudged data not yet written and the non-misjudged data not yet written.

[0034] Step 2.2, establish two partitioned hash mapping tables V and T for the data already written and the non-misjudged data not yet written respectively, as Figure 2 shown. The data structure of the partitioned hash mapping table consists of two levels. The outer layer contains multiple block structures, where the block structures correspond one-to-one with the blocks in the partitioned Bloom filter. Each block structure contains multiple bucket structures, where the bucket structures correspond one-to-one with each bit in the Bloom filter. The bucket structure is used to record the data identifier mapped to its corresponding bit.

[0035] Step 2.3, adjust the hash function set of the data already written to reduce the number of misjudged data not yet written. The specific steps are as follows: 1. Traverse the misjudged data not yet written eck, map it using the hash function f to obtain its corresponding partitioned Bloom filter and the partitioned hash mapping table Vi of the data already written. 2. Use the default hash function set to obtain k buckets of eck in the partitioned hash mapping table Vi of the data already written, and take the data already written es corresponding to any one of the buckets. 3. Replace the hash function hi that maps es to this bucket and map es to other buckets. 4. According to the new bit position where es is mapped, obtain the bucket at the corresponding position in the partitioned hash mapping table Ti of the non-misjudged data not yet written. 5. Judge whether the non-misjudged data not yet written mapped to this position will be misjudged due to the replacement of the hash function. As long as there is one bucket that does not have a misjudgment, the hash function replacement is successful.

[0036] Step 2.4, store the hash function set of each data already written into the partitioned hash expressor, as Figure 3 shown. Among them, the partitioned hash expressor contains multiple block structures, the block structure contains multiple tuples, the position of the block corresponds one-to-one with the block in the partitioned Bloom filter, and the position of the tuple corresponds one-to-one with the position of the bit in the Bloom filter. The specific operation of inserting the hash function into the partitioned hash expressor is as follows: 1. Use the hash function f to obtain the block in the hash expressor corresponding to the data es. 2. Use a unified hash function to map es to a tuple in the block and randomly insert one of the hash functions in the hash function set into this tuple. 3. Continue to use the inserted hash function to remap es to another tuple within this block. 4. Repeat the above process until the entire hash function set is inserted.

[0037] Step 2.5, re-insert the written data into the chunk Bloom filter using the corresponding hash function set.

[0038] Phase 3: Deployment and application phase of the chunk hash adaptive filter.

[0039] Step 3.1, the chunk Bloom filter and the chunk hash mapping table together constitute the chunk hash adaptive filter and are deployed into the system.

[0040] Step 3.2, the process of judging whether the data has been written, as Figure 4 shown, 1. Receive the user's request for querying data. 2. Judge whether the data has been written through the chunk Bloom filter. If it has been written, the system allows the LSM tree to perform the query operation and ends the process. Otherwise, go to step 3. 3. Use the hash function set corresponding to this data given by the hash expresser to judge again. If it has been written, the system allows the LSM tree to perform the query operation. Otherwise, the system rejects the LSM tree from performing the query operation. Among them, when detecting whether k bit positions in a block in the chunk Bloom filter are all 1, since the size of the block is the same as the cache line size, SIMD can be reasonably used to detect multiple bit positions in a block simultaneously, thereby further improving the detection speed.

[0041] Furthermore, the performance of the novel and efficient filtering method for accelerating LSM tree queries described in the present invention is verified through comparative experiments. All programs in the experiment are implemented in C++, and the O3 optimization of the g++ compiler is enabled at the same time. The configuration of the test machine is as follows: Intel(R) Xeon(R) Gold 6248 CPU @ 10 cores, 2.5 GHZ, 106 GB RAM. The comparison algorithms are the hash adaptive filter (HABF), the Bloom filter (BF), and the chunk Bloom filter (BBF). As Figure 5 shown, in terms of the false positive rate (FPR), the performance of the method of the present invention is improved by more than 10 times compared with BF and BBF. As Figure 6 shown, in terms of query latency, the performance of the method of the present invention is improved by about 10 times compared with HABF, and the time consumption is only 77% of that of BF. The experimental results show that compared with HABF, the introduction of the chunk technology and the SIMD technology greatly reduces the cache miss situation and effectively reduces the query latency.

[0042] In summary, the novel and efficient filtering method proposed by the present invention for accelerating the query of the LSM tree fully considers the overhead caused by the discrete memory access of the Bloom filter, divides the filter into blocks according to the cache line size, uses SIMD to parallelly detect the bit data within the blocks, greatly improves the query efficiency, and combines the hash adaptation technology to effectively avoid the problem of reduced accuracy caused by block division, and can be used to accelerate the query operation of the LSM tree.

Claims

1. An efficient filtering method for accelerating the query of the LSM tree in a cloud platform database, comprising the following steps: (1) Initialization stage of the block hash adaptive filter: First, all data is divided into written data and unwritten data according to whether it has been written into the LSM tree. Among them, the unwritten data refers to the data that is missing in the current data block but is frequently queried in history. The unwritten data is obtained by collecting logs. Secondly, the written data is divided into independent dataset sub-blocks according to its own characteristics. Finally, a block Bloom filter is constructed for each dataset sub-block; (2) Adaptive stage of the block hash adaptive filter: First, the unwritten data is divided into the block Bloom filter, and the unwritten data is divided into misjudged unwritten data and non-misjudged unwritten data. Secondly, two block hash mapping tables are established respectively according to the written data and the non-misjudged unwritten data. Thirdly, the hash function set of the written data is adjusted to reduce the number of misjudged unwritten data. Finally, each written data is re-inserted into the block Bloom filter according to the adjusted hash function set, and the adjusted hash function set is stored in the block hash expressor at the same time; (3) Deployment and application stage of the block hash adaptive filter: The block Bloom filter and the two block hash mapping tables together constitute the block hash adaptive filter and are deployed into the system; when querying, the single instruction multiple data stream technology is adopted to improve the comparison speed of the bits in the block Bloom filter.

2. An efficient filtering method for accelerating the query of the LSM tree in the cloud platform database according to claim 1, characterized in that, In the step (1), the written data is divided into different blocks according to its own characteristics and inserted into the corresponding block Bloom filters with the cache line size respectively, and then the Bloom filters with the cache line size are merged to form the block Bloom filter.

3. An efficient filtering method for accelerating the query of the LSM tree in a cloud platform database according to claim 1, characterized in that, In the step (1), the initialization construction process of the block Bloom filter includes the following steps: First, a bit array with a length of m bits is applied for, and the bit array is divided into n equal-length parts, and each part is called a block. Secondly, for each inserted written data, the written data is mapped to a block of the bit array by using the hash function f. Finally, for the written data that has been mapped to the block, k bits in the block are obtained by using k hash functions, and the k bits are set to 1.

4. An efficient filtering method for accelerating the query of the LSM tree in the cloud platform database according to claim 1, characterized in that, In the step (2), the block hash expressor includes multiple block structures, the block structures include multiple tuples, the positions of the block structures correspond to the blocks in the block Bloom filter one by one, and the positions of the tuples correspond to the positions of the bits in the Bloom filter with the cache line size one by one; Each block of the block hash expressor stores the hash function set of the written data in the corresponding block Bloom filter.

5. An efficient filtering method for accelerating the query of the LSM tree in a cloud platform database according to claim 1, characterized in that, In step (3), when operating on the partitioned Bloom filter, single instruction multiple data streams are used to perform multiple data operations simultaneously within the execution cycle of one instruction, thereby improving the execution efficiency of the program; a group of patterns are predefined, where a pattern refers to the positions of random k bits within a block; when preparing to insert into or query the partitioned Bloom filter, only one hash function is used to select a pattern, and then the pattern is inserted into or detected in the block using single instruction multiple data streams.

Images