Method for managing cuckoo filter during sstable merging in LSM tree

Abstract

The present invention relates to a method for managing a cuckoo filter during SSTable merging in an LSM tree. The method according to the present invention comprises the steps of: when a first cuckoo filter and a second cuckoo filter are connected to a first SSTable file and a second SSTable file, respectively, and the load factor of a cuckoo filter formed by merging the first cuckoo filter and the second cuckoo filter is less than or equal to a predetermined threshold value, merging the first cuckoo filter and the second cuckoo filter; and connecting a third cuckoo filter obtained by merging the first cuckoo filter and the second cuckoo filter to a third SSTable file obtained by merging the first SSTable file and the second SSTable file. According to the present invention, a cuckoo filter can be more efficiently managed during SSTTable merging in an LSM tree. In addition, the amount of database I / O can be reduced compared to the prior art. In addition, the occurrence of false negatives can be prevented or reduced.

Description

How to manage cooker filters when merging SSTables in LSM Trees

[0001] The present invention relates to a method for managing a cooker filter when merging SSTables in an LSM Tree implemented in a computing device.

[0002] The Log Structured Merge Tree (LSM-Tree) is a disk-based data structure designed to provide a low-cost index for files with a high rate of record insertions (and deletions) over long periods of time. The LSM-Tree consists of two data structures: one in memory and one persistent on disk. In an LSM-Tree, data starts in memory and is merged into increasingly larger disk layers.

[0003] The first structure in an LSM Tree is a memory table (Memtable), which is maintained in memory (RAM). A Memtable is implemented as a sorted tree structure, such as a Red-Black Tree or an Adelson-Velskii-Landis Tree (AVL Tree), and data in a Memtable is stored in the form of key-value pairs.

[0004] The second structure in the LSM Tree is the Sorted String Table (SSTable), which is larger than the Memtable and stored on disk. The SSTable is implemented as a sequential file, and data is stored in the form of key-value pairs.

[0005] Meanwhile, database query performance can be improved by using probabilistic data structures such as bloom filters or cuckoo filters in LSM-Tree.

[0006] Figure 1 illustrates an example of a database structure that applies a probabilistic data structure to an LSM Tree.

[0007] Referring to Figure 1, when there is a write request for data, the data is stored in the Memtable located in memory and the WAL (Write-Ahead Log) located in disk (11).

[0008] When the Memtable reaches a predetermined size, an Immutable Memtable is created (12) and flushed to the disk (200) as an SSTable (220). This process is called Minor Compaction (13). A probabilistic data structure filter (e.g., Bloom filter (Bf), Cooker filter (Ckf), etc.) corresponding to the Immutable Memtable is connected to the SSTable to which the Immutable Memtable is flushed, and the filter (Ckf, Bf) can be used to search for data in the connected SSTable.

[0009] For example, when there is a read request (Read), the Memtable is checked first, and if the corresponding key is not found, the data is sequentially retrieved from the SSTable stored on the disk using the filter (Ckf, Bf).

[0010] Meanwhile, when the volume of each disk layer (level) of an SSTable exceeds a certain value or a predetermined condition is met, such as the passage of a predetermined amount of time, the SSTable is merged. This process is called Major Compaction. When an SSTable is merged, a new filter corresponding to the merged SSTable is created, loaded into memory, and connected to the SSTable.

[0011] Cooker filters have the ability to efficiently delete or remove elements from a set, whereas standard Bloom filters do not. Considering this, the use of cooker filters in LSM-Tree has been increasing in recent years, as it can improve the performance of databases like HBase, where SSTables continuously merge and compress files for file optimization.

[0012] However, the existing method of creating a new cooker filter corresponding to the merged SSTable and connecting it when merging SSTables has a problem that it can increase overhead in the system due to the time and load required to create a new cooker filter.

[0013] The technical problem to be solved by the present invention is to provide a method for more efficiently managing cooker filters when merging SSTables.

[0014] A method for managing a cuckoo filter when merging SSTables in an LSM Tree (Log-Structured Merge Tree) according to the present invention for solving the above technical problem includes, when a first cooker filter and a second cooker filter are connected to a first SSTable file and a second SSTable file, respectively, a step of merging the first cooker filter and the second cooker filter if a load factor of a cooker filter created by merging the first cooker filter and the second cooker filter is lower than or equal to a predetermined threshold value, and a step of connecting a third cooker filter obtained by merging the first cooker filter and the second cooker filter to a third SSTable file obtained by merging the first SSTable file and the second SSTable file.

[0015] The merging of the first cooker filter and the second cooker filter can be performed by adding a fingerprint extracted from the second cooker filter to the first cooker filter while the first cooker filter and the second cooker filter are loaded into the memory, and deleting the second cooker filter from the memory.

[0016] The method may further include a step of linking a linked list linking the first cooker filter and the second cooker filter to the third SSTable file when the load ratio of the cooker filter in which the first cooker filter and the second cooker filter are merged exceeds a predetermined threshold value.

[0017] The method may further include a step of connecting a first linked list connecting a plurality of cooker filters to the first SSTable file, a second linked list connecting a plurality of cooker filters to the second SSTable file, or a fourth cooker filter, if the length of a third linked list created when connecting the second linked list or the fourth cooker filter to the first linked list is less than or equal to a predetermined length threshold value, connecting the third linked list to the third SSTable file.

[0018] The method may further include a step of generating a cooker filter corresponding to the third SSTable file and linking the cooker filter to the third SSTable file when the length of the third linked list exceeds a predetermined length threshold.

[0019] The load ratio of the above cooker filter can be obtained by the mathematical formula below.

[0020] [Mathematical Formula 1]

[0021] Load Factor = A / (B×C)

[0022] Here, A is the number of fingerprints filled in the cooker filter, B is the total number of buckets in the cooker filter, and C is the bucket size of the cooker filter.

[0023] A computing device according to the present invention for solving the above-mentioned technical problem includes a processor; and a memory storing instructions or programs executable by the processor.

[0024] When the above command or program is executed by the processor, the above method is executed.

[0025] The present invention enables more efficient management of cooker filters when merging SSTables in an LSM Tree. Furthermore, the amount of database I / O can be reduced compared to conventional methods. Furthermore, the occurrence of false negatives can be prevented or reduced.

[0026] Figure 1 illustrates an example of a database structure that applies a probabilistic data structure to an LSM Tree.

[0027] FIG. 2 is a diagram showing the configuration of a computing device according to one embodiment of the present invention.

[0028] FIG. 3 illustrates an example of a cooker filter management method according to one embodiment of the present invention.

[0029] FIG. 4 is an exemplary diagram illustrating a process of creating a third cooker filter by merging the first cooker filter and the second cooker filter in FIG. 3.

[0030] FIG. 5 illustrates another example of a cooker filter management method according to one embodiment of the present invention.

[0031] Figures 6 and 7 illustrate another example of a cooker filter management method according to another embodiment of the present invention.

[0032] Figure 8 is a flowchart for explaining a cooker filter management method according to one embodiment of the present invention.

[0033] Then, with reference to the attached drawings, an embodiment of the present invention will be described in detail so that a person having ordinary knowledge in the technical field to which the present invention pertains can easily carry out the present invention.

[0034] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and "and / or" includes each and any combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present invention.

[0035] As used herein, the term "computing device" encompasses a variety of devices capable of performing computational processing and providing results to a user. For example, computing devices may include desktop PCs, notebook computers, and server computers, as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous / asynchronous IMT-2000 (International Mobile Telecommunication-2000) mobile terminals, Palm PCs (Palm Personal Computers), and personal digital assistants (PDAs).

[0036] FIG. 2 is a diagram showing the configuration of a computing device according to one embodiment of the present invention.

[0037] Referring to FIG. 2, a computing device according to the present invention may include a memory (110), non-volatile storage (120), and a control unit (130).

[0038] The memory (110) is implemented as a volatile memory such as a RAM (Random Access Memory), and can store commands, data, program execution status information, etc. related to the running program, and can store a memory table (Memtable), an Immutable Memtable, and a Cuckoo filter used in a Log Structured Merge Tree (LSM-Tree) data structure.

[0039] Non-volatile storage (120) can be implemented as a non-volatile memory such as a hard disk, SSD (Solid State Drive), etc., and one or more SSTables (Sorted String Tables) used in LSM-Tree can be stored.

[0040] The control unit (130) can perform basic arithmetic, logic, and input / output operations, and execute a process corresponding to one or more instructions loaded into the memory (110). The control unit (130) can execute a method for managing a cooker filter when merging SSTables in an LSM Tree, and can be referred to as a processor.

[0041] The control unit (130) may consider the following methods to manage the cooker filter, which is a probabilistic data structure (PDS) combined with an LSM Tree (Log-Structured Merge Tree).

[0042] Below, an example is described of merging a first SSTable file (SSTable 1) and a second SSTable (SSTable 2) stored in non-volatile storage (120) into a third SSTable file (SSTable new).

[0043] FIG. 3 illustrates an example of a cooker filter management method according to one embodiment of the present invention.

[0044] Referring to FIG. 3, the control unit (130) can calculate the load factor of the third cooker filter (Merged Ckf) created by merging the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2), while the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) connected to the first SSTable file (SSTable 1) and the second SSTable file (SSTable 2) are loaded into the memory (110).

[0045] Here, merging the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) loaded into the memory (110) means extracting a fingerprint (FingerPrint) from one filter among the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) and adding the extracted fingerprint to the other filter.

[0046] FIG. 4 is an exemplary diagram illustrating a process of creating a third cooker filter by merging the first cooker filter and the second cooker filter in FIG. 3.

[0047] In Fig. 4, each cooker filter includes 9 buckets, and the case where each bucket size is 2 is exemplified.

[0048] As illustrated in Fig. 4, a third cooker filter (Merged Ckf) can be created by merging the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) by adding a fingerprint extracted from the second cooker filter (Ckf 2) to the first cooker filter (Ckf 1).

[0049] The cooker filter merging according to FIGS. 3 and 4 can be performed only when the load ratio of the merged cooker filter is below a predetermined threshold value. The threshold value may be determined differently depending on the embodiment, but since there is a high risk of false negatives occurring when the load ratio of the cooker filter exceeds 65%, the threshold value can be set to 65%.

[0050] The load factor of the cooker filter can be calculated using the mathematical formula below.

[0051] [Mathematical Formula 1]

[0052] Load Factor = A / (B×C)

[0053] Here, A is the number of fingerprints filled in the cooker filter, B is the total number of buckets in the cooker filter, and C is the bucket size of the cooker filter.

[0054] Referring again to FIG. 3, as a result of the completion of the Compaction Process, in which SSTable 1 and SSTable 2 are merged and Ckf 1 and Ckf 2 are merged, SSTable 1 and SSTable 2 are deleted from the non-volatile storage (120) and SSTable new is newly created and stored. In addition, Ckf 2, which was merged into Ckf 1, is deleted from the memory (110).

[0055] FIG. 5 illustrates another example of a cooker filter management method according to one embodiment of the present invention.

[0056] Meanwhile, if the load ratio of the third cooker filter (Merged Ckf) in the example of FIG. 3 is calculated to exceed the threshold value, the control unit (130) determines not to merge the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) in order to prevent the occurrence of a false negative. Then, as illustrated in FIG. 5, a linked list (Ckf 1, Ckf 2) linking the first cooker filter (Ckf 1) and the second cooker filter (Ckf 2) can be created and then linked to the third SSTable file (SSTable new).

[0057] In the example of Fig. 5, after the Compaction Process is completed, SSTable 1 and SSTable 2 are deleted from non-volatile storage (120) and SSTable new is newly created and stored. In addition, Ckf 1 and Ckf 2 are maintained in a linked list in memory (110).

[0058] Figures 6 and 7 illustrate another example of a cooker filter management method according to another embodiment of the present invention.

[0059] As illustrated in FIGS. 6 and 7, when the linked list connecting the cooker filters is connected to at least one of the first SSTable file (SSTable 1) or the second SSTable file (SSTable 2), the control unit (130) checks whether the length of the linked list to be connected to the third SSTable file (SSTable new) exceeds a length threshold value. Here, the length of the linked list can be defined as the number of cooker filters included in the linked list.

[0060] Referring to FIG. 6, for example, if the length threshold is set to 3, in the example of FIG. 6, three cooker filters (Ckf 1, Ckf 2, Ckf 3) can be linked into a linked list and then linked to a third SSTable file (SSTable new).

[0061] In the example of Fig. 6, after the Compaction Process is completed, SSTable 1 and SSTable 2 are deleted from non-volatile storage (120) and SSTable new is newly created and stored. In addition, Ckf 1, Ckf 2, and Ckf 3 are connected and maintained in a linked list in memory (110).

[0062] In the example of Fig. 7, four cooker filters (Ckf 1, Ckf 2, Ckf 3, Ckf 4) are connected, so that the length of the linked list becomes 4, which exceeds the length threshold of 3. If the length of the linked list connected to the SSTable becomes too long, the time complexity of the query process may increase, which may cause a problem in that the database response waiting time may become long. Therefore, in order to prevent this problem, if the length of the linked list exceeds a predetermined threshold, the control unit (130) may be implemented to create a new cooker filter (Ckf New) for the third SSTable file (SSTable new), and then connect the created cooker filter (Ckf New) to the third SSTable file (SSTable new). Here, creation of a cooker filter (Create) refers to the task of creating a new cooker filter (Ckf New) in memory (110) separately from the existing cooker filter, and then storing a fingerprint obtained by using two hash functions for data included in the third SSTable file (SSTable new) in the bucket of Ckf New.

[0063] In the case of Fig. 7, after the Compaction Process is completed, SSTable 1 and SSTable 2 are deleted in the non-volatile storage (120) and SSTable new is newly created and stored. In addition, in the memory (110), Ckf 1, Ckf 2, Ckf 3 and Ckf 4 are deleted and Ckf new is newly created and maintained.

[0064] Figure 8 is a flowchart for explaining a cooker filter management method according to one embodiment of the present invention.

[0065] Referring to FIG. 8, when merging SSTable 1 and SSTable 2, the control unit (130) can first check whether a linked list in which multiple cooker filters are connected is connected to at least one of SSTable 1 and SSTable 2 (S810).

[0066] If Ckf 1 and Ckf 2 are connected to SSTable 1 and SSTable 2, that is, if no linked list is connected to either SSTable 1 or SSTable 2 (S810-N), the control unit (130) calculates in advance the load factor of the Merged Ckf (e.g., a cooker filter created by merging Ckf 2 with Ckf 1) and checks whether it exceeds a threshold value (S820).

[0067] If the load ratio of the Merged Ckf is below the threshold value (S820-N), the control unit (130) merges Ckf 1 and Ckf 2 and connects SSTable 1 and SSTable 2 to the merged SSTable new (S821). In step (S821), the control unit (130) can add the fingerprint extracted from Ckf 2 to Ckf 1 and merge it while Ckf 1 and Ckf 2 are loaded into the memory (110), and then delete Ckf 2 from the memory (110).

[0068] Meanwhile, if the load ratio of Merged Ckf exceeds the threshold value (S820-Y), the control unit (130) connects the linked list (Ckf 1, Ckf 2) connecting Ckf 1 and Ckf 2 to SSTable new (S823).

[0069] If at least one of SSTable 1 and SSTable 2 has a linked list (S810-Y), it is checked whether the length of the linked list to be linked to SSTable new exceeds the length threshold value (S830).

[0070] For example, if a linked list (Ckf 1, Ckf 2) is connected to SSTable 1 and Ckf 3 is connected to SSTable 2, and the length threshold value is set to 3, since the length of the linked list to be connected to SSTable new does not exceed the length threshold value (S830-N), the control unit (130) performs step (S831) to connect the linked list (Ckf 1, Ckf 2, Ckf 3) to SSTable new. If it is assumed that the length threshold value is set to 2, since the length of the linked list to be connected to SSTable new exceeds the length threshold value (S830-Y), the control unit (130) performs step (S833) to connect the newly created Ckf new to SSTable new.

[0071] Meanwhile, if the linked list (Ckf 1, Ckf 2) is connected to SSTable 1 and the linked list (Ckf 3, Ckf 4) is connected to SSTable 2, and the length threshold value is set to 3, the length of the linked list to be connected to SSTable new exceeds the length threshold value (S830-Y), so the control unit (130) performs step (S833) to connect the newly created Ckf new to SSTable new. If it is assumed that the length threshold value is set to 4, the length of the linked list to be connected to SSTable new does not exceed the length threshold value (S830-N), so the control unit (130) can perform step (S831) to connect the linked list (Ckf 1, Ckf 2, Ckf 3, Ckf4) to SSTable new.

[0072] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose computing devices or special-purpose computing devices, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0073] Software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0074] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0075] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. How to manage the cuckoo filter when merging SSTables in LSM Tree (Log-Structured Merge Tree) If the first cooker filter and the second cooker filter are connected to the first SSTable file and the second SSTable file, respectively, If the load factor of the cooker filter created by merging the first cooker filter and the second cooker filter is below a predetermined threshold value, a step of merging the first cooker filter and the second cooker filter, and A step of connecting a third cooker filter that merges the first cooker filter and the second cooker filter to a third SSTable file that merges the first SSTable file and the second SSTable file. A method of managing a cooker filter executed by a computing device comprising:

2. In paragraph 1, The merging of the first cooker filter and the second cooker filter is A method for managing a cooker filter, executed by a computing device, wherein the first cooker filter and the second cooker filter are loaded into a memory, a fingerprint extracted from the second cooker filter is added to the first cooker filter, and the second cooker filter is deleted from the memory.

3. In paragraph 2, When the load ratio of the cooker filter in which the first cooker filter and the second cooker filter are merged exceeds a predetermined threshold value, a step of linking a linked list connecting the first cooker filter and the second cooker filter to the third SSTable file A method of managing a cooker filter executed by a computing device further comprising:

4. In paragraph 1, If a first linked list connecting multiple cooker filters is connected to the first SSTable file, and a second linked list connecting multiple cooker filters or a fourth cooker filter is connected to the second SSTable file, A step of connecting the third linked list created when connecting the second linked list or the fourth cooker filter to the first linked list, if the length of the third linked list is less than or equal to a predetermined length threshold value, to the third SSTable file. A method of managing a cooker filter executed by a computing device further comprising:

5. In paragraph 4, If the length of the third linked list exceeds a predetermined length threshold, a step of creating a cooker filter corresponding to the third SSTable file and connecting it to the third SSTable file. How to include more.

6. In paragraph 1, The load rate of the above cooker filter is calculated by the mathematical formula below, [Mathematical Formula 1] Load Factor = A / (B×C) Here, A is the number of fingerprints filled in the cooker filter, B is the total number of buckets in the cooker filter, and C is the bucket size of the cooker filter.

7. As a computing device, processor; and A memory storing instructions or programs executable by the processor; A computing device in which the method described in claim 1 is executed when the above command or program is executed by the processor.

8. As a computer-readable recording medium, A computer-readable recording medium storing a computer program that executes the method described in claim 1 when executed by at least one processor.

Images