Apparatus and method for dynamically supporting compaction of key-value store by considering workload patterns

Abstract

An apparatus and method for dynamically supporting compaction of a key-value store by considering workload patterns according to one embodiment comprises: a memory storing a MemTable and an Immutable MemTable in which data composed of key-value pairs is temporarily stored; a storage comprising a plurality of Sorted String Tables (SSTables) in one or more key ranges to store the data flushed from the memory; and a processor performing a compaction operation to compact data in a plurality of SSTables existing at one level within the storage to the next level and the flush operation. The processor detects the size of a level 0 (zero) (L0) where a flush is performed, the rate of the flush, and the rate of the compaction for compaction performed periodically through a compaction size recommender, and based on the size of the level 0, the rate of the flush, and the rate of the compaction, at a level 0 stall threshold size Calculate the latency until it is reached and the level 0 compaction latency of the next cycle, determine the compaction size of the corresponding cycle based on the compaction rate, the latency until it is reached and the level 0 compaction latency of the next cycle, and select the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle to perform compaction to the next level.

Description

Technology Field

[0001] Below, a compaction technique for an LSM tree-based key-value store is provided, which adjusts the compaction size by considering the workload pattern to reduce write stalls. Background Technology

[0003] Computer systems need to select a database with an appropriate data structure for processing data according to a specific workload. A Key-Value Store is a type of non-relational database that stores data using simple key-value methods, and can also be referred to as a NoSQL database. Examples of Key-Value Stores include Dynamo, Level DB, Rocks DB, Cassandra, and HB BASE. Computer systems utilizing this approach can store data in the Key-Value Store as a collection of key-value pairs where the key serves as a unique identifier. Due to the high partitionability of these Key-Value Stores in data processing, they can manage data more flexibly and efficiently than relational databases.

[0004] The background technology described above is possessed or acquired by the inventor in the process of deriving the content of the disclosure of the present application, and cannot necessarily be considered as prior art disclosed to the general public prior to the filing of this application. Prior art literature

[0006] Korean Published Patent Application No. 10-2024-0096005 (published June 26, 2024) presents a storage system based on a key-value store and a method for processing data in a storage system based on a key-value store. Korean Published Patent Application No. 10-2023-0080210 (published June 7, 2023) presents a data management method and apparatus for a key-value store. The problem to be solved

[0007] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment can detect a workload pattern at runtime and dynamically adjust the compaction size based on the detected workload pattern.

[0008] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment can calculate the latency until a level 0 stall threshold size is reached and the level 0 compaction latency of the next cycle based on the level 0 size, the flush rate, and the compaction rate, and determine the compaction size of the corresponding cycle based on the compaction rate, the calculated latency until a level 0 stall threshold size is reached, and the level 0 compaction latency of the next cycle.

[0009] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment can perform compaction to the next level by selecting the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle determined by a compaction size recommender.

[0010] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist. means of solving the problem

[0012] An apparatus for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment comprises: a memory storing a MemTable and an Immutable MemTable in which data composed of key-value pairs is temporarily stored; a storage comprising a plurality of Sorted String Tables (SSTables) in one or more key ranges to store the data flushed from the memory; and a processor performing a compaction operation to compact data within a plurality of SSTables existing at one level within the storage to the next level and the flush operation. The processor detects the size of Level 0 (zero) (L0) where a flush is performed, the flush rate, and the compaction rate for compaction performed periodically through a compaction size recommender, and based on the size of Level 0, the flush rate, and the compaction rate, the latency until reaching the Level 0 stall threshold size and the Level 0 of the next cycle The compaction latency is calculated, and the compaction size of the corresponding cycle is determined based on the compaction rate, the latency until the level 0 stall threshold size is reached, and the level 0 compaction latency of the next cycle, and compaction to the next level can be performed by selecting the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle.

[0013] The processor can calculate the latency until the level 0 stall threshold size is reached based on the predetermined level 0 stall threshold size, the size of the level 0, and the rate of the flush, and calculate the level 0 compaction latency of the next cycle based on the level 0 compaction size of the next cycle and the rate of the compaction.

[0014] The processor may perform the compaction with a fixed number of key ranges for level 0, and perform the compaction for levels after level 0 based on the compaction size of the corresponding cycle.

[0015] The processor checks whether the sub-level where the compaction is performed and stored is available before performing the compaction at a level after level 0, and if the sub-level is available, performs the compaction based on the compaction size determined using the compaction size recommender, and if the sub-level is not available, checks whether the sub-level of the next level of the level is available and performs the compaction.

[0016] The flush rate is determined as a value based on the amount of data flushed during a predetermined time, and the compaction rate can be determined as a value obtained by dividing the total compaction size up to date by the total compaction time up to date.

[0017] The level 0 compaction size of the above next cycle may be the value obtained by dividing the level 0 stall threshold size by the number of key ranges at level 0.

[0018] The compaction size of the above cycle can be determined by multiplying the compaction speed by the difference between the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle.

[0019] The above processor can perform storage of key-value pairs based on a Log Structured Merge-Tree (LSM-Tree) data structure.

[0020] A method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment may include: detecting the size of a level 0 (zero) (L0) where a flush is performed, the rate of the flush, and the rate of compaction for compaction performed on a periodic basis through a compaction size recommender; calculating the latency until a level 0 stall threshold size is reached and the level 0 compaction latency for the next period based on the size of the level 0, the rate of the flush, and the rate of compaction; determining the compaction size for the corresponding period based on the compaction rate, the latency until a level 0 stall threshold size is reached, and the level 0 compaction latency for the next period; and selecting the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding period and performing compaction to the next level.

[0021] The step of calculating the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle may include calculating the latency before reaching the level 0 stall threshold size based on a predetermined level 0 stall threshold size, the size of the level 0, and the speed of the flush, and calculating the level 0 compaction latency of the next cycle based on the level 0 compaction size of the next cycle and the compaction speed.

[0022] The step of performing the compaction may include performing the compaction with a fixed number of key ranges for level 0, and performing the compaction for levels after level 0 based on the compaction size of the corresponding cycle.

[0023] The step of performing the compaction may further include the step of checking whether the sub-level where the compaction is performed and stored is available before performing the compaction at a level after level 0, and if the sub-level is available, performing the compaction based on the compaction size determined using the compaction size recommender, and if the sub-level is not available, checking whether the sub-level of the next level of the level is available and performing the compaction.

[0024] The above flush rate is a value based on the amount of data flushed during a predetermined time, and the above compaction rate can be determined by the value obtained by dividing the total compaction size up to now by the total compaction time up to now.

[0025] The level 0 compaction size of the above next cycle may be the value obtained by dividing the level 0 stall threshold size by the number of key ranges at level 0.

[0026] The compaction size of the above cycle can be determined by multiplying the compaction speed by the difference between the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle.

[0027] The above processor can perform storage of key-value pairs based on a Log Structured Merge-Tree (LSM-Tree) data structure. Effects of the invention

[0029] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment can perform optimal compaction and minimize write stalls at the same time by detecting a workload pattern at runtime and dynamically adjusting the compaction size based on the detected workload pattern.

[0030] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment calculates the latency until reaching a level 0 stall threshold size and the level 0 compaction latency of the next cycle based on the level 0 size, the flush rate, and the compaction rate, and determines the compaction size of the corresponding cycle based on the compaction rate, the calculated latency until reaching a level 0 stall threshold size, and the level 0 compaction latency of the next cycle, thereby having a small compaction size for write-intensive workloads and a large compaction size for read-intensive workloads.

[0031] An apparatus and method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment can reduce write stalls in write-intensive workloads compared to existing LSM tree-based key-value stores by selecting the number of key ranges at a corresponding level corresponding to the compaction size of the corresponding cycle determined by a compaction size recommender and performing compaction to the next level, thereby improving write performance, and can also improve read performance by setting a large lower-level compaction size in read-intensive workloads to reduce the number of levels where data resides, and thereby reducing the number of disk IOs caused by read queries. Brief explanation of the drawing

[0033] FIG. 1 illustrates a block diagram of a device that supports dynamic compaction of a key-value store by considering a workload pattern according to one embodiment. FIG. 2 exemplarily illustrates the structure of a key-value store of a device that supports dynamic compaction of a key-value store by considering a workload pattern according to one embodiment. FIG. 3 exemplarily shows a compaction bit array showing whether compaction occurs between L0 and L1 in a device that supports compaction of a key-value store dynamically in consideration of a workload pattern according to one embodiment. FIG. 4 exemplarily illustrates a process of selecting the number of key ranges at a corresponding level based on the compaction size of the corresponding cycle determined in a device that dynamically supports compaction of a key-value store considering a workload pattern according to one embodiment. FIG. 5 illustrates an exemplary compaction process between L1 and L2 in a device that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment. FIG. 6 illustrates a flowchart of a method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment. FIG. 7 shows a graph of normalized throughput according to write / read workload of a method that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment. Specific details for implementing the invention

[0034] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0035] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0036] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0037] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0038] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0039] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0040] Key-value stores are representative NoSQL databases used in many applications due to their high scalability. The LSM tree structure, a type of key-value store, is widely used in various data-intensive applications because it offers high write performance due to its sequential write characteristics. Processors utilizing LSM tree-based key-value store technology store data by dividing it between memory and disk. When data is inserted, it is stored in a memory buffer; if the buffer size exceeds a threshold, the buffer is flushed to disk. Subsequently, the processor manages the flushed files stored on disk by dividing them into a multi-level structure. More specifically, the processor maintains a sorted state for every level except the first level, L0. To maintain this sorted state, the processor periodically performs compaction by merging and sorting the files.

[0041] If data insertion rates are high in write-intensive workloads, unaligned files may be flushed rapidly, which can lead to high read amplification and space amplification. To prevent this, the processor may perform a write stall, which arbitrarily adjusts the data insertion rate. The processor may perform a write stall in specific cases, such as when there are too many buffers in memory (e.g., MemTable), when too many files reside in L0, or when the size of pending compaction exceeds a threshold. In particular, if too many files reside in L0, the processor may perform write stalls most frequently because a large number of newly flushed files occur in L0 while compaction is being performed at lower levels other than L0 (e.g., levels storing old data). This means that write stalls may occur due to low-latency lower-level compaction, potentially leading to a decrease in write speeds. In addition, if it is a read-intensive workload, write stalls rarely occur because the insertion speed of new data is slow. However, even in read-intensive workloads, since the processor still manages files in a multi-level structure, there is a problem where multiple disk I / O operations may occur to find a key.

[0042] Furthermore, workload patterns regarding the ratio of actual write and read operations can change over time. For example, sensing-related data may be inserted or retrieved in periodic patterns, such as diurnality. Accordingly, processors utilizing LSM tree-based key-value store technology may need to adopt different compaction strategies based on workload patterns to improve performance. For write-intensive workload patterns, it may be important to write as much data as possible to disk (e.g., L0) as quickly as possible. In this case, the processor may need to minimize compaction while simultaneously minimizing write stalls. On the other hand, for read-intensive workload patterns, reducing the number of files to access may be important. Therefore, the processor needs to actively perform compaction to align inserted key-value pairs to a single level or as few levels as possible. However, current compaction strategies use a uniform, static compaction size regardless of the workload.

[0043] Accordingly, an apparatus and method that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment can perform optimal compaction and minimize write stalls at the same time by detecting a workload pattern at runtime and dynamically adjusting the compaction size based on the detected workload pattern.

[0044] FIG. 1 illustrates a block diagram of a device that supports dynamic compaction of a key-value store by considering a workload pattern according to one embodiment.

[0045] The device (100) may include a processor (110), memory (120), and storage (130). Structural features of each component are described later in FIG. 2. The device (100) may be used to access data stored in a Key Value Store (KVS) and to ensure its integrity.

[0046] The processor (110) can perform compaction and flush operations to compact data within multiple SSTables existing at one level in storage to the next level. In the compaction operation, the processor (110) can detect the size of the level 0 (zero) (L0) where flushing is performed, the flush rate, and the compaction rate for compaction performed on a periodic basis through a compaction size recommender. Subsequently, the processor (110) can calculate the latency until the level 0 stall threshold size is reached and the level 0 compaction latency for the next period based on the level 0 size, the flush rate, and the compaction rate. The processor (110) can determine the compaction size for the corresponding period based on the compaction rate, the latency until the level 0 stall threshold size is reached, and the level 0 compaction latency for the next period. Then, the processor (110) can select the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle and perform compaction to the next level.

[0047] In addition, the processor (110) can retrieve the object key of the lookup request from memory (120) and storage (130) in response to a lookup request from a user.

[0048] According to various embodiments, the processor (110) can execute computer-readable code (e.g., software) stored in memory (120) and instructions triggered by the processor (110). The processor (110) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. Desired operations may include, for example, code or instructions included in a program. The data processing device implemented in hardware may include, for example, a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA).

[0049] The memory (120) may store a MemTable and an Immutable MemTable in which data consisting of key-value pairs is temporarily stored. This memory (120) is a volatile memory and may be implemented within the same package as the processor (110), but is not limited thereto. Volatile memory may represent a memory device that stores data only when power is supplied and loses the stored data when the power supply is cut off. For example, volatile memory may include at least one of Static Random Access Memory (SRAM) and Dynamic Random Access Memory (DRAM).

[0050] Additionally, the memory (120) may store at least one instruction (e.g., a program) executable by the processor (110). For example, the instructions may include instructions for executing the operation of the processor (110) and / or the operation of each component of the processor (110).

[0051] Storage (130) may include a plurality of sorted string tables (SSTables) in one or more key ranges to store data flushed from memory. This storage (130) may include a storage device that is distinct from memory (120) and has a capacity greater than that of memory (120). For example, storage (130) may include at least one of a solid state drive (SSD) and a hard disk drive (HDD).

[0052] FIG. 2 exemplarily illustrates the structure of a key-value store of a device that supports dynamic compaction of a key-value store by considering a workload pattern according to one embodiment.

[0053] The processor (110) can perform storage for key-value pairs based on a data structure, for example, a Log Structured Merge-Tree (LSM-Tree). Accordingly, the structure (210) of the key-value store stored in the storage (130) has multiple levels (e.g., L0 to L n-1 , where n is the number of levels) can have. Among these levels, L0 maintains the sorted order of a sorted string table (SSTable) so that the MemTable is effectively flushed and can contain multiple different key ranges within this SSTable.

[0054] FIG. 3 exemplarily shows a compaction bit array showing whether compaction occurs between L0 and L1 in a device that supports compaction of a key-value store dynamically in consideration of a workload pattern according to one embodiment.

[0055] The processor (110) can perform compaction for Level 0 (L0) based on a fixed number of key ranges. For example, when performing L0-L1 compaction, one compaction can handle the data corresponding to one key range with a preset default value. A compaction bit array may be additionally used for compaction. The compaction bit array can indicate whether compaction between L0 and L1 is performed in correspondence with each SSTable. The processor (110) can perform compaction for L0-L1 in key range units using a round-robin method. FIG. 3 illustrates each SSTable and the corresponding compaction bit array in the state before compaction (310), and each SSTable and the corresponding compaction bit array in the state after compaction (320). For each SSTable, the processor (110) [in the order of] key ranges that are 0 Starting from, the compaction bit can be checked. A compaction bit with a value of 0 may indicate that it has not yet been compacted. In response to a compaction bit with a value of 0, the processor (110) may perform compaction on the corresponding KV pair. After compaction is performed, the processor (110) may modify the corresponding bit (321) (e.g., compaction bit) to 1. Additionally, a compaction bit with a value of 1 may indicate that it has already been compacted. The processor (110) may not perform separate compaction on the KV pair corresponding to the compaction bit with a value of 1 (e.g., skip compaction for the corresponding KV pair). If the compaction bit of the entire array is marked as 1, the processor (110) may delete the file of the corresponding SSTable.

[0056] For the entire level, the total key space can be divided into key ranges. For example, Figure 2 illustrates an example divided into eight key ranges. And among the levels, L0 and L n-1 Intermediate levels excluding (e.g., L1 to L n-2 Each level of ) (e.g., i-th level L i , where i is an integer greater than 0 and less than n-1) is divided into a certain number of sub-levels. In each sub-level (e.g., each of the four sub-levels S0 to S3 in FIG. 2), KV pairs within the same key range can be sorted and stored as one or more SSTable files. KV pairs within a given key range can be distinguished from KV pairs in other key ranges. Therefore, SSTable files belonging to the same sub-level may not overlap with each other. In other words, intermediate levels (e.g., L1 to L n-2 ) is each key range( Sub-levels (S0 to S) by ) p-1 , where p is an integer representing the number of sub-levels) can have L1 to L n-2 The structure of the level may mean including multiple SSTables corresponding to sub-levels within each of the key ranges. For example, referring to FIG. 2, the key ranges of Level 1 (L1) (e.g., ... Any one of the key ranges (e.g., SSTable files may be included for each of the multiple sub-levels (e.g., four sub-levels S0 through S3). Conversely, the L0 structure may contain multiple key ranges within each SSTable. Additionally, key ranges may be logically divided to avoid adding flush overhead. The overall key ranges may be configured using an initially flushed file so that the size of the data contained in each key range is uniform. Each level may have a different number of key ranges, and for example, the number of key ranges may double for the next level (e.g., L0 has 4 key ranges and L1 has 8 key ranges).

[0057] Additionally, the processor (110) can perform compaction based on the compaction size for the corresponding cycle recommended by the compaction size recommender (CSR) (220) for levels after level 0. Additionally, among the levels, L0 and L n-1 In levels excluding those, the processor (110) can perform compaction on the data of the sub-levels. The processor (110) can perform compaction by selecting a number of key ranges corresponding to the compaction size of the corresponding period recommended by the compaction size recommender (CSR) (220).

[0058] To determine the compaction size of the cycle, first, the processor (110) may determine the flush rate as a value based on the amount of data flushed during a predetermined time (e.g., 5 seconds). Additionally, the processor (110) may determine the compaction rate as a value obtained by dividing the total compaction size up to date by the total compaction time up to date. If the flush rate is fast, the cycle may be a write-intensive workload, and if the flush rate is slow, the cycle may be a read-intensive workload.

[0059] Afterward, the processor (110) can determine the compaction size of the corresponding cycle using mathematical formula 1. The compaction size of the corresponding cycle can be determined by multiplying the compaction speed (C) by the difference between the latency (A) before reaching the level 0 stall threshold size and the level 0 compaction latency (B) of the next cycle.

[0060]

[0061] Accordingly, the processor (110) can calculate the latency (A) until the level 0 stall threshold size is reached based on the predetermined level 0 stall threshold size (T), the level 0 size (M0), and the flush speed (F). The latency (A) until the level 0 stall threshold size is the difference between the predetermined level 0 stall threshold size (T) and the level 0 size (M0) divided by the flush speed (F). For example, if the level 0 stall threshold size (T) is 800 MB, the level 0 size (M0) is 400 MB, and the flush speed is 80 MB / s, the latency (A) until the level 0 stall threshold size is reached may be 5 seconds. Latency (A) until reaching this level 0 stall threshold size depends on the query rate, so it can have a low value for write-intensive workloads and a high value for read-intensive workloads.

[0062] The processor (110) can calculate the next cycle's level 0 compaction latency (B) based on the next cycle's level 0 compaction size (RS0) and compaction rate (C). The next cycle's level 0 compaction latency (B) may be the next cycle's level 0 compaction size (RS0) divided by the compaction rate (C). The next cycle's level 0 compaction size (RS0) may be the next cycle's level 0 stall threshold size (T) divided by the number of key ranges (r0) at level 0. For example, if the level 0 stall threshold size (T) is 800 MB, the number of key ranges (r0) at level 0 is 4, and the compaction rate (C) is 100 MB / s, the next cycle's level 0 compaction latency (B) may be 2 seconds. The number of key ranges (r0) at level 0 may represent a parameter value that is a predetermined fixed value. Accordingly, the compaction size of the corresponding cycle may be 300 MB, which is the difference between the latency (A) before reaching the level 0 stall threshold size and the level 0 compaction latency (B) of the next cycle (e.g., 5 seconds - 2 seconds = 3 seconds) multiplied by the compaction speed (C) (e.g., 100 MB / s).

[0063] Accordingly, the processor (110) calculates the latency until the level 0 stall threshold size is reached and the level 0 compaction latency for the next cycle based on the level 0 size, the flush rate and the compaction rate, and determines the compaction size for the corresponding cycle based on the compaction rate, the calculated latency until the level 0 stall threshold size is reached and the level 0 compaction latency for the next cycle, so that it may have a small compaction size for write-intensive workloads and a large compaction size for read-intensive workloads.

[0064] FIG. 4 exemplarily illustrates a process of selecting the number of key ranges at a corresponding level based on the compaction size of the corresponding cycle determined in a device that dynamically supports compaction of a key-value store considering a workload pattern according to one embodiment.

[0065] A key range may be a basic compaction unit. Accordingly, the processor (110) may select the number of key ranges for a corresponding level (420) corresponding to the compaction size (410) of the corresponding cycle, for example, 300 MB (for example, a size smaller than or equal to the compaction size (410) of the corresponding cycle). A round-robin method may be used to select the number of key ranges for compaction at each level. Additionally, for this purpose, the processor (110) may track the size of each key range and use the tracked size of each key range when performing compaction. For example, key range 1 (R 1 The size of ) is 100MB, and the key range is 2(R 2 Since the size of ) is 150MB, the processor (110) has two key ranges (R 1 , R 2 ) can be selected. Accordingly, the processor (110) can select the corresponding level (L) provided by the compaction size recommender (CSR) (220). i Two key ranges of ) (R 1 , R 2 Based on ), the next level (L i+1 Compaction into ) can be performed.

[0066] FIG. 5 illustrates an exemplary compaction process between L1 and L2 in a device that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment.

[0067] For example, if compaction between L1 and L2 is performed, the processor (110) uses two key ranges recommended by the compaction size recommender (CSR) (220) through the process of FIG. 4 described above ( , ) can be selected as the compaction target (510). The processor (110) has two key ranges of L1 ( , ) My data (e.g.: of S0 to S3 and Merge and sort the KV pairs stored in S0 to S3 of L2's key range ( inside SSTables (522) for each key range can be created by partitioning based on ) (520). Then, the processor (110) can place the created SSTables (522) in each key range (530). Since the created SSTables (522) are placed on top of the previously stored SSTables (523), they can have different placement locations for each key range. For example, a key range ( The placement position in ) is S2, and the key range( , The placement location in ) is S3, and the key range( The placement location in ) can be S0.

[0068] Additionally, the processor (110) can use a compaction size recommender (CSR) (220) to check whether compaction can be performed before determining the compaction size for the corresponding cycle. The processor (110) can check levels after level 0 (e.g., L1 to L n-2 Before performing compaction in ), you can check the availability of sub-levels of each key range within the next level where compaction is performed. L n-1 Regarding this, the processor (110) L n-1 Since it does not have this sub-level, L n-1The availability of each key range's SSTable can be checked. Checking the availability of a sub-level may mean checking for the presence of an empty space (e.g., 521 in FIG. 5) at the highest position of the sub-level according to the storage characteristics of the sub-levels stored sequentially from sub-level S0 to S3. Accordingly, if the sub-level is available, the processor (110) can perform compaction based on the compaction size determined using the compaction size recommender (CSR) (220). Additionally, if the sub-level is not available, the processor (110) can perform compaction by checking the availability of the sub-level of the next level after that level. For example, when performing compaction between L1 and L2, the processor (110) can determine whether compaction is possible by identifying the usage status of each key range's SSTable in L2. Accordingly, if there are insufficient available sub-levels within L2, the processor (110) may attempt to perform compaction between L2 and L3.

[0069] FIG. 6 illustrates a flowchart of a method for dynamically supporting compaction of a key-value store by considering a workload pattern according to one embodiment.

[0070] In step (610), the processor can detect the size of level 0 (zero) (L0) where flushing is performed, the rate of flushing, and the compaction rate for compaction performed periodically through a compaction size recommender. The rate of flushing can be determined as a value based on the amount of data flushed during a predetermined time. Additionally, the compaction rate can be determined as a value obtained by dividing the total compaction size up to date by the total compaction time up to date.

[0071] In step (620), the processor can calculate the latency until the level 0 stall threshold size is reached and the level 0 compaction latency for the next cycle based on the level 0 size, the flush rate, and the compaction rate.

[0072] In step (630), the processor may determine the compaction size of the corresponding cycle based on the compaction rate, the latency before reaching the level 0 stall threshold size, and the level 0 compaction latency of the next cycle. The compaction size of the corresponding cycle may be determined by multiplying the compaction rate by the difference between the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle.

[0073] In step (640), the processor can perform compaction to the next level by selecting the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle.

[0074] In step (620), the processor may calculate the latency until the level 0 stall threshold size is reached based on the predetermined level 0 stall threshold size and the level 0 size and flush rate, and calculate the level 0 compaction latency for the next cycle based on the level 0 compaction size and compaction rate for the next cycle. The level 0 compaction size for the next cycle may be the level 0 stall threshold size divided by the number of key ranges at level 0.

[0075] In step (640), the processor may perform compaction for level 0 with a fixed number of key ranges and for levels after level 0 based on the compaction size of the corresponding cycle. In addition, the processor may check the availability of the sub-level to be compacted and stored before performing compaction on levels after level 0. If the sub-level is available, the processor may perform compaction based on the compaction size determined using a compaction size recommender, and if the sub-level is not available, it may check the availability of the sub-level of the next level of that level and perform compaction.

[0076] The processor can perform storage of key-value pairs based on a Log Structured Merge-Tree (LSM-Tree) data structure.

[0077] FIG. 7 shows a graph of normalized throughput according to write / read workload of a method that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment.

[0078] The Y-axis of the graph represents the throughput of a method that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment, normalized to the throughput of a conventional compaction method. Additionally, the X-axis of the graph represents a workload of 100% write (Load), a workload of 50% write and 50% read (W50), a workload of 5% write and 95% read (W5), and a workload of 100% read (W0), respectively, for a Uniform distribution and a Zipfian distribution. As can be seen in FIG. 7, the method that dynamically supports compaction of a key-value store by considering a workload pattern according to one embodiment generally shows a result (710) of a value significantly greater than 1 for any workload, so it can be confirmed that it has superior performance compared to a conventional compaction method.

[0079] This means that by selecting the number of key ranges at the corresponding level corresponding to the compaction size of the cycle determined by the compaction size recommender and performing compaction to the next level, write stalls in write-intensive workloads can be reduced compared to existing LSM tree-based key-value stores, thereby improving write performance, and in read-intensive workloads, the number of levels where data resides can be reduced by setting a large lower-level compaction size, thereby reducing the number of disk IO operations caused by read queries and improving read performance as well.

[0080] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0081] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0082] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0083] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0084] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0085] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A device that supports dynamic compaction of a key-value store by considering workload patterns, comprising: a memory storing a MemTable and an Immutable MemTable in which data consisting of key-value pairs is temporarily stored; and a storage comprising a plurality of sorted string tables (SSTables) in one or more key ranges to store the data flushed from the memory. and includes a processor that performs a compaction operation and a flush operation to compact data within a plurality of SSTables existing at one level within the storage to the next level, wherein the processor detects, for compaction performed periodically through a compaction size recommender, the size of Level 0 (zero) (L0) where flush is performed, a flush rate based on the amount of data flushed during a predetermined time, and a compaction rate based on the value obtained by dividing the total compaction size to date by the total compaction time to date, and calculates the latency until a Level 0 stall threshold size is reached and the Level 0 compaction latency for the next period based on the size of Level 0, the flush rate, and the compaction rate, determines the compaction size for the corresponding period based on the compaction rate, the latency until a Level 0 stall threshold size is reached, and the Level 0 compaction latency for the next period, and a key of the corresponding level corresponding to the compaction size of the corresponding period. A device that selects the number of ranges to perform compaction to the next level. Claim 2 A device according to claim 1, wherein the processor calculates the latency until the level 0 stall threshold size is reached based on a predetermined level 0 stall threshold size, the size of the level 0, and the rate of the flush, and calculates the level 0 compaction latency of the next cycle based on the level 0 compaction size of the next cycle and the rate of the compaction. Claim 3 A device according to claim 1, wherein the processor performs the compaction with a fixed number of key ranges for level 0 and performs the compaction for levels after level 0 based on the compaction size of the corresponding cycle. Claim 4 In paragraph 3, the processor checks whether a sub-level of the level where the compaction is performed and stored is available before performing the compaction at a level after level 0, and if the sub-level is available, performs the compaction based on a compaction size determined using the compaction size recommender, and if the sub-level is not available, checks whether a sub-level of the next level of the level is available and performs the compaction. Claim 5 A device according to claim 1, wherein the flushing speed is determined as a value based on the amount of data flushed during a predetermined time, and the compaction speed is determined as a value obtained by dividing the total compaction size up to the present by the total compaction time up to the present. Claim 6 A device according to claim 1, wherein the level 0 compaction size of the next cycle is a value obtained by dividing the level 0 stall threshold size by the number of key ranges at level 0. Claim 7 A device according to claim 1, wherein the compaction size of the corresponding cycle is determined by multiplying the compaction speed by the difference between the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle. Claim 8 In claim 1, the processor is a device that performs storage for key-value pairs based on a Log Structured Merge-Tree (LSM-Tree) data structure. Claim 9 A method for dynamically supporting compaction of a key-value store by considering workload patterns performed by a processor, comprising: a step of detecting, for compaction performed periodically through a compaction size recommender, the size of a level 0 (zero) (L0) where a flush is performed, a flush rate based on the amount of data flushed during a predetermined time, and a compaction rate based on the value obtained by dividing the total compaction size up to now by the total compaction time up to now; a step of calculating the latency until a level 0 stall threshold size is reached and the level 0 compaction latency for the next cycle based on the size of the level 0, the flush rate, and the compaction rate; and a step of determining the compaction size for the corresponding cycle based on the compaction rate, the latency until a level 0 stall threshold size is reached, and the level 0 compaction latency for the next cycle. A method comprising the step of selecting the number of key ranges of the corresponding level corresponding to the compaction size of the corresponding cycle and performing compaction to the next level. Claim 10 In claim 9, the step of calculating the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle comprises the step of calculating the latency before reaching the level 0 stall threshold size based on a predetermined level 0 stall threshold size, the size of the level 0, and the rate of the flush, and calculating the level 0 compaction latency of the next cycle based on the level 0 compaction size of the next cycle and the compaction rate. Claim 11 A method according to claim 9, wherein the step of performing the compaction comprises performing the compaction with a fixed number of key ranges for level 0, and performing the compaction for levels after level 0 based on the compaction size of the corresponding cycle. Claim 12 In claim 11, the step of performing the compaction further comprises: a step of checking whether the sub-level where the compaction is performed and stored is available before performing the compaction at a level after level 0; and, if the sub-level is available, performing the compaction based on a compaction size determined using the compaction size recommender, and if the sub-level is not available, checking whether the sub-level of the next level of the corresponding level is available and performing the compaction. Claim 13 A method according to claim 9, wherein the flush rate is determined as a value based on the amount of data flushed during a predetermined time, and the compaction rate is determined as a value obtained by dividing the total compaction size up to the present by the total compaction time up to the present. Claim 14 In claim 9, the method wherein the level 0 compaction size of the next cycle is the value obtained by dividing the level 0 stall threshold size by the number of key ranges at level 0. Claim 15 A method according to claim 9, wherein the compaction size of the corresponding cycle is determined by multiplying the compaction rate by the difference between the latency before reaching the level 0 stall threshold size and the level 0 compaction latency of the next cycle. Claim 16 In claim 9, the processor performs storage for key-value pairs based on a log-structured merge-tree (LSM-Tree) data structure.

Images