Data attribute-aware storage system and data management method for key-value database

Abstract

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

Description

Key-value database storage system and data management method based on data attribute perception TECHNICAL FIELD

[0001] The present application relates to the field of storage technology, specifically the field of key-value databases, and a key-value database storage system and data management method based on data attribute perception. BACKGROUND

[0002] LSM-tree is a widely adopted data structure in many modern storage systems. It is designed specifically for managing high write loads and is widely used in databases, key-value stores, and distributed storage. A key feature of this data structure is the conversion of write operations into sequential writes. This is achieved by appending new data to a structure called Write-Ahead Log (WAL) instead of writing directly to the main storage area. When the space at a certain level is completely occupied, a compaction process will be initiated to compress all overlapping key-value items to adjacent lower levels. However, each compaction operation usually involves reading and rewriting a large amount of data. Therefore, the amount of data actually written to the storage device can be much larger than the input data, resulting in a serious I / O amplification effect. Key-value storage systems based on LSM-tree mainly adopt a two-tier structure, including a volatile memory component and a persistent disk component.

[0003] MemTable is the main in-memory data structure in LSM-Tree. It serves as a temporary write buffer to efficiently handle incoming key-value pairs before they are persisted to disk as SSTable. Typically, MemTable is implemented as a sorted data structure, often using skip lists, red-black trees, or other ordered data structures. When MemTable accumulates a certain amount of data or reaches a specified capacity threshold, it will transform into an immutable MemTable. At this point, the contents of the MemTable become immutable or read-only. This process marks the transition of data from volatile memory space to a more persistent form, preparing for eventual storage as SSTable on disk.

[0004] For the disk component, SSTable is usually stored as a single file on disk and serves as the basic storage unit for LSM-tree-based key-value stores. SSTable is divided into index blocks and data blocks. Index blocks contain pointers or offsets to the locations of key-value pairs within data blocks. This index facilitates fast lookup and retrieval of specific keys. To reduce disk space usage and improve I / O efficiency, SSTable may implement compression techniques and encoding methods. These methods aim to minimize storage overhead while maintaining data integrity.

[0005] When a MemTable reaches a certain size limit or a preconfigured entry threshold, a flush process is triggered. This threshold ensures that the MemTable does not become too large, thus addressing potential memory overflow issues. The flush mechanism periodically transfers the contents of the MemTable to disk, converting it into a new SSTable. This transition ensures the durability and stability of the most recently written data.

[0006] When the system reaches a pre-set SSTable threshold or the total amount of data on disk exceeds a certain limit, a merge process is triggered to optimize storage efficiency, reduce disk fragmentation, and enhance read performance by merging and reorganizing SSTables. During this process, the system identifies overlapping or redundant data in SSTables. It selects key-value pairs from different SSTables, resolves conflicts, and generates a merged SSTable with no redundant entries. Since the merge process involves sorting and organizing data, it requires the relocation and reorganization of data from different SSTables. These processes result in a larger amount of data actually written than initially inserted, leading to significant write amplification.

[0007] In LSM-trees, data is only sorted by the dictionary order of keys, and cold and hot data with different update frequencies are mixed together within SSTables. A small group of frequently updated KV data can repeatedly trigger merge operations for some SSTables, leading to repeated writing of cold data in SSTables to new SSTables, resulting in significant I / O overhead. Merge operations quickly disperse these hot data to other SSTables at the same level, polluting a large number of SSTables with cold and hot data, further amplifying the amplification effect.

[0008] In actual workloads, key value ranges tend to be clustered, with some ranges of keys being dense and others being sparse. A small portion of SSTables with excessively sparse keys can cause them to contain too many lower-level SSTables during merging, triggering a lot of unnecessary data rewriting and resulting in significant I / O amplification.

[0009] It should be noted that the information disclosed in the above BACKGROUND section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art.

[0010] SUMMARY

[0011] To address the problems existing in the prior art, this invention provides a key-value database storage system and data management method based on data attribute awareness. By analyzing attributes such as data popularity and density, data with different attributes are separated into different data partitions, thereby reducing I / O amplification caused by repeated read and write operations during compression.

[0012] To address the aforementioned technical problem, the present invention employs the following technical solution: a data attribute-aware key-value database storage system, comprising a K-Means classifier, a memory component, and a disk component. The K-Means classifier is connected to write requests and is used to record and periodically update frequently updated hot keys within the current database. The memory component includes a cold data area and a hot data area. The cold data area includes a cold data memory table (MemTable) and a cold data immutable memory table (Immutable MemTable). The hot data area includes a hot data memory table (MemTable) and a hot data immutable memory table (Immutable MemTable). The disk component includes a cold SSTable and a hot SSTable. The K-Means classifier clusters key-value pairs in write requests into cold key-value pairs and hot key-value pairs, and then stores them in the corresponding locations of the memory component.

[0013] Furthermore, the update frequency of the key is recorded to represent the data popularity of key-value pairs. The update frequency of the key is represented by the number of write accesses to the database within a time period.

[0014] Furthermore, in the disk component, a prefix is ​​added to the hot key-value pairs so that the hot key-value pairs and cold key-value pairs form a non-overlapping key-value range, and the cold key-value pairs and hot key-value pairs are classified into cold SSTable and hot SSTable.

[0015] Furthermore, the merging process of the key-value database is modified. After the key-value pairs in the SSTable are extracted and sorted into a sequence, the key-value pairs are taken out one by one and the binary difference between any two adjacent key-value pairs is calculated. If the difference is greater than a set threshold, the current SSTable is stopped from being filled and a new SSTable is created.

[0016] Furthermore, the set thresholds are dynamically adjusted based on workload characteristics, and the SSTable splitting thresholds are determined through statistical analysis of load characteristics.

[0017] Furthermore, the average density of the SSTable at level 0 is calculated periodically, and the segmentation threshold for each level is calculated based on the statistical results. The specific calculation formula is as follows:

[0018] Where Threshold(i) represents the SSTable splitting threshold of the i-th layer, α i The coefficient representing the splitting threshold of the i-th layer, AVGdensity (L0) represents the average density of the 0th layer SSTable, Diff(i) represents the difference value split threshold, and β i represents the coefficient of the i-th layer difference value split threshold.

[0019] The application further discloses a key-value database data management method based on data attribute perception, which comprises two parts of data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing is to use a one-dimensional K-Means clustering algorithm to cluster data into two clusters, indicating that the key is determined to be cold or hot. Meanwhile, the memory component comprises a cold data area and a hot data area. The cold data area comprises a cold data memory table MemTable and a cold data immutable memory table Immutable MemTable. The hot data area comprises a hot data memory table MemTable and a hot data immutable memory table Immutable MemTable. The disk component comprises a cold SSTable and a hot SSTable. The cold and hot data are separated and stored through the data hotness attribute-based writing strategy. The data density attribute-based merging is to extract key-value pairs in the SSTable and form a sorted key-value pair sequence after a merging operation, then take out the key-value pairs one by one and calculate the binary difference value between each two adjacent key-value pairs. If the difference value is greater than a set threshold, the current SSTable is stopped from being filled, and a new SSTable is created.

[0020] Further, the data hotness of the key-value pair is represented by recording the update frequency of the key, and the update frequency of the key is represented by the number of write access times of the database in a time period.

[0021] Further, the set threshold is dynamically adjusted based on workload characteristics, and the split threshold of the SSTable is determined by statistical analysis of load characteristics.

[0022] Further, the average density of the 0th layer SSTable is calculated periodically, and the split threshold of the SSTable is calculated according to the statistical result.

[0023] The split threshold of each level is calculated according to the following formula,

[0024] wherein Threshold(i) represents the i-th layer SSTable split threshold, and α i represents the coefficient of the i-th layer split threshold, and AVG density (L0) represents the average density of the 0th layer SSTable, Diff(i) represents the difference value split threshold, and β i represents the coefficient of the i-th layer difference value split threshold.

[0025] The application has the beneficial effects of identifying and processing special properties of key-value data, isolating data of different properties, thereby reducing invalid data rewriting in the merging process, and greatly reducing the read-write amplification of the key-value database. By extracting workload characteristics, the storage separation granularity is adaptively adjusted for different workloads, so that our storage separation architecture is suitable for various environments.

[0026] The above general description and the following description are only exemplary and explanatory, and are not used to limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0027] One or more embodiments are exemplified by the accompanying drawings corresponding thereto, which do not constitute a limitation on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute a proportional limit, and wherein:

[0028] Figure 1 is a schematic block diagram of the storage system according to embodiment 1;

[0029] Figure 2 is a schematic diagram of a density-aware merging process. DETAILED DESCRIPTION

[0030] In order to be able to understand the features and technical content of the embodiments of the present disclosure more fully, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0031] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0032] Unless otherwise specified, the term "a plurality of" means two or more.

[0033] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0034] The term "and / or" is a description of the relationship between the objects, which means that there can be three relationships. For example, A and / or B, which means: A or B, or, A and B, the three relationships.

[0035] The term "corresponding" can refer to a kind of association or binding relationship, A corresponding to B means that there is an association or binding relationship between A and B.

[0036] Embodiment 1

[0037] This embodiment discloses a key-value database storage system based on data attribute awareness, as shown in FIG. 1, which includes a K-Means classifier, a memory component and a disk component. The K-Means classifier is connected with a write request, used to record and regularly update the hot keys frequently updated in the current database. The memory component includes a cold data area and a hot data area. The cold data area includes a cold data MemTable and a cold data Immutable MemTable. The hot data area includes a hot data MemTable and a hot data Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The key-value pairs in the write request are clustered into cold key-value pairs and hot key-value pairs through the K-Means classifier, and then stored in the corresponding positions of the memory component.

[0038] The storage system described in this embodiment is similar to the traditional LSM-tree design. We absorb small random key-value pair writes by reserving MemTable and Immutable MemTable as buffers and convert them into large, sequential disk writes. In order to support the distinction between cold and hot, we maintain a K-Means classifier to record and regularly update the hot keys frequently updated in the current database. At the same time, we maintain a set of MemTable and Immutable MemTable for cold and hot keys respectively, so as to separate the storage of cold and hot keys in the database.

[0039] This embodiment does not make excessive modifications to the disk component. Since cold and hot keys are placed in different MemTable, in order to distinguish the cold and hot keys in the disk, we add a prefix to the hot key, so as to form a non-overlapping key-value range with the cold key-value pair. Therefore, cold and hot keys will be naturally separated into different SSTable. On the other hand, since the key-value pairs are separated according to their density attributes, the SSTable generated at the first level and deeper levels have different sizes.

[0040] This embodiment records the update frequency of the key to represent the data hotness of the key-value pair. The update frequency of the key is represented by the number of write accesses to the database within a time period. A one-dimensional K-Means clustering algorithm is used to cluster the data into two clusters. The two areas respectively represent that the key is determined to be cold or hot. For this part of the hot key-value pairs, since the number is smaller compared with the cold key-value pairs, we add a byte of special character prefix to the key to realize separate storage with the cold key-value pairs in the database.

[0041] For the data density property, we measure the sparsity between keys by converting them into binary values and calculating their difference. A larger difference indicates that two keys are more sparse, suggesting that they need to be separated. To minimize the computational cost, we modify the merge process of the key-value database. As shown in Fig. 2, after the merge operation extracts key-value pairs from SSTable and forms a sorted sequence of key-value pairs, we take out the key-value pairs one by one and calculate the binary difference between each two adjacent key-value pairs. If the difference is greater than the set threshold, we stop filling the current SSTable and create a new SSTable. Thus, we achieve storage separation based on data density.

[0042] For the merge process for data density, our goal is to split SSTable to alleviate write amplification. However, the density between key-value pairs in SSTable varies with the workload. To ensure the adaptability of the method of splitting key-value pairs based on density between different workloads, we propose a dynamic threshold adjustment mechanism based on workload characteristics, which determines the splitting threshold of SSTable by statistical analysis of load characteristics. That is, the set threshold is dynamically adjusted based on workload characteristics, and the splitting threshold of SSTable is determined by statistical analysis of load characteristics.

[0043] Specifically, we periodically calculate the average density of SSTable at level 0 and calculate the splitting threshold of each level according to the statistical results. The specific calculation formula is:

[0044] where Threshold(i) represents the splitting threshold of SSTable at level i, a i represents the coefficient of the splitting threshold at level i, AVG density (L0) represents the average density of SSTable at level 0, Diff(i) represents the difference value splitting threshold, and β i represents the coefficient of the difference value splitting threshold at level i.

[0045] Since LSM-tree is constantly merged to deeper levels, SSTable at deeper levels is more dense and requires a smaller splitting threshold, so the coefficient decreases as the level increases.

[0046] Embodiment 2

[0047] The embodiment discloses a key-value database data management method based on data attribute perception, including two parts of data hotness attribute-based writing and data density attribute-based merging, the data hotness attribute-based writing is to use a one-dimensional K-Means clustering algorithm to cluster data into two clusters, the two regions respectively represent that the key is determined to be cold or hot, and meanwhile, a memory component includes a cold data area and a hot data area, the cold data area includes a cold data MemTable and a cold data Immutable MemTable, the hot data area includes a hot data MemTable and a hot data Immutable MemTable, a disk component includes a cold SSTable and a hot SSTable, and the cold and hot data are separately stored through the data hotness attribute-based writing strategy; the data density attribute-based merging is to extract key-value pairs in the SSTable and form a sorted key-value pair sequence after a merging operation, then the key-value pairs are taken out one by one, and a binary difference value between each two adjacent key-value pairs is calculated, if the difference value is greater than a set threshold value, then filling the current SSTable is stopped, and a new SSTable is created.

[0048] In the embodiment, the update frequency of the key is recorded to represent the data hotness of the key-value pair, and the update frequency of the key is represented by the write access times of the database in a time period.

[0049] In the embodiment, the set threshold value is dynamically adjusted based on workload characteristics, and the split threshold value of the SSTable is determined through statistical analysis of the load characteristics. Specifically, the average density of the 0th SSTable is calculated periodically, and the split threshold value of each level is calculated according to the statistical result, and the specific calculation formula is as follows:

[0050] Wherein, Threshold(i) represents the SSTable split threshold value of the ith layer, alpha i represents the coefficient of the split threshold value of the ith layer, AVG density (L0) represents the average density of the 0th SSTable, Diff(i) represents the difference split threshold value, beta i represents the coefficient of the difference split threshold value of the ith layer.

[0051] The detailed implementation process of the application is given below, and the specific calculation process of the key-value storage separation strategy based on data attributes proposed by the application is further described in detail in combination with the algorithm pseudo code.

[0052] 1. Data hotness attribute-based writing strategy

[0053] The data hotness attribute-based writing strategy proposed by the application realizes the separate storage of cold and hot data, reduces the merging pressure of the key-value database, and improves the overall throughput of the database.

[0054] 2. Merge based on data density attribute

[0055] The merge based on data density attribute realizes the separate storage of sparse data, reduces the rewriting of invalid data in the key-value database merge operation, and reduces the write amplification of the database.

[0056] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments are merely representative of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be varied. Parts and features of some embodiments can be included or substituted for parts and features of other embodiments. Also, the words used in this application are for describing the embodiments and are not intended to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising," etc., mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element. In this document, each embodiment can focus on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the methods, products, etc. disclosed by the embodiments, if they correspond to the method part disclosed by the embodiments, the relevant parts can be referred to the description of the method part.

Claims

1. A key-value database storage system based on data attribute awareness, characterized in that: The method comprises a K-Means classifier, a memory component and a disk component, the K-Means classifier is connected with a write request, and is used for recording and regularly updating hot keys frequently updated in a current database, the memory component comprises a cold data area and a hot data area, the cold data area comprises a cold data MemTable and a cold data Immutable MemTable, the hot data area comprises a hot data MemTable and a hot data Immutable MemTable, and the disk component comprises a cold SSTable and a hot SSTable; the key-value pairs in the write request are clustered into cold key-value pairs and hot key-value pairs through the K-Means classifier, and then are stored in corresponding positions of the memory component.

2. The data attribute awareness based key-value database storage system of claim 1, wherein: The data hotness of the key-value pairs is represented by recording the update frequency of the keys, and the update frequency of the keys is represented by the write access times of the database in a time period.

3. The data attribute awareness based key-value database storage system of claim 1, wherein: In the disk component, prefixes are added to the hot key-value pairs, so that the hot key-value pairs and the cold key-value pairs form non-overlapping key-value ranges, and the cold key-value pairs and the hot key-value pairs are divided into the cold SSTable and the hot SSTable.

4. The data attribute awareness based key-value database storage system of claim 1, wherein: The merging process of the key-value database is modified, after the key-value pairs in the SSTable are extracted and a sorted key-value pair sequence is formed in the merging operation, the key-value pairs are taken out one by one, and the binary difference between each two adjacent key-value pairs is calculated, if the difference is greater than a set threshold, the filling of the current SSTable is stopped, and a new SSTable is created.

5. The data attribute awareness based key-value database storage system of claim 4, wherein: The set threshold is dynamically adjusted based on workload characteristics, and the splitting threshold of the SSTable is determined by statistical analysis of load characteristics.

6. The data attribute awareness based key-value database storage system of claim 5, wherein: The average density of the 0th SSTable is calculated periodically, and the split threshold of each level is calculated according to the statistical result, and the specific calculation formula is as follows, where Threshold(i) represents the SSTable split threshold of the ith level, a i represents the split of the ith level coefficient of the threshold value, AVG density (L0) represents the average density of the 0th layer SSTable, Diff(i) indicates the difference partition threshold value, β i represents the coefficient of the i-th layer difference partition threshold value.

7. A data management method for a key-value database based on data attribute awareness, characterized in that: The method comprises two parts of write based on data hotness attribute and merge based on data density attribute, the write based on data hotness attribute is to cluster data into two clusters by using a one-dimensional K-Means clustering algorithm, so that the keys are determined as cold or hot, meanwhile, the memory component comprises a cold data area and a hot data area, the cold data area comprises a cold data MemTable and a cold data Immutable MemTable, the hot data area comprises a hot data MemTable and a hot data Immutable MemTable, the disk component comprises a cold SSTable and a hot SSTable, and the write based on data hotness attribute is used to realize the separate storage of cold and hot data; the merge based on data density attribute is to take out the key-value pairs one by one after the key-value pairs in the SSTable are extracted and a sorted key-value pair sequence is formed in the merging operation, and the binary difference between each two adjacent key-value pairs is calculated, if the difference is greater than a set threshold, the filling of the current SSTable is stopped, and a new SSTable is created. 8.The data attribute-aware based key-value database data management method of claim 7, wherein: The data hotness of the key-value pairs is represented by recording the update frequency of the keys, and the update frequency of the keys is represented by the write access times of the database in a time period.

9. The data attribute-aware based key-value database data management method of claim 7, wherein: The set threshold is dynamically adjusted based on workload characteristics, and the splitting threshold of the SSTable is determined by statistical analysis of load characteristics.

10. The data attribute-aware based key-value database data management method of claim 7, wherein: The average density of the 0th SSTable is calculated periodically, and the split threshold of each level is calculated according to the statistical result, and the specific calculation formula is as follows, where Threshold(i) represents the SSTable split threshold of the i-th level, a i represents the coefficient of the split threshold of the i-th level, AVG density (L0) represents the average density of the SSTable of the 0-th level, Diff(i) indicates the difference split threshold, β i represents the coefficient of the difference split threshold of the i-th level.

Images