Key-Value Database Storage System and Data Management Method Based on Data Attribute Perception

Abstract

This application relates to the field of storage technology, and discloses a key-value database storage system and a data management method based on data attribute perception, including two parts: writing based on data heat attribute and merging based on data density attribute. The writing based on data heat attribute means dividing key-value pairs into hot and cold attributes, and realizing the separated storage of hot and cold data through the writing strategy based on data heat attribute; the merging based on data density attribute is to extract key-value pairs in the SSTable and form a sorted key-value pair sequence during the merge operation, then take out key-value pairs one by one and calculate the binary difference between every two adjacent key-value pairs. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable. By analyzing data heat and density, the present invention separates data with different attributes into different data partitions, thereby reducing the I / O amplification caused by repeated reading and writing during the compression process.

Description

Technical Field

[0001] This application relates to the field of storage technology, specifically to the field of key-value databases, and a key-value database storage system and data management method based on data attribute perception. Background Art

[0002] The LSM-tree is a data structure widely adopted in many modern storage systems. It is designed specifically for managing high write loads and is widely used in databases, key-value stores, and distributed storage. Each merge operation of the LSM-tree usually involves reading and rewriting a large amount of data. Therefore, the amount of data actually written to the storage device may be much larger than the input data, resulting in a serious I / O amplification effect.

[0003] The MemTable is the main in-memory data structure in the LSM-Tree. It serves as a temporary write buffer and effectively processes incoming key-value pairs before persisting them to disk as SSTables. When the system reaches the preset SSTable threshold or the total data volume on disk exceeds a certain limit, the merge process is triggered, and the read performance is enhanced by merging and reorganizing SSTables. Since the merge process involves sorting and organizing data, it is necessary to relocate and reorganize data from different SSTables. These processes result in a larger amount of data being written than the initially inserted data, leading to a large amount of write amplification.

[0004] In the LSM-tree, data is only sorted in lexicographical order of keys, and cold data and hot data with different update frequencies are mixed together inside the SSTable. A small group of frequently updated KV data may repeatedly cause partial SSTable merge operations, resulting in the continuous rewriting of cold data in the SSTable into new SSTables, thus generating a huge I / O overhead. The merge operation will quickly disperse this hot data into other SSTables at the same level, thus contaminating a large number of SSTables with cold data and hot data, which further amplifies the amplification effect. In actual workloads, key-value ranges often exhibit clustering, with some ranges of keys being dense and others being sparse. A small number of overly sparse keys in an SSTable may cause it to include too many lower-level SSTables during merging, leading to a lot of unnecessary data rewriting and thus a large amount of I / O amplification.

[0005] It should be noted that the information disclosed in the above Background Art section is only used to enhance the understanding of the background of this application, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] In view of the problems existing in the prior art, the present invention provides a key-value database storage system and a data management method based on data attribute perception. By analyzing attributes such as data heat and density, data with different attributes are separated into different data partitions, thereby reducing the I / O amplification caused by repeated reading and writing during the compression process.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is as follows: A key-value database storage system based on data attribute perception includes a K-Means classifier, a memory component, and a disk component. The K-Means classifier is connected to the write request and is used to record and periodically update the hot keys that are frequently updated inside the current database. The memory component includes a cold data area and a hot data area. The cold data area includes a cold data memory table MemTable and a cold data immutable memory table Immutable MemTable. The hot data area includes a hot data memory table MemTable and a hot data immutable memory table Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The key-value pairs in the write request are clustered into cold key-value pairs and hot key-value pairs by the K-Means classifier and then stored in the corresponding positions of the memory component.

[0008] Furthermore, the data heat of the key-value pair is represented by recording the update frequency of the key, and the update frequency of the key is represented by the number of write accesses to the database within a time period.

[0009] Furthermore, in the disk component, a prefix is added to the hot key-value pairs so that the hot key-value pairs and the cold key-value pairs form non-overlapping key-value ranges, and the cold key-value pairs and the hot key-value pairs are divided into the cold SSTable and the hot SSTable.

[0010] Furthermore, the merging process of the key-value database is modified. After the key-value pairs in the SSTable are extracted and formed into a sorted key-value pair sequence during the merge operation, the key-value pairs are taken out one by one and the binary difference between every two adjacent key-value pairs is calculated. If the difference is greater than the set threshold, the filling of the current SSTable is stopped and a new SSTable is created.

[0011] Furthermore, the set threshold is dynamically adjusted based on the workload characteristics, and the splitting threshold of the SSTable is determined by statistical analysis of the load characteristics.

[0012] Furthermore, the average density of the level 0 SSTable is calculated regularly, and the splitting threshold for each level is calculated based on the statistical results. The specific calculation formula is as follows:

[0013] ,

[0014] where Threshold(i) represents the iThe SSTable splitting threshold of the layer, α i represents the i coefficient of the splitting threshold of the layer, AVG density (L 0 ) represents the average density of the SSTable of layer 0, Diff(i) represents the differential splitting threshold, β i represents the i coefficient of the differential splitting threshold of the layer.

[0015] The present invention also discloses a key - value database data management method based on data attribute perception. This method includes two parts: writing based on data heat attributes and merging based on data density attributes. Writing based on data heat attributes uses a one - dimensional K - Means clustering algorithm to cluster data into 2 clusters, indicating that the key is determined to be cold or hot. At the same time, the memory component includes a cold data area and a hot data area. The cold data area includes a cold data memory table MemTable and a cold data immutable memory table Immutable MemTable. The hot data area includes a hot data memory table MemTable and a hot data immutable memory table Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The separate storage of cold and hot data is achieved through the writing strategy based on data heat attributes. Merging based on data density attributes is to extract key - value pairs in the SSTable and form a sorted key - value pair sequence during the merge operation, then take out key - value pairs one by one and calculate the binary difference between every two adjacent key - value pairs. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable.

[0016] Furthermore, the data heat of the key - value pair is represented by recording the update frequency of the key, and the update frequency of the key is represented by the number of write accesses to the database within a time period.

[0017] Furthermore, the set threshold is dynamically adjusted based on the workload characteristics, and the splitting threshold of the SSTable is determined by statistical analysis of the load characteristics.

[0018] Furthermore, regularly calculate the average density of the SSTable of layer 0, and calculate the splitting threshold of each level according to the statistical results. The specific calculation formula is as follows, ,

[0019] where Threshold(i) represents the i SSTable splitting threshold of the layer, α i represents the iCoefficient of layer splitting threshold, AVG density (L 0 ) represents the average density of the 0th layer SSTable, Diff(i) represents the difference splitting threshold, β i represents the i coefficient of the difference splitting threshold of the layer.

[0020] Advantages of the present invention: Identify and process the special attributes of key-value data, isolate data with different attributes, thereby reducing the rewriting of invalid data during the merging process, and greatly reducing the read-write amplification of the key-value database. By extracting the workload characteristics, adaptively adjust the storage separation granularity for different workloads, so that our storage separation architecture is applicable to various environments.

[0021] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. Brief Description of the Drawings

[0022] One or more embodiments are exemplarily illustrated by the corresponding drawings. These exemplary illustrations and the drawings do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and among them:

[0023] Figure 1 is the principle block diagram of the storage system described in Embodiment 1;

[0024] Figure 2 is the schematic diagram of the merging process based on density awareness. Detailed Embodiments

[0025] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the drawings. The attached drawings are only for reference and explanation, and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of explanation, multiple details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be shown in a simplified manner.

[0026] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so as to implement the embodiments of the present disclosure described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0027] Unless otherwise specified, the term "plurality" means two or more.

[0028] In the embodiments of the present disclosure, the character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0029] The term "and / or" is an associative relationship describing an object, indicating that there can be three relationships. For example, A and / or B means: A or B, or, A and B, these three relationships.

[0030] The term "corresponding" can refer to an associative relationship or a binding relationship. A corresponding to B means that there is an associative relationship or a binding relationship between A and B.

[0031] Embodiment 1

[0032] This embodiment discloses a key-value database storage system based on data attribute perception, as Figure 1 shown, including a K-Means classifier, a memory component, and a disk component. The K-Means classifier is connected to the write request and is used to record and periodically update the hot keys that are frequently updated inside the current database. The memory component includes a cold data area and a hot data area. The cold data area includes a cold data MemTable and a cold data Immutable MemTable. The hot data area includes a hot data MemTable and a hot data Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The key-value pairs in the write request are clustered into cold key-value pairs and hot key-value pairs by the K-Means classifier, and then stored in the corresponding positions of the memory component.

[0033] The storage system described in this embodiment is similar to the traditional LSM-tree design. We retain the MemTable and the Immutable MemTable as buffers to absorb the write of tiny random key-value pairs and convert them into large, sequential disk writes. To support the distinction between cold and hot, we maintain a K-Means classifier to record and periodically update the hot keys that are frequently updated inside the current database. At the same time, we maintain a set of MemTable and Immutable MemTable for cold and hot keys respectively to store cold and hot keys separately in the database.

[0034] In this embodiment, not many modifications are made to the disk component. Since cold and hot keys are placed in different MemTables, in order to distinguish between cold and hot keys in the disk, we add a prefix to the hot keys, so as to form non-overlapping key-value ranges with cold key-value pairs. Therefore, cold and hot keys will be naturally separated into different SSTables. On the other hand, since they are separated according to the density attribute of the key-value pairs, the SSTables generated at the first layer and deeper layers have different sizes.

[0035] In this embodiment, the data heat of the key-value pair is represented by recording the update frequency of the key, and the update frequency of the key is represented by the number of write accesses to the database within a time period. The one-dimensional K-Means clustering algorithm is used to cluster the data into two clusters. These two regions respectively represent that the key is determined to be cold or hot. For this part of the hot key-value pairs, since the quantity is less than that of the cold key-value pairs, we add a special character prefix of one byte to its key to achieve separate storage from the cold key-value pairs in the database.

[0036] Regarding the data density attribute, we measure the sparsity between keys by converting the keys into binary values and calculating their differences. A larger difference indicates that the two keys are sparser, indicating that separation is required. To minimize the calculation cost, modify the merging process of the key-value database, as Figure 2 shown. After the merging operation extracts the key-value pairs in the SSTable and forms a sorted sequence of key-value pairs, the key-value pairs are taken out one by one and the binary difference between every two adjacent key-value pairs is calculated. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable. Thus, the storage separation based on data density is achieved.

[0037] For the merging process for data density, our goal is to split the SSTable to reduce write amplification. However, the density between key-value pairs in the SSTable varies depending on the workload. To ensure the adaptability of the method of splitting key-value pairs according to density among different workloads, we propose a dynamic threshold adjustment mechanism based on workload characteristics to determine the splitting threshold of the SSTable by statistically analyzing the workload characteristics. That is, the set threshold is dynamically adjusted based on the workload characteristics, and the splitting threshold of the SSTable is determined by statistically analyzing the workload characteristics.

[0038] The specific approach is as follows: Regularly calculate the average density of the SSTables in layer 0, and calculate the splitting threshold for each level according to the statistical results. The specific calculation formula is:

[0039] ,

[0040] where Threshold(i) represents the splitting threshold of the SSTable in the i th layer.α i The coefficient representing the i layer splitting threshold, AVG density (L 0 ) represents the average density of the SSTable of layer 0, Diff(i) denotes the difference splitting threshold, β i represents the i coefficient of the difference splitting threshold of the layer.

[0041] Since the LSM-tree is continuously merged to deeper levels, the SSTables at deeper levels are denser and require smaller splitting thresholds, so the coefficient decreases as the level increases.

[0042] Embodiment 2

[0043] This embodiment discloses a data management method for a key-value database based on data attribute perception, including two parts: writing based on data heat attributes and merging based on data density attributes. The writing based on data heat attributes uses the one-dimensional K-Means clustering algorithm to cluster data into 2 clusters, and these 2 regions respectively represent that the key is determined to be cold or hot. At the same time, the memory component includes a cold data area and a hot data area. The cold data area includes a cold data MemTable and a cold data ImmutableMemTable, and the hot data area includes a hot data MemTable and a hot data ImmutableMemTable. The disk component includes cold SSTables and hot SSTables, and realizes the separate storage of cold and hot data through the writing strategy based on data heat attributes; the merging based on data density attributes is after extracting key-value pairs in the SSTable during the merge operation and forming a sorted sequence of key-value pairs, taking out key-value pairs one by one and calculating the binary difference between every two adjacent key-value pairs. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable.

[0044] This embodiment represents the data heat of the key-value pair by recording the update frequency of the key, and the update frequency of the key is represented by the number of write accesses to the database within a time period.

[0045] In this embodiment, the set threshold is dynamically adjusted based on the workload characteristics, and the splitting threshold of the SSTable is determined by statistical analysis of the load characteristics. Specifically, the average density of the SSTable of layer 0 is calculated regularly, and the splitting threshold for each level is calculated according to the statistical results. The specific calculation formula is:

[0046] ,

[0047] where Threshold(i) Represents the SSTable splitting threshold for the i layer, α i Represents the i coefficient of the splitting threshold for the AVG density (L 0 ) represents the average density of the SSTable for layer 0, Diff(i) represents the difference splitting threshold, β i represents the i coefficient of the difference splitting threshold for the

[0048] The detailed implementation process of the present invention is given below, and the specific calculation process of the key-value storage separation strategy based on data attributes proposed by the present invention is further described in detail in combination with the algorithm pseudocode.

[0049] 1. Write strategy based on data heat attribute

[0050] The write strategy based on data heat attribute proposed by the present invention realizes the separated storage of hot and cold data, reduces the merging pressure of the key-value database, and improves the overall throughput of the database.

[0051] Table 1 Write strategy based on data heat attribute

[0052]

[0053] 2. Merging based on data density attribute

[0054] The merging based on data density attribute proposed by the present invention realizes the separated storage of sparse data, reduces the rewriting of invalid data in the key-value database merging operation, and reduces the write amplification of the database.

[0055] Table 2 Merging based on data density attribute

[0056]

[0057] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments only represent possible variations. Unless explicitly required, the individual components and functions are optional, and the order of operations can vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terms used in this application are only for describing the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations including one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" etc. mean the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, or apparatus comprising the element. In this document, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method parts disclosed in the embodiments, the relevant parts may refer to the description of the method parts.

Claims

1. A key-value database storage system based on data attribute perception, characterized by: It includes a K-Means classifier, a memory component, and a disk component. The K-Means classifier is connected to the write request and is used to record and regularly update the hot keys that are frequently updated in the current database. The memory component includes a cold data area and a hot data area. The cold data area includes a cold data MemTable and a cold data Immutable MemTable. The hot data area includes a hot data MemTable and a hot data Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The key-value pairs in the write request are clustered into cold key-value pairs and hot key-value pairs through the K-Means classifier, and then stored in the corresponding position of the memory component. Modify the merge process of the key-value database. After the merge operation extracts the key-value pairs in the SSTable and forms a sorted key-value pair sequence, take out the key-value pairs one by one and calculate the binary difference between every two adjacent key-value pairs. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable.

2. The key-value database storage system based on data attribute perception according to claim 1 is characterized in that: The data popularity of key-value pairs is indicated by recording the update frequency of the key. The update frequency of the key is represented by the number of write accesses to the database within a period of time.

3. The key-value database storage system based on data attribute perception according to claim 1 is characterized in that: In the disk component, a prefix is ​​added to the hot key-value pairs so that the hot key-value pairs and the cold key-value pairs form a non-overlapping key value range, and the cold key-value pairs and the hot key-value pairs are divided into cold SSTable and hot SSTable.

4. The key-value database storage system based on data attribute perception according to claim 1 is characterized in that: The set threshold is dynamically adjusted based on the workload characteristics, and the SSTable split threshold is determined by statistically analyzing the load characteristics.

5. The key-value database storage system based on data attribute perception according to claim 4 is characterized in that: The average density of the 0th layer SSTable is calculated regularly, and the segmentation threshold of each level is calculated based on the statistical results. The specific calculation formula is as follows: ； in Threshold(i) Representative i The SSTable split threshold of the layer, α i Representative i The coefficient of the layer splitting threshold, AVG density (L 0 ) Represents the average density of the 0th layer SSTable, Diff(i) represents the difference segmentation threshold, β i Indicates i The coefficient of the layer difference segmentation threshold.

6. A key-value database data management method based on data attribute perception, characterized in that: This method includes two parts: writing based on data heat attributes and merging based on data density attributes. The writing based on data heat attributes uses a one-dimensional K-Means clustering algorithm to cluster the data into two clusters, indicating that the key is judged to be cold or hot. At the same time, the memory component includes a cold data area and a hot data area. The cold data area includes a cold data MemTable and a cold data Immutable MemTable, and the hot data area includes a hot data MemTable and a hot data Immutable MemTable. The disk component includes a cold SSTable and a hot SSTable. The separate storage of cold and hot data is achieved through a writing strategy based on data heat attributes. The merging based on data density attributes is to extract the key-value pairs in the SSTable in the merge operation and form a sorted key-value pair sequence, then take out the key-value pairs one by one and calculate the binary difference between each two adjacent key-value pairs. If the difference is greater than the set threshold, stop filling the current SSTable and create a new SSTable.

7. The key-value database data management method based on data attribute perception according to claim 6 is characterized in that: The data popularity of key-value pairs is indicated by recording the update frequency of the key. The update frequency of the key is represented by the number of write accesses to the database within a period of time.

8. The key-value database data management method based on data attribute perception according to claim 6 is characterized in that: The set threshold is dynamically adjusted based on the workload characteristics, and the SSTable split threshold is determined by statistically analyzing the load characteristics.

9. The key-value database data management method based on data attribute perception according to claim 6 is characterized in that: The average density of the 0th layer SSTable is calculated regularly, and the segmentation threshold of each level is calculated based on the statistical results. The specific calculation formula is as follows: ， in Threshold(i) Representative i The SSTable split threshold of the layer, α i Representative i The coefficient of the layer splitting threshold, AVG density (L 0 ) Represents the average density of the 0th layer SSTable, Diff(i) represents the difference segmentation threshold, β i Indicates i The coefficient of the layer difference segmentation threshold.

Images