Distributed key-value storage system

Abstract

The embodiment of the present application discloses a distributed key-value storage system. The distributed key-value storage system is composed of multiple cluster nodes. The distributed key-value storage system stores key-value data, and divides all key-value data into multiple data slices, each of which is stored in a different cluster node. The cluster node includes a memory, a fast disk, and a slow disk. The memory stores a first index and a tracking table; the first index is used to point to the key-value data stored in the fast disk; the tracking table is used to record the clock value corresponding to the key. The storage performance of the fast disk is higher than that of the slow disk. The fast disk stores a data slice called a first data set, a second index and a first write-ahead log, the second index is used to point to the key-value data stored in the slow disk, and the first write-ahead log is used to record newer log entries. The slow disk stores a data slice called a second data set, and a second write-ahead log for recording older log entries. Therefore, the present application can take into account the scalability, reliability, and fault tolerance of the distributed system, and utilize the characteristics of different storage media to optimize the trade-off between storage cost and performance.

Description

Technical Field

[0001] The present application belongs to the field of database technology, and in particular, relates to a distributed key-value storage system. Background Art

[0002] Distributed key-value storage systems are systems designed to store key-value data on multiple servers or nodes, which are usually connected by a network. Compared with single-machine key-value storage systems, they can provide more powerful performance, better scalability, higher reliability and fault tolerance. In terms of scalability, distributed systems can easily scale horizontally by adding more nodes to increase storage capacity and processing power. Compared with single-machine systems, they are not limited by the hardware resources of a single server and can support larger data sets and higher numbers of concurrent users. In terms of performance, distributed systems can improve performance by storing data in sharding on multiple nodes, allowing parallel processing of read and write requests. They can also use geographically distributed data centers to reduce the latency of data access and provide users with faster services. In terms of reliability, through the multi-copy mechanism, distributed systems can save copies of data on multiple nodes, thereby improving data durability and availability. Even if some nodes fail, the system can still continue to operate without losing data. In terms of fault tolerance, distributed systems can maintain availability when problems occur through automatic failover and recovery mechanisms.

[0003] Although there are many distributed key-value storage systems on the market, the design of these systems is still not compatible with the technological development of modern storage media. Currently, the commonly used storage media in computer systems include dynamic random access memory (DRAM), persistent memory (PM), etc. Different memories have significant differences in read and write latency and input / output operations per second (IOPS). Dynamic random access memory, that is, conventional memory, has extremely low read and write latency, usually at the nanosecond level, while IOPS is very high, reaching millions or even billions of IOPS. However, the existing distributed key-value storage system only uses conventional memory and single-medium disk storage, which makes it difficult to optimize system performance under fixed costs. In general, there is still a lack of distributed key-value storage systems on the market that take into account the scalability, reliability and fault tolerance of distributed systems, and can make full use of the characteristics of different storage media to make a relatively perfect trade-off between storage cost and performance.

[0004] How to utilize the characteristics of different storage media to optimize the system performance of a key-value storage system is a technical problem that needs to be solved urgently by those skilled in the art.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] Based on this, it is necessary to propose a distributed key-value storage system to address the above problems, which can take into account the scalability, reliability and fault tolerance of distributed systems, and utilize the characteristics of different storage media to optimize the trade-off between storage cost and performance.

[0007] The present application solves the technical problem by adopting the following technical solutions:

[0008] The present application provides a distributed key-value storage system, which is composed of multiple cluster nodes. The distributed key-value storage system stores key-value data, divides all key-value data into multiple data shards, and each data shard has a preset number of copies. The copies are divided into a unique master copy and multiple slave copies according to a preset consensus protocol. The master copy and the slave copy corresponding to a data shard are called a copy group; different copies of each data shard are stored in different cluster nodes; the cluster nodes include memory, fast disk and slow disk, and the cluster nodes are used to store copies of multiple different data shards; the memory stores a first index and a tracking table; the first index is used to guide the key-value data stored in the fast disk, and adopts a B-tree index structure; the tracking table is used to record the clock values ​​corresponding to a preset number of keys, and the clock value is used to indicate the frequency of access to the corresponding key; the storage performance of the fast disk is higher than that of the slow disk, and the data shards stored in the fast disk are called first data sets, and the first data set is associated with the first index; the fast disk has a higher storage performance than the slow disk, and the data shards stored in the fast disk are called first data sets, and the first data set is associated with the first index; The disk also stores a second index and a first write-ahead log, the second index is used to point to the key-value data stored in the slow disk, the second index adopts an SST file index structure, the SST file index in the second index contains the offset position of each key-value pair in the corresponding SST file, the start and end ranges of the key in the corresponding SST file, and a Bloom filter for checking whether a key exists in the corresponding SST file; the first write-ahead log is used to record newer log entries of a preset space size; the fast disk uses a file with a slab data layout to save the first data set, and each slab file in the fast disk stores key-value data of a preset size range; the key-value data stored in the slow disk is called a second data set, and the second data set is associated with the second index; the slow disk also stores a second write-ahead log, and the second write-ahead log is used to record older log entries that cannot be accommodated by the first write-ahead log; the slow disk uses an SST file structure of an LSM tree to save the second data set, and the SST file stores key-value data within a non-intersecting key range.

[0009] In an optional embodiment of the present application, the distributed key-value storage system is configured to include: when a read request is obtained, parsing the read request to determine the target key, locating the first optimal cluster node according to the target key, the first optimal cluster node being the cluster node that meets the first optimal condition in the distributed key-value storage system; retrieving whether the first index of the first optimal cluster node stores the storage information of the target key in the fast disk; if the storage information of the target key is retrieved according to the first index, reading the target data from the fast disk, initiating a tracking table update operation and returning the value of the key; if the storage information of the target key is not retrieved in the first index, retrieving whether the storage information of the target key in the slow disk is stored in the second index of the first optimal cluster node; if the storage information of the target key is retrieved according to the second index, reading the target data from the slow disk, initiating a tracking table update operation and returning the value of the key; if the storage information of the target key is not retrieved according to the second index, directly initiating a tracking table update operation and returning information that the key is not found.

[0010] In an optional embodiment of the present application, the distributed key-value storage system is configured to include: when a write request is obtained, parsing the write request to determine the target key and the corresponding value, locating the corresponding data shard according to the target key, and determining the corresponding master copy according to the data shard; obtaining the first write-ahead log of the master copy, and adding log entries of the target key and value in the first write-ahead log; when the storage space occupied by the first write-ahead log exceeds a preset space size, dividing the first write-ahead log into an old log and a new log according to predetermined conditions; maintaining the new log in a fast disk for storage, and converting the old log into a second write-ahead log for storage in a slow disk; synchronously sending the log entry to all slave copies corresponding to the master copy, so that the slave copies synchronously update the corresponding first write-ahead log; and performing a commit operation on the log entry of the target key and value.

[0011] In an optional embodiment of the present application, the submission operation includes: retrieving whether the storage information of the target key in the fast disk is stored in the first index corresponding to the master copy; if the storage information of the target key is retrieved according to the first index, determining whether the data size of the value is suitable for the slab file where the target key is located; if suitable, updating the slab file in situ according to the value; if not suitable, deleting the target key and the corresponding original value from the slab file, locating the target key and value in a suitable slab file according to the target key and value, and associating the positioning information into the first index; if the storage information of the target key is not retrieved in the first index, locating the target key and value in a suitable slab file according to the target key and value, and associating the positioning information into the first index; after writing the target key and value, initiating a tracking table update operation.

[0012] In an optional embodiment of the present application, the tracking table update operation includes: if the target key is retrieved in the tracking table, the clock value corresponding to the target key is set to the maximum value; if the target key is not retrieved in the tracking table, the key and its corresponding clock value are added to the tracking table, and the clock value is set to the maximum value; if the tracking table is full, the key with a clock value of 0 is removed from the tracking table; if there is no key with a clock value of 0 in the tracking table, the clock values ​​of all keys in the tracking table are gradually reduced in steps of 1 until a key with a clock value of 0 appears in the tracking table after a certain step, and then the key with a clock value of 0 is removed from the tracking table.

[0013] In an optional embodiment of the present application, the distributed key-value storage system shown is configured to include: when a cluster node crashes and restarts, a first index is constructed based on the first data set in the fast disk and stored in the memory; at the same time, the data shards of the cluster node are added to the corresponding replica group and marked as slave replicas in the replica group; the corresponding master replica is determined based on the slave replica, and the log entries of the master replica are obtained and synchronized; the last log entry that completes the commit operation is obtained; after the construction of the first index is completed and stored in the memory, the commit operations of all log entries generated after the last log entry that completes the commit operation are executed one by one.

[0014] In an optional embodiment of the present application, the distributed key-value storage system is configured to include: monitoring the fast disk and the tracking table, when the usage rate of the fast disk exceeds the occupancy rate threshold, it is defined as satisfying the write trigger timing; when the hot key ratio of the tracking table is lower than a preset ratio threshold, it is defined as satisfying the read trigger timing, and the hot key ratio is the ratio of the number of keys belonging to the fast disk in the tracking table to the number of keys in the tracking table; performing data compression operations according to the write trigger timing; or performing multiple rounds of data compression operations according to the read trigger timing until the difference between the hot key ratio of the current round of tracking table and the hot key ratio of the previous round of tracking table is lower than a preset improvement threshold; the data compression operation includes: determining a target compression interval; and performing compression operations according to the write trigger timing or the read trigger timing respectively. Determine the hot keys and cold keys within the target compression interval; remove the cold key from the slab file where it is located and merge and sort it with the SST file containing the cold key within the key range in the slow disk to generate a new SST file to replace the original SST file; after the cold key is moved to the second data set, the clock value of the cold key is maintained unchanged; if the hot key is not in the fast disk, select a suitable slab file according to the value size of the hot key, and write the hot key value pair in the selected slab file, otherwise only delete the hot key value pair from the SST file where it is located, and set the clock value of the hot key to the maximum value; update the first index according to the updated first data set, and update the second index according to the updated second data set.

[0015] In an optional embodiment of the present application, determining the target compression interval includes: dividing the entire key value space into multiple continuous key spaces, the continuous key spaces are called compression intervals, and the compression intervals are determined by the key ranges of a preset fixed number of continuous SST files in the slow disk; randomly selecting a preset number of compression intervals as candidate compression intervals, and calculating the scores of the candidate compression intervals; selecting the candidate compression interval with the highest score as the target compression interval.

[0016] In an optional embodiment of the present application, the score of the candidate compression interval is calculated, including: if the key is in the tracking table, setting the coldness of the key to 1 / (clock value of the key + 1); if the key is not in the tracking table, setting the coldness of the key to 1; summing up the coldness of all keys in the candidate compression interval and marking the sum as the interval benefit; obtaining the interval cost of the candidate compression interval, the interval cost is the sum of the read and write costs of the compression operation to be performed in the slow disk; and determining the score according to the ratio of the interval benefit to the interval cost.

[0017] In an optional embodiment of the present application, hot keys and cold keys in the target compression interval are determined according to the write trigger timing or the read trigger timing, including: at the read trigger timing, from all the keys in the target compression interval that are stored simultaneously in the slow disk and the tracking table, the keys with the largest clock value of a first preset percentage are selected and marked as hot keys; at the read trigger timing, the keys with a clock value of 0 that are stored simultaneously in the fast disk and the tracking table in the target compression interval are marked as cold keys; at the write trigger timing, the keys with the maximum clock value that are stored simultaneously in the slow disk and the tracking table in the target compression interval are marked as hot keys; at the write trigger timing, from all the keys in the target compression interval that are stored simultaneously in the fast disk and the tracking table, the keys with the smallest clock value of a second preset percentage are selected and marked as cold keys, and the second preset percentage is used to ensure that there are more cold keys than hot keys.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] The distributed key-value storage system proposed in this application can take into account the scalability, reliability and fault tolerance of the distributed system through the architecture design of the hybrid storage medium and the above-mentioned management process, and can make full use of the characteristics of different storage media to optimize the trade-off between storage cost and performance. Among them, the sharding mechanism ensures the scalability of the system, the multi-copy mechanism ensures the reliability of the system, and the combination of the pre-write log technology and the multi-copy mechanism enables the system to have a strong fault tolerance, that is, a single point failure does not affect the normal access of the system and the fault point can be quickly synchronized to the latest data state after restarting. More importantly, the system can reduce hardware costs while maintaining similar performance levels by storing relatively less frequently accessed data on the fast disk and relatively more infrequently accessed data on the slow disk. At the same time, the system introduces a slab-based data layout for the fast disk, supports fast random writes and in-situ updates, and introduces the data structure of the SST file in the LSM tree for the slow disk, supporting efficient reading and sequential writing. In other words, the system realizes high-speed random read and write on the fast disk medium and high-speed sequential read and write on the slow disk medium, giving full play to the respective advantages of the two storage media. In addition, the system maintains the first index of the first data set in memory and the second index of the second data set in the fast disk, which improves the overall reading performance of the system. Finally, it should be pointed out that the system manages recent log entries in the fast disk, and through the synchronization mechanism of the pre-write log, the data status of multiple copies of the shards can be quickly synchronized after data is written or the cluster node is restarted, ensuring the final consistency of the data.

[0020] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with the accompanying drawings. It should be understood that the above general description and the detailed description below are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] in:

[0023] Figure 1 A schematic diagram of the system architecture of a distributed key-value storage system provided by an embodiment;

[0024] Figure 2A schematic diagram of data distribution stored in a distributed key-value storage system according to an embodiment of the present invention is provided;

[0025] Figure 3 A flowchart of a read request response provided by an embodiment;

[0026] Figure 4 A schematic diagram of a write request response process provided by an embodiment;

[0027] Figure 5 A schematic diagram of a submission operation flow provided by an embodiment;

[0028] Figure 6 A schematic diagram of a node recovery process provided by an embodiment;

[0029] Figure 7 A flowchart of a data compression operation provided by an embodiment. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] Distributed key-value storage systems are systems designed to store key-value data on multiple servers or nodes, which are usually connected by a network. Compared with stand-alone key-value storage systems, they can provide more powerful performance, better scalability, higher reliability and fault tolerance. These features make distributed key-value storage systems an ideal choice for large-scale, high-concurrency applications. There are already many distributed key-value storage systems on the market, but most of the existing distributed key-value storage systems use stand-alone key-value storage systems as core components.

[0032] In addition, the storage media commonly used in computer systems include dynamic random access memory (DRAM), persistent memory (PM), Intel Optane solid-state drives, and other brands of solid-state drives. They have significant differences in read and write latency and the number of read and write operations per second (IOPS). Dynamic random access memory, that is, conventional memory, has extremely low read and write latency, usually at the nanosecond level, while IOPS is very high, reaching millions or even billions of IOPS. Persistent memory (such as Intel Optane DC) has relatively low read and write latency, usually between tens and hundreds of microseconds, and IOPS is also very high, reaching millions of IOPS. The read and write latency of Intel Optane solid-state drives is also usually between tens and hundreds of microseconds, and IOPS ranges from hundreds of thousands to millions of IOPS depending on the model. The read and write latency of other brands of SSDs is usually between a few hundred microseconds and a few milliseconds. NVMe SSDs usually have lower latency than SATA SSDs. IOPS varies depending on the model and type. NVMe SSDs range from tens of thousands to millions of IOPS, while SATA SSDs usually range from thousands to tens of thousands of IOPS. Different storage media also have significant differences in price, but basically the price is proportional to the read and write latency and inversely proportional to IOPS.

[0033] Although there are many distributed key-value storage systems on the market, the design of these systems is still not compatible with the development of modern storage media technology. Existing distributed key-value storage systems only use conventional memory and single-medium disk storage, which makes it difficult to optimize system performance under fixed costs. To this end, this application proposes a distributed key-value storage system for optimizing the architecture of the storage system. To clearly describe the distributed key-value storage system provided in this embodiment, please refer to Figure 1 to Figure 7 .

[0034] To facilitate understanding of the architecture of the distributed key-value storage system 10 provided in this application, please refer to Figure 1 Do understand.

[0035] A distributed key-value storage system 10, the distributed key-value storage system 10 is composed of multiple cluster nodes 100, the distributed key-value storage system 10 stores key-value data, divides all key-value data into multiple data shards, each data shard has a preset number of copies, and the copies are divided into a unique master copy and multiple slave copies according to a preset consensus protocol, and the master copy and slave copy corresponding to a data shard are called a copy group; different copies of each data shard are stored in different cluster nodes 100; the cluster node 100 includes a memory 110, a fast disk 120 and a slow disk 130, and the cluster node 100 is used to store copies of multiple different data shards; the memory 110 stores a first index and a tracking table; the first index is used to point to the key-value data stored in the fast disk 120 memory 110, and adopts a B-tree index structure; the tracking table is used to record the clock values ​​corresponding to a preset number of keys, and the clock value is used to indicate the frequency of access to the corresponding key; the storage performance of the fast disk 120 is higher than that of the slow disk 130, and the data shards stored in the fast disk 120 memory 110 are called first data sets, and the first data set is The first index is associated; the fast disk 120 also stores a second index and a first write-ahead log, the second index is used to point to the key-value data stored in the memory 110 of the slow disk 130, the second index adopts an SST file index structure, and the SST file index in the second index contains the offset position of each key-value pair in the corresponding SST file, the start and end ranges of the key in the corresponding SST file, and a Bloom filter for checking whether a key exists in the corresponding SST file; the first write-ahead log is used to record newer log entries of a preset space size; the fast disk 120 uses a slab data layout file to save the first data set, and each slab file in the fast disk 120 stores key-value data of a preset size range; the key-value data stored in the slow disk 130 is called a second data set, and the second data set is associated with the second index; the slow disk 130 also stores a second write-ahead log, and the second write-ahead log is used to record older log entries that cannot be accommodated by the first write-ahead log; the slow disk 130 uses the SST file structure of the LSM tree to save the second data set, and the SST file stores key-value data within a non-intersecting key range.

[0036] In one embodiment, a distributed key-value storage system 10 (hereinafter referred to as the system) is composed of a plurality of cluster nodes 100, each of which includes a memory 110, a fast disk 120, and a slow disk 130. The distributed key-value storage system 10 stores key-value data, and divides the entire key-value data into a plurality of data shards, each of which has a preset number of replicas, and the number of replicas in the replica group can be set by the user. Different replicas of each data shard are stored in different cluster nodes 100.

[0037] Furthermore, during the operation of the system, the replicas can be divided into primary replicas and secondary replicas according to a preset consensus protocol (including but not limited to the Paxos protocol or the Raft protocol, etc.). The set of primary replicas and secondary replicas corresponding to a data shard is called a replica group. However, it is worth noting that the replicas are divided into primary replicas and secondary replicas. There is only one primary replica in a replica group, which is uniquely determined; and all other replicas in the replica group are called secondary replicas. And this uniqueness is not fixed. For example, when the cluster node 100 where the primary replica is located crashes due to a failure, the primary replica cannot be retrieved in the replica group within the system operation. In this case, the primary replica can be re-determined by the consensus protocol. In other words, in order to determine the reliability of the system operation, the identities of the primary and secondary replicas can be converted if necessary. But it is also necessary to ensure that there is only one primary replica at the same time.

[0038] The write-ahead log uses a multi-copy management mechanism in the system, which mainly has two functions: one is to maintain the data consistency of the shard copy group, and the other is to ensure that all shards in the node can be restored to the latest data state after the cluster node 100 crashes and restarts. The system stores recent log entries in the fast disk 120 to support high-speed sequential reading and writing of the write-ahead log; in addition, the old log entries that exceed the storage limit of the fast disk 120 are stored in the slow disk 130 to ensure that unsynchronized key-value data is not lost in the database synchronization update scenario.

[0039] The cluster node 100 includes a memory 110, a fast disk 120, and a slow disk 130, that is, the cluster node 100 uses the memory 110, the fast disk 120, and the slow disk 130 in combination to store copies of multiple different data shards. In addition to storing copies of data shards corresponding to key-value data, the memory 110, the fast disk 120, and the slow disk 130 also store corresponding other data. To facilitate the description of the storage method of data in the system, for this embodiment, reference can be made to Figure 2 Do understand. Specific Figure 2 Taking three cluster nodes 100 (cluster node 1 1001, cluster node 2 1002, and cluster node 3 1003) and two shards (shard 1, shard 2) as an example, a schematic diagram of the above system architecture is given. It should be noted that: Figure 2 In the figure, the objects enumerated in the rectangular box are Figure 1 The corresponding specific hardware structure; the objects listed in the rounded rectangular box are the corresponding data included in the specific hardware, and the corresponding direction keys indicate which hardware the data is stored in, what data is included in the hardware, and the relationship between the data.

[0040] According to the description above, each data shard will be processed into a set of replica groups, and the replicas in the replica groups are stored in different cluster nodes 100 respectively; and the replicas are divided into master replicas and slave replicas according to the consensus protocol. Therefore, the replica group where shard 1 is located can include at least three replicas for the three cluster nodes 100, and they correspond to: 1 master replica of shard 1, and 2 slave replicas of shard 1 according to the consensus protocol. Among them, the master replica of shard 1 is stored in cluster node 11001, and the two slave replicas of shard 1 are stored in cluster node 2 1002 and cluster node 3 1003 respectively. Similarly, for the replica group corresponding to shard 2, the master replica of shard 2 is stored in cluster node 3 1003, and the slave replicas of shard 2 are stored in cluster node 1 1001 and cluster node 2 1002 respectively.

[0041] The shard data in each cluster node 100 is stored in the memory 110 , the fast disk 120 and the slow disk 130 respectively. The storage relationship of the data is the same, which is not described separately here, and only one of the cluster nodes 100 is used for explanation.

[0042] Memory 110 can specifically be a dynamic random access memory, storing a first index and a tracking table. Among them, the first index is used to guide the key value data stored in memory 110 of the fast disk 120, and specifically adopts a B-tree index structure (Balanced Tree, multi-way balanced search tree). The tracking table is used to record the clock values ​​corresponding to a preset number of keys, and the management number can be set arbitrarily by the user. The clock value is used to indicate the frequency of access to the corresponding key. The clock value is specifically an integer between 0 and the maximum clock value set by the user (such as 3). Generally, the clock value is proportional to the frequency of access to its corresponding key. The tracking table will be updated during the key value reading and writing, restart recovery and compression process. The specific update process will be explained in detail later, so it will not be repeated here.

[0043] The storage performance of the fast disk 120 is higher than that of the slow disk 130, that is, the fast disk 120 is a disk medium with high performance but relatively expensive, such as persistent memory 110 or Intel Optane solid-state drive. The data shards stored in the memory 110 of the fast disk 120 are referred to as the first data set. The first data set is associated with the first index, that is, the first data set in the fast disk 120 can be located by querying the first index in the memory 110. Specifically, the fast disk 120 uses a file with a slab data layout to save the first data set, and each slab file in the fast disk 120 stores key value data of a preset size range (for example, less than 128 bytes, 129-256 bytes, etc.). Further, the fast disk 120 also stores a second index and a first pre-write log. Among them, the second index is used to guide the key value data stored in the memory 110 of the slow disk 130, and the second index can include the offset position of the key value pair in the SST file, the start and end range of the key in the file, and a Bloom filter for quickly checking whether a key exists in the SST file. In addition, a first write-ahead log of a user-set size is additionally allocated in the fast disk 120, and the first write-ahead log is used to record newer log entries of a preset space size. A second write-ahead log is correspondingly stored in the slow disk 130, and the two are related, which will be described in detail later when describing the slow disk 130, and will not be expanded here.

[0044] The distributed key-value storage system 10 also includes a slow disk 130, which can be a disk medium with low performance and price, such as a general solid-state hard disk. The data shards of the key-value data stored in the slow disk 130 are called the second data set, and the second data set is associated with the second index in the fast disk 120, that is, according to the query of the second index, it can be determined whether the corresponding key-value data exists in the second data set, and the specific location when it exists. Specifically, the slow disk 130 can use the SST file structure (Sorted StringTable, ordered string table) of the LSM tree (Log Structure Merging tree) to save the second data set, and the SST file stores the key-value data within the non-intersecting key range. Further, the slow disk 130 also stores a second write-ahead log, which is used to record older log entries that the first write-ahead log cannot accommodate. Specifically, the first write-ahead log and the second write-ahead log can be space to store write-ahead logs (Write-Ahead Log). And the fast disk 120 can use the garbage collection thread to regularly clean up the space occupied by expired write-ahead logs. For example, when the occupancy rate of the log space reaches the upper limit of the log occupancy rate set by the user (for example, 80%), the system migrates the older first write-ahead logs in batches to the slow disk 130 for storage, where the block size of each log entry is an integer multiple of the page size, and the write-ahead logs stored in the slow disk 130 are called second write-ahead logs. After several rounds of migration, the space occupancy rate drops to the lower limit of the log occupancy rate set by the user (for example, 50%).

[0045] Furthermore, the present application does not make any further limitations on the tiering strategy of the SST files in the slow disk 130. These SST files can be stored in a single layer or in multiple layers. If it is stored in multiple layers, different compression strategies in the LSM tree can also be selected to implement the downward merge migration of high-level SST files. The present application allows the SST files in the slow disk 130 to be organized and managed using a fixed tiering strategy (i.e., selecting a single layer or multiple layers) or a read-write sensitive dynamic tiering strategy (i.e., dynamically switching between single layers and multiple layers according to the read-write situation), and also allows the selection of different LSM tree compression strategies for multi-layer management. Other variations may make further limitations on the tiering strategy of SST files, but as long as they comply with the technical features of using two disk media with different read-write efficiencies, supporting dynamic switching between the first data set and the second data set, and introducing multi-copy management of write-ahead logs, they should not be considered to exceed the technical scope to be protected by the present application.

[0046] The sharding mechanism adopted by the system architecture ensures the scalability of the system, and the multi-copy mechanism ensures the reliability of the system. The combination of the write-ahead log technology and the multi-copy mechanism enables the system to have a certain fault tolerance, that is, a single point failure does not affect the normal access of the system. The system reduces the hardware cost while maintaining a similar performance level by storing relatively less frequently accessed data on the fast disk 120 and relatively more infrequently accessed data on the slow disk 130. At the same time, the system realizes high-speed random reading and writing on the fast disk 120 medium and high-speed sequential reading and writing on the slow disk 130 medium, giving full play to the respective advantages of the two storage media.

[0047] In one embodiment, the distributed key-value storage system 10 is configured to include: when a read request is obtained, the read request is parsed to determine the target key, and the first optimal cluster node 100 is located according to the target key, where the first optimal cluster node 100 is the cluster node 100 that meets the first optimal condition in the distributed key-value storage system 10; whether the storage information of the target key in the fast disk 120 is stored in the first index of the first optimal cluster node 100; if the storage information of the target key is retrieved according to the first index, the target data is read from the fast disk 120, and the value of the key is returned after the tracking table update operation is initiated; if the storage information of the target key is not retrieved in the first index, whether the storage information of the target key in the slow disk 130 is stored in the second index of the first optimal cluster node 100; if the storage information of the target key is retrieved according to the second index, the target data is read from the slow disk 130, and the value of the key is returned after the tracking table update operation is initiated; if the storage information of the target key is not retrieved according to the second index, the tracking table update operation is directly initiated and the information that the key is not found is returned.

[0048] In one embodiment, when the system receives a read request, that is, when the key value data stored in the system memory 110 needs to be obtained, the system provided by the present application will execute the read request response process. For a clear description of the read request response process, please refer to Figure 3 , including steps S310~S360.

[0049] Step S310: Parse the read request to determine the target key, and locate the first optimal cluster node according to the target key.

[0050] In one embodiment, when the system obtains a read request initiated by a user, the key that the user wants to read, that is, the target key, can be determined by parsing the read request. The system can locate the first optimal cluster node 100 based on the target key, and the first optimal cluster node 100 is a cluster node 100 that meets the first optimal condition in the distributed key-value storage system 10. The first optimal condition can specifically be that the cluster node 100 where the data copy of the target key is located meets the closest location, or the current load is the lightest, etc.

[0051] Step S320: searching whether the first index of the first optimal cluster node stores the storage information of the target key in the fast disk.

[0052] In one embodiment, the memory 110 of the cluster node 100 stores a first index, and the first index indicates the key value data stored in the memory 110 of the fast disk 120. Therefore, by searching the first index, it can be determined whether the key value data corresponding to the target key exists in the fast disk 120.

[0053] If the storage information of the target key is retrieved according to the first index, step S330 is executed: reading the target data from the fast disk.

[0054] In one embodiment, if the storage information of the target key is retrieved in the first index, it is determined that the key value data corresponding to the target key is stored in the fast disk 120. Therefore, the target data, that is, the value of the key, can be directly read from the fast disk 120.

[0055] If the storage information of the target key is not retrieved in the first index, step S340 is executed: searching whether the second index of the first optimal cluster node stores the storage information of the target key in the slow disk.

[0056] In one embodiment, if the key-value data cannot be found in the first index, the second index in the fast disk 120 may be used to search, and the second index indicates the key-value data stored in the memory 110 of the slow disk 130 .

[0057] If the storage information of the target key is retrieved according to the second index, step S350 is executed: reading the target data from the slow disk.

[0058] If the storage information of the target key is not retrieved in the second index, and after steps S330 and S340, step S360 is directly executed: the tracking table update operation is initiated and the reading result is returned.

[0059] In one embodiment, if the storage information of the target key is retrieved in the second index, that is, it is determined that the corresponding key-value data is stored in the slow disk 130, the target data is determined through the second index, and then the value of the key is read from the corresponding SST file. On the contrary, if there is no storage information of the target key in the second index, it means that the key-value data corresponding to the corresponding read request is not stored in the system, so the information that the key is not found can be returned.

[0060] It is also worth noting that, regardless of whether the key value data required to be retrieved by the read request exists in the system, in the fast disk 120 or in the slow disk 130, before each process returns the result, a tracking table update operation will be performed to update the clock value of the key in the tracking table. The tracking table update operation will appear again in the system execution process, which will not be described here for the time being, and will be fully explained later.

[0061] From the reading process, the system proposed in the present application reads data randomly at high speed on the fast disk 120 medium and reads data sequentially at high speed on the slow disk 130 medium, giving full play to the respective advantages of the two storage media in data reading. In addition, the system maintains the fast disk 120 data index in the memory 110 and the slow disk 130 data index in the fast disk 120, thereby improving the overall reading performance of the system.

[0062] In one embodiment, the distributed key-value storage system 10 is configured to include: when a write request is obtained, the write request is parsed to determine the target key and the corresponding value, the corresponding data shard is located according to the target key, and the corresponding master copy is determined according to the data shard; the first write-ahead log of the master copy is obtained, and the log entry of the target key and value is added to the first write-ahead log; when the storage space occupied by the first write-ahead log exceeds a preset space size, the first write-ahead log is divided into an old log and a new log according to predetermined conditions; the new log is maintained in the fast disk 120 and the memory 110 is stored, and the old log is converted into a second write-ahead log and stored in the slow disk 130; the log entry is synchronously sent to all slave copies corresponding to the master copy, so that the slave copies synchronously update the corresponding first write-ahead log; and a commit operation is performed on the log entry of the target key and value.

[0063] In one embodiment, when the system receives a write request, that is, when new key-value data needs to be ingested into the system, the system provided by the present application will execute the write request response process. Figure 4 , including steps S410~S450.

[0064] Step S410: parse the write request to determine the target key and the corresponding value, locate the corresponding data shard according to the target key, and determine the corresponding primary copy according to the data shard.

[0065] In one embodiment, when the system obtains a write request, it indicates that the user intends to write a key-value pair (key, value) in the system. To this end, the write request can be parsed to determine the target key and the corresponding value. The system first locates the data shard according to the target key, and then corresponds to the master copy according to the replica group of the data shard, that is, the cluster node 100 where the master copy is located.

[0066] Step S420: Obtain the first write-ahead log of the primary copy, and add a log entry of the target key and value in the first write-ahead log.

[0067] In one embodiment, after determining the cluster node 100 where the primary copy is located, the first write-ahead log stored in the flash disk 120 memory 110 of the cluster node 100 can be obtained, and a log entry of the target key and value can be added to the first write-ahead log.

[0068] Step S430: When the storage space occupied by the first write-ahead log exceeds a preset space size, the first write-ahead log is divided into an old log and a new log according to a predetermined condition; the new log is maintained in the fast disk for storage, and the old log is converted into a second write-ahead log and stored in the slow disk.

[0069] In one embodiment, after the log entries of the target key and value are added to the first write-ahead log, a judgment can be made on the first write-ahead log: whether the storage space occupied by the first write-ahead log exceeds the preset space size, that is, whether the log space occupancy rate reaches the upper limit of the log occupancy rate set by the user (for example, 80%). If it exceeds, the first write-ahead log needs to be transferred. The first write-ahead log of the system is divided into old logs and new logs according to predetermined conditions. The new log is maintained in the fast disk 120 memory 110 for storage; the older log entries are converted into the second write-ahead log and migrated to the slow disk 130 in batches for storage, where the block size of each log entry is an integer multiple of the page size. After several rounds of transfer, the space occupancy rate drops to the lower limit of the log occupancy rate set by the user (for example, 50%). This ensures the rational use of the fast disk 120 space.

[0070] Step S440: synchronously sending the log entry to all slave replicas corresponding to the master replica, so that the slave replicas synchronously update the corresponding first write-ahead log.

[0071] In one embodiment, the log entry of the target key and value added to the first write-ahead log of the master replica is also synchronously sent to the slave replica of the same data shard, so that the slave replica of the cluster node 100 where the slave replica is located will also synchronously update the first write-ahead log to add the log entry of the target key and value.

[0072] Step S450: Perform a commit operation on the log entry of the target key and value.

[0073] In one embodiment, after completing the log update described above, the corresponding data needs to be written to the disk, and the execution process of completing the writing to the disk is the commit operation, which will be described in detail later. It is also worth noting that the commit operation is synchronously executed in the cluster node 100 where the master copy and the slave copy are located, that is, all copies of the copy group where the key-value data corresponding to the target key and value are located will execute the commit operation to write the corresponding key-value data.

[0074] From the perspective of the writing process, the system proposed in this application introduces a slab-based data layout in the fast disk 120, supporting fast random writing and in-place updates. In addition, the system manages recent log entries in the fast disk 120. As long as the master copy holds the most complete pre-write log, the synchronization mechanism of the pre-write log can be used to ensure the final consistency of multiple replica data.

[0075] In one embodiment, the submission operation includes: retrieving whether the storage information of the target key in the fast disk 120 is stored in the first index corresponding to the master copy; if the storage information of the target key is retrieved according to the first index, determining whether the data size of the value is suitable for the slab file where the target key is located; if suitable, updating the slab file in situ according to the value; if not suitable, deleting the target key and the corresponding original value from the slab file, locating the target key and value in a suitable slab file according to the target key and value, and associating the positioning information into the first index; if the storage information of the target key is not retrieved in the first index, locating the target key and value in a suitable slab file according to the target key and value, and associating the positioning information into the first index; after writing the target key and value, initiating a tracking table update operation.

[0076] In one implementation, for ease of explanation of the submission operation, please refer to Figure 5 , Figure 5 A schematic diagram of the submission operation flow is provided, including steps S510 to S560.

[0077] Step S510: searching whether the first index corresponding to the primary copy stores the storage information of the target key in the fast disk 120.

[0078] If the storage information of the target key is retrieved according to the first index, step S520 is executed: determining whether the data size of the value is suitable for the slab file where the target key is located.

[0079] In one embodiment, it is worth noting that the submission operation process provided in this embodiment can be applied to any replica, whether it is a master replica or a slave replica. For ease of understanding, the master replica will be used as the execution subject for explanation, and in the actual execution process, in a preferred embodiment, the value is first written to the master replica, and after the master replica completes the submission operation of the write, the slave replica will execute accordingly. Therefore, when submitting the write value, the master replica can search for the key from the first index of the memory 110 to determine whether the key value data corresponding to the target key is stored in the fast disk 120. By giving priority to writing in the fast disk 120, the writing efficiency can be improved.

[0080] If the storage information of the target key in the fast disk 120 is retrieved in the first index, it means that the corresponding key value data is stored in the fast disk 120, so the value can be directly updated in the fast disk 120. However, since the data in the fast disk 120 are all stored in the first data set according to the slab data layout file, when the storage information of the target key is retrieved, it is also necessary to determine whether the file size of the slab file where the target key is located is suitable for the value to be updated, so as to ensure that the data is updated and written completely, correctly and efficiently.

[0081] If appropriate, execute step S530: update the slab file in situ according to value.

[0082] If not suitable, execute step S540: delete the target key and the corresponding original value from the slab file, locate the target key and value in a suitable slab file according to the target key and value, and associate the positioning information into the first index.

[0083] In one embodiment, if the data size of the value is suitable for the slab file where the target key is located, the value is updated in situ in the original slab file. On the contrary, if it is not suitable, the original value corresponding to the target key is deleted from the original slab file. The target key and value are written to a new slab file, and the data size of the new slab file is suitable for the data size of the target key and value. After completing the writing of the value, the first data set where the data is located has actually changed, and the corresponding first index associated with the first data set also needs to be updated. For this purpose, the first data set after the value is updated can be used to update the disk position pointed to by the target key in the first index, so as to facilitate the subsequent access to the updated key-value pair.

[0084] If the storage information of the target key is not retrieved in the first index, step S550 is executed: the target key and value are located in a suitable slab file according to the target key and value, and the location information is associated with the first index.

[0085] In one embodiment, if the storage information of the target key cannot be retrieved in the first index, it means that the corresponding key-value data is not stored in the fast disk 120. To this end, in order to improve the writing efficiency, the key-value pair of the target key and value can be newly written into the fast disk 120 to improve the writing efficiency. More specifically, the process can be to locate the target key and value in a suitable slab file according to the target key and value to store the target key and value, wherein the suitable condition can be that the size of the slab file is suitable for storing the key-value pair. After completing the writing of the target key and value key-value pair, because the first data set in which it is located has also changed, the first copy associated with the changed first data set can be updated to ensure subsequent correct access.

[0086] After steps S530 , S540 and S550 , step S560 is executed: a tracking table update operation is initiated.

[0087] In one embodiment, after steps S530, S540 and S550, the key value is actually written. Of course, the access frequency of the corresponding key has changed. Therefore, the clock value recording the access frequency of the key and the tracking table storing the clock value also need to be updated. It can be understood that the tracking table update operation also appears in the previous text, so it is not explained here as a separate operation, and will be explained in detail later.

[0088] In one embodiment, the tracking table update operation includes: if the target key is retrieved in the tracking table, the clock value corresponding to the target key is set to the maximum value; if the target key is not retrieved in the tracking table, the key and its corresponding clock value are added to the tracking table, and the clock value is set to the maximum value; if the tracking table is full, the key with a clock value of 0 is removed from the tracking table; if there is no key with a clock value of 0 in the tracking table, the clock values ​​of all keys in the tracking table are gradually reduced in steps of 1 until a key with a clock value of 0 appears in the tracking table after a certain step, and then the key with a clock value of 0 is removed from the tracking table.

[0089] In one embodiment, according to the foregoing description, the tracking table is used to record the clock value corresponding to the key, and the clock value is used to indicate the access frequency of the corresponding key. Generally, the numerical value of the clock value is proportional to the access frequency. The larger the numerical value, the more frequently the corresponding key is accessed; conversely, the less frequently it is accessed. In addition, the fast disk 120 and the slow disk 130 are mainly used to store key value data in the present application. There are differences in performance between the fast disk 120 and the slow disk 130. In order to utilize the performance of storage media with different performances, in a preferred embodiment, the key value data that should be accessed more frequently should be stored in the fast disk 120 with better storage performance, while the key value data that is accessed less frequently should be stored in the slow disk 130. Therefore, determining the key value data is a more critical step. The tracking table will be updated at the end of the previous reading and writing process. In this embodiment, the tracking table update operation will be explained separately.

[0090] The tracking table update operation may specifically include: if the target key is retrieved in the tracking table, the clock value corresponding to the target key is set to the maximum value.

[0091] Regardless of whether it is in the reading process or the writing process, the first index or the second index will be searched to determine whether the target key has corresponding storage information in the index. The above operation is actually an access. Therefore, in one embodiment of the present application, each time the target key is accessed, that is, the first index or the second index retrieves the storage information of the target key and performs the corresponding operation at the same time or after, the clock value corresponding to the target key in the tracking table can be set to a predetermined maximum value.

[0092] If the target key is not retrieved in the tracking table, add the key and its corresponding clock value to the tracking table and set the clock value to the maximum value.

[0093] In one embodiment, the tracking table only stores some of the keys, and the clock value is the mapping value of the key in the tracking table, so only some of the keys have clock values. If the target key is not retrieved by searching the tracking table during the access process, that is, the target key is not stored in the tracking table, the key and its corresponding clock value can be added to the tracking table, and the clock value is set to the maximum value.

[0094] If there is no key with a clock value of 0 in the tracking table, the clock values ​​of all keys in the tracking table will be gradually reduced in steps of 1 until a key with a clock value of 0 appears in the tracking table after a certain step, and then the key with a clock value of 0 will be removed from the tracking table.

[0095] If the trace table is full, the key with a clock value of 0 is removed from the trace table.

[0096] In one embodiment, when the tracking table is full, the system removes the key with a clock value of 0 from the tracking table to make room for other recently accessed keys. If there is no key with a clock value of 0 in the tracking table, the clock value of each key in the tracking table is gradually reduced by 1 until a key with a clock value of 0 appears in the tracking table, and then the key with a clock value reduced to 0 is removed from the tracking table.

[0097] Based on the above tracking table update operation, it can be ensured that the hotter, i.e., frequently accessed keys can be stored in the tracking table, thereby controlling the corresponding key-value data to be stored in the fast disk 120; conversely, the infrequently accessed keys are not stored in the tracking table. By storing keys in the tracking table, it is possible to ensure the dynamic switching between the first data set and the second data set: ensuring that enough frequently accessed key-value data are migrated to the fast disk 120, thereby improving the read hit rate of the fast disk 120; ensuring that enough infrequently accessed key-value data are migrated to the slow disk 130, thereby freeing up some space on the fast disk 120.

[0098] In one embodiment, the distributed key-value storage system 10 shown is configured to include: when the cluster node 100 crashes and restarts, a first index is constructed based on the first data set in the fast disk 120 and stored in the memory 110; at the same time, the data shards of the cluster node 100 are added to the corresponding replica group and marked as slave replicas in the replica group; the corresponding master replica is determined based on the slave replica, and the log entries of the master replica are obtained and synchronized; the last log entry that completes the commit operation is obtained; after the construction of the first index is completed and stored in the memory 110, the commit operations of all log entries generated after the last log entry that completes the commit operation are executed one by one.

[0099] In one embodiment, during the operation of the system, it is inevitable that a node needs to be restarted, including but not limited to failure, maintenance, equipment update, etc. As for how to quickly restore service after a node crashes and restarts, this application provides a node recovery process. For the convenience of explaining the process, please refer to Figure 6 , including steps S610~S650.

[0100] Step S610: The cluster node crashes and restarts.

[0101] In one embodiment, after the cluster node 100 is restarted, it can execute the necessary processes for startup, including but not limited to startup self-check, establishing connection with other nodes, etc. The specific executable operations can be set by the user, and the corresponding function startup is a relatively existing technology, which will not be repeated here.

[0102] Step S620: construct a first index according to the first data set in the fast disk and store it in the memory.

[0103] In one embodiment, the cluster node 100 includes a memory 110. The data stored in the memory 110 will not be saved after power failure, so the original corresponding data in the memory 110 has been erased. The first index and tracking table required to be stored in the memory 110 need to be re-established after restart. The establishment process can be to build a first index based on the first data set in the fast disk 120 and store it in the memory 110. Ensure that the key value data stored in the fast disk 120 and the memory 110 can be located by searching the first index after restart.

[0104] Step S630: Add all data shards of the cluster nodes to the corresponding replica group and mark them as slave replicas in the replica group.

[0105] Step S640: Determine the corresponding master copy according to the slave copy, obtain and synchronize the log entry of the master copy; obtain the last log entry that completes the commit operation.

[0106] In one embodiment, according to the description above, the key-value data in the cluster node 100 is divided into multiple data shards, each data shard includes multiple replicas, and all replicas corresponding to a data shard are called a replica group. A replica group is further divided into a master replica and a slave replica according to a preset consensus protocol. The system ensures that there is only one master replica in the replica group at the same time. Therefore, even if there is a master replica of a replica group in the cluster node 100, after the node goes offline, the system will re-mark a master replica because the replica group where it is located lacks a master replica. That is, the replica that was originally the master replica will become a slave replica in the replica group after restarting; let alone the other slave replicas. But at the same time, the replica cannot exist independently from the replica group. Based on this, all the replicas in the cluster node 100 can be located in the replica group in the system. Then each data shard stored in the node is added to the replica group to which it belongs as a slave replica, and a connection with the other replica groups in the system is established.

[0107] Because the replica group includes a uniquely determined master replica, the log entry corresponding to the master replica is the basis for updating the corresponding log from the replica. To this end, the log entry of the master replica corresponding to the slave replica can be obtained, and the pre-write log in the cluster node 100 is updated according to the log entry of the master replica. That is, the pre-write log corresponding to the shard is synchronized with the pre-write log of the master replica in the replica group where the shard is located. At the same time, find the last log entry that completes the commit operation. The commit operation is also the process described in steps S510 to S560 above, which will not be repeated here.

[0108] After steps S620 and S640, step S650 is performed: the commit operation of all log entries generated after the last log entry that completes the commit operation is performed one by one.

[0109] In one embodiment, after the step of building the first index corresponding to step S620 is completed and the pre-write log of the primary copy is updated, and the last log entry that completes the commit operation is found, all log entries generated after the last completed log entry are gradually replayed, that is, the commit operation corresponding to the log entry is re-executed, ensuring the synchronization of the logs of the cluster node 100 with the remaining cluster nodes 100 after restart.

[0110] The system proposed in this application mainly reads and writes recent log entries at high speed in the fast disk 120. The combination of this write-ahead log technology and the multi-copy mechanism enables the system to have a strong fault tolerance capability, that is, the node can be quickly synchronized to the latest data state after it crashes and restarts. Through the synchronization mechanism of the write-ahead log, the system can also quickly restore the corresponding data after the node crashes and restarts, ensuring the final consistency of the data between the cluster nodes 100 in the system.

[0111] In one embodiment, the distributed key-value storage system 10 is configured to include: monitoring the fast disk 120 and the tracking table, when the usage rate of the fast disk 120 exceeds the occupancy rate threshold, it is defined as meeting the write trigger timing; when the hot key ratio of the tracking table is lower than the preset ratio threshold, it is defined as meeting the read trigger timing, and the hot key ratio is the ratio of the number of keys belonging to the fast disk 120 in the tracking table to the number of keys in the tracking table; performing data compression operations according to the write trigger timing; or performing multiple rounds of data compression operations according to the read trigger timing until the difference between the hot key ratio of the current round tracking table and the hot key ratio of the previous round tracking table is lower than the preset improvement threshold; the data compression operation includes: determining the target compression interval; and performing the compression operation according to the write trigger timing or the read trigger timing. Determine the hot key and the cold key in the target compression interval; remove the cold key from the slab file where it is located and merge and sort it with the SST file containing the cold key in the key range in the slow disk 130 to generate a new SST file to replace the original SST file; after the cold key is moved to the second data set, the clock value of the cold key is maintained unchanged; if the hot key is not in the fast disk 120, select a suitable slab file according to the value size of the hot key, and write the hot key value pair in the selected slab file, otherwise only delete the hot key value pair from the SST file where it is located, and set the clock value of the hot key to the maximum value; update the first index according to the updated first data set, and update the second index according to the updated second data set.

[0112] In one embodiment, the system proposed in the present application uses compression operations to exchange part of the key-value data between the fast disk 120 and the slow disk 130 to ensure that frequently accessed data is stored in the fast disk 120 as much as possible, and infrequently accessed data is stored in the slow disk 130 as much as possible, thereby improving read and write performance. As for when the compression operation needs to be performed, the fast disk 120 and the tracking table can be monitored. Two triggering moments are defined, a write triggering moment and a read triggering moment: when the usage rate of the fast disk 120 exceeds the occupancy rate threshold, it is defined as satisfying the write triggering moment; when the hot key ratio of the tracking table is lower than the preset ratio threshold, it is defined as satisfying the read triggering moment, where the hot key ratio is the ratio of the number of keys in the tracking table belonging to the fast disk 120 to the number of keys in the tracking table. For satisfying the write triggering moment, it means that the fast disk 120 is about to be full, and part of the data needs to be exchanged out of the fast disk 120 to free up part of the fast disk 120 space to accommodate more frequently accessed data. As for the read-read trigger timing, it means that the read hit rate of the fast disk 120 is not high, and relatively more accessed data needs to be migrated to the fast disk 120 to improve the read hit rate of the fast disk 120. Both require dynamic data exchange between the first data set in the fast disk 120 and the second data set in the slow disk 130, that is, to perform data compression operations.

[0113] For a clear description of data compression operations, please refer to Figure 7 , including steps S710~S760.

[0114] Step S710: Determine the target compression interval.

[0115] In one embodiment, determining the target compression interval includes: dividing the entire key value space into multiple continuous key spaces, the continuous key spaces are called compression intervals, and the compression intervals are determined by the key ranges of a preset fixed number of continuous SST files in the slow disk 130; randomly selecting a preset number of compression intervals as candidate compression intervals, and calculating the scores of the candidate compression intervals; selecting the candidate compression interval with the highest score as the target compression interval.

[0116] In one embodiment, the entire continuous key space is divided into multiple smaller continuous key spaces, which are called compression intervals. Each compression interval is determined by the key range of a fixed number of continuous SST files in the slow disk 130, and the fixed number is the window size set by the user. For example, assuming that SST file 1 stores the key range [1,1000], SST file 2 stores the key range [1001,2000], and SST file 3 stores the key range [2001,3000], and the window size is set to 2, then compression interval 1 is determined by SST file 1 and SST file 2, corresponding to the key range [1,2000], and compression interval 2 is determined by SST file 2 and SST file 3, corresponding to the key range [1001,3000]. If there is no SST file in the slow disk 130 and the SST file cannot be used to determine the compression interval, the key space in the fast disk 120 is divided into multiple smaller continuous key spaces according to the fixed size set by the user.

[0117] A number of candidate compression intervals are randomly selected from all compression intervals to calculate the corresponding scores, and the compression interval with the highest score is selected as the target compression interval. The number of candidate intervals can be set arbitrarily by the user, for example, 8. The score of the compression interval is determined according to the compression triggering timing and the distribution of all keys in the interval in the fast disk 120, the slow disk 130 and the tracking table. The candidate compression interval with the highest score is selected as the target compression interval.

[0118] In one embodiment, the score of the candidate compression interval is calculated, including: if the key is in the tracking table, setting the coldness of the key to 1 / (clock value of the key + 1); if the key is not in the tracking table, setting the coldness of the key to 1; summing up the coldness of all keys in the candidate compression interval and marking the sum as the interval benefit; obtaining the interval cost of the candidate compression interval, the interval cost is the sum of the read and write costs of the compression operation to be performed in the slow disk 130; and determining the score according to the ratio of the interval benefit to the interval cost.

[0119] In one embodiment, the score for the candidate compression interval is defined as the ratio of the interval benefit and the interval cost. Specifically, the interval benefit benefit is the sum of the coldness (indicating the degree of infrequent access) of all keys in the candidate compression interval. The calculation method can be determined by determining whether the key is in the tracking table: if the key is in the tracking table, the coldness of the key is set to 1 / (the clock value of the key + 1); if the key is not in the tracking table, the coldness of the key is set to 1. The coldness of all keys in the candidate compression interval is summarized and summed, and the sum is marked as the interval benefit.

[0120] The interval cost cost is the sum of the read and write costs of the compression operation to be performed in the slow disk 130, specifically, the read cost t 慢 and write cost p*t 快 +(t 慢 -t 置热 ) is summed. That is, p*t 快 +2*t 慢 -t 置热 , where t 慢 is the number of keys in the interval stored in the slow disk 130, t 快 is the number of keys in the interval stored in fast disk 120, p is the cooling ratio set in write-triggered compression, and t 置热 It is the number of keys with the maximum clock value stored in the slow disk 130 and the tracking table within the interval.

[0121] After determining the specific values ​​of the interval benefit and interval cost, the score can be determined based on the ratio of the two.

[0122] Step S720: Determine the hot keys within the target compression interval.

[0123] Step S730: If the hot key is not in the fast disk, select a suitable slab file according to the value of the hot key, and write the hot key value pair in the selected slab file. Otherwise, only delete the hot key value pair from the SST file where it is located, and set the clock value of the hot key to the maximum value.

[0124] In one implementation, after the target compression interval is determined, the hot key and the cold key in the target compression interval are determined respectively. The determination process is related to the timing of triggering the data compression operation, which will be described in detail later.

[0125] The determination and compression transfer of the hot key and the cold key are performed independently, that is, the two can be executed at the same time. In this embodiment, the hot key is first explained. As a whole, the hot key can be a key with a larger clock value stored in the slow disk 130 and the tracking table within the target compression interval. The hot key is stored in the slow disk 130 and needs to be transferred from the slow disk 130 to the fast disk 120, that is, the SST file where it is located is migrated to a suitable slab file. The specific execution process can first determine whether the hot key is in the fast disk 120. If the hot key is not in the fast disk 120, a suitable slab file is selected according to the value size of the hot key, and the hot key value pair is written in the selected slab file; otherwise, the hot key value pair is only deleted from the SST file where it is located. After the hot key is migrated, its corresponding clock value is replaced with the maximum value.

[0126] Step S740: Determine the cooling key within the target compression interval.

[0127] Step S750: remove the cold key from the slab file where it is located and merge and sort it with the SST file containing the cold key within the key range of the slow disk to generate a new SST file to replace the original SST file; after the cold key is moved to the second data set, the clock value of the cold key is maintained unchanged.

[0128] In one embodiment, the cold key is a key with a smaller clock value stored in both the fast disk 120 and the tracking table. The cold key is removed from the fast disk 120, that is, it can be removed from the slab file where it is located and merged and sorted with the SST file containing the cold key within the key range in the slow disk 130 to generate a new SST file to replace the original SST file. After the cold key is migrated, its corresponding clock value remains unchanged.

[0129] After steps S730 and S750, step S760 is performed: the first index is updated according to the updated first data set, and the second index is updated according to the updated second data set.

[0130] In one embodiment, the transfer of the cold key or the hot key will actually cause the first data set in the fast disk 120 or the second data set in the slow disk 130 to change, and both data sets have corresponding index files. To facilitate subsequent retrieval and access to data, it is necessary to update the corresponding first index of the updated first data set, or update the second index according to the updated second data set.

[0131] The system introduces read trigger timing and write trigger timing. Through data compression operations at these two timings, it ensures that frequently accessed keys can be migrated to the fast disk 120, and infrequently accessed keys can be migrated to the slow disk 130. The read trigger timing means that the read hit rate of the fast disk 120 is not high, and relatively more frequently accessed data needs to be migrated to the fast disk 120 to improve the read hit rate of the fast disk 120; the write trigger timing means that the fast disk 120 is about to be full, and some data needs to be swapped out of the fast disk 120 to free up some space on the fast disk 120 to accommodate more frequently accessed data. The system defines cold keys that need to be migrated to the slow disk 130 and hot keys that need to be migrated to the fast disk 120 according to different trigger timings to achieve the goal of compression operation. The main goal of read-triggered compression is to improve the read hit rate of the fast disk 120, while the main goal of write-triggered compression is to free up some space on the fast disk 120.

[0132] According to the above, the specific definition of the cold key and the hot key depends on the triggering timing of the compression operation. Therefore, in this embodiment, how to determine the cold key and the hot key will be described according to the read triggering timing and the write triggering timing respectively.

[0133] At the read trigger time, the cold key is a key with a clock value of 0 stored in both the fast disk 120 and the tracking table within the target compression interval. The hot key is a key with the largest clock value of the first preset percentage selected from all the keys stored in both the slow disk 130 and the tracking table within the target compression interval. For example, assuming that the range of the clock value is 0 to 3, the first preset percentage is set to 60%. There are 100 keys stored in both the slow disk 130 and the tracking table, of which 20 have a clock value of 3, 30 have a clock value of 2, 30 have a clock value of 1, and the remaining 20 have a clock value of 0; there are 80 keys stored in both the fast disk 120 and the tracking table, of which 30 have a clock value of 0. Then the 50 keys with a clock value greater than 1 in the slow disk 130 and the 10 keys randomly sampled with a clock value equal to 1 are all used as hot keys and will be migrated to the fast disk 120 according to the data compression operation; and the 30 keys with a clock value equal to 0 in the fast disk 120 are all migrated to the slow disk 130 as cold keys.

[0134] Furthermore, the data compression operation triggered by the read trigger timing can be executed in multiple rounds: if after a round of compression operation is completed, the hot key ratio belonging to the fast disk 120 in the tracking table is compared with the hot key ratio before the execution of this round of compression operation, and is higher than the user-set increase threshold, then the next round of data compression operation is executed; otherwise, the data compression operation is terminated. The hot key ratio is the ratio of the number of keys belonging to the fast disk 120 in the tracking table to the number of keys in the tracking table.

[0135] Correspondingly, if it is under the write trigger timing, the hot key is the key with the maximum clock value stored in the slow disk 130 and the tracking table within the target compression interval; the cold key is the key with the smallest clock value of the second preset percentage selected from all the keys stored in the fast disk 120 and the tracking table within the target compression interval, and the second preset percentage is used to ensure that there are more cold keys than hot keys. For example, assuming that the range of clock values ​​is 0 to 3, there are 100 keys stored in the fast disk 120 and the tracking table, of which 20 clock values ​​are 3, 30 clock values ​​are 2, 30 clock values ​​are 1, and the remaining 20 clock values ​​are 0; there are 150 keys stored in the slow disk 130 and the tracking table, of which 50 clock values ​​are 3. Then when performing the data compression operation, the second preset percentage can be set to 60%. As a result, 50 keys with clock values ​​less than 2 and 10 randomly sampled keys with clock values ​​equal to 2 in the fast disk 120 are migrated to the slow disk 130 as cold keys; and 50 keys with clock values ​​equal to 3 in the full disk are migrated to the fast disk 120 as hot keys, thereby reducing the key-value pairs in the fast disk 120 by 10, thereby freeing up some space in the fast disk 120.

[0136] The core points of this specific calculation method are: when read triggers compression, the number of hot keys is controlled by a first preset percentage parameter to ensure that enough frequently accessed key value data is migrated to the fast disk 120, thereby improving the read hit rate of the fast disk 120; when write triggers compression, the number of cold keys is controlled by a second preset percentage parameter to ensure that enough infrequently accessed key value data is migrated to the slow disk 130, thereby freeing up part of the fast disk 120 space. Other variations It is not difficult to give other calculation methods for cold keys and hot keys, but as long as it meets the core points of the specific calculation method in the present invention, the variation does not exceed the technical scope to be protected by the present invention.

[0137] Therefore, the distributed key-value storage system 10 proposed in this application can take into account the scalability, reliability and fault tolerance of the distributed system through the architecture design of the hybrid storage medium and the above-mentioned management process, and can make full use of the characteristics of different storage media to optimize the trade-off between storage cost and performance. Among them, the sharding mechanism ensures the scalability of the system, the multi-copy mechanism ensures the reliability of the system, and the combination of the write-ahead log technology and the multi-copy mechanism enables the system to have a strong fault tolerance, that is, a single point failure does not affect the normal access of the system and the fault point can be quickly synchronized to the latest data state after restarting. More importantly, the system can reduce hardware costs while maintaining similar performance levels by storing relatively less frequently accessed data on the fast disk 120 and relatively more infrequently accessed data on the slow disk 130. At the same time, the system introduces a slab-based data layout for the fast disk 120, supports fast random writes and in-situ updates, and introduces the data structure of the SST file in the LSM tree for the slow disk 130, supporting efficient reading and sequential writing. That is to say, the system implements high-speed random reading and writing on the fast disk 120 medium and high-speed sequential reading and writing on the slow disk 130 medium, giving full play to the respective advantages of the two storage media. In addition, the system maintains the first index of the first data set in the memory 110 and the second index of the second data set in the fast disk 120, which improves the overall reading performance of the system. Finally, it should be pointed out that the system manages recent log entries in the fast disk 120. Through the synchronization mechanism of the pre-write log, the data status of multiple copies of the shards can be quickly synchronized after data is written or the cluster node 100 is restarted, ensuring the final consistency of the data.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed key-value storage system, characterized in that: The distributed key-value storage system is composed of multiple cluster nodes. The distributed key-value storage system stores key-value data. The entire key-value data is divided into multiple data shards. Each data shard has a preset number of replicas. The replicas are divided into a unique master replica and multiple slave replicas according to a preset consensus protocol. The master replica and the slave replica corresponding to one data shard are called a replica group. Different replicas of each data shard are stored in different cluster nodes. The cluster node includes a memory, a fast disk and a slow disk, and the cluster node is used to store copies of multiple different data shards; The memory stores a first index and a tracking table; the first index is used to refer to the key value data stored in the fast disk, and adopts a B-tree index structure; the tracking table is used to record the clock values ​​corresponding to a preset number of keys, and the clock values ​​are used to indicate the access frequency of the corresponding keys; The fast disk has a higher storage performance than the slow disk. The data fragments stored in the fast disk are referred to as a first data set. The first data set is associated with the first index. The fast disk also stores a second index and a first write-ahead log, the second index is used to refer to the key-value data stored in the slow disk, the second index adopts an SST file index structure, and the SST file index in the second index includes the offset position of each key-value pair in the corresponding SST file, the start and end ranges of the key in the corresponding SST file, and a Bloom filter for checking whether a key exists in the corresponding SST file; The first write-ahead log is used to record newer log entries of a preset space size; the fast disk uses a file with a slab data layout to save the first data set, and each slab file in the fast disk stores key value data of a preset size range; The key-value data stored in the slow disk is called a second data set, and the second data set is associated with the second index; the slow disk also stores a second write-ahead log, and the second write-ahead log is used to record older log entries that cannot be accommodated by the first write-ahead log; the slow disk uses an SST file structure of an LSM tree to save the second data set, and the SST file stores key-value data within a non-intersecting key range.

2. The distributed key-value storage system according to claim 1, characterized in that: The distributed key-value storage system is configured to include: When a read request is obtained, the read request is parsed to determine a target key, and a first optimal cluster node is located according to the target key, where the first optimal cluster node is a cluster node in the distributed key-value storage system that meets a first optimal condition; Retrieve whether the first index of the first optimal cluster node stores the storage information of the target key in the fast disk; If the storage information of the target key is retrieved according to the first index, the target data is read from the fast disk, and the value of the key is returned after the tracking table update operation is initiated; If the storage information of the target key is not retrieved in the first index, searching whether the storage information of the target key in the slow disk is stored in the second index of the first optimal cluster node; If the storage information of the target key is retrieved according to the second index, the target data is read from the slow disk, and the value of the key is returned after the tracking table update operation is initiated; If the storage information of the target key is not retrieved according to the second index, the tracking table update operation is directly initiated and then the information that the key is not found is returned.

3. The distributed key-value storage system according to claim 1, characterized in that: The distributed key-value storage system is configured to include: When a write request is obtained, the write request is parsed to determine the target key and the corresponding value, the corresponding data shard is located according to the target key, and the corresponding primary copy is determined according to the data shard; Obtaining a first write-ahead log of the primary copy, and adding log entries of the target key and the value in the first write-ahead log; when the storage space occupied by the first write-ahead log exceeds a preset space size, dividing the first write-ahead log into an old log and a new log according to a predetermined condition; maintaining the new log in the fast disk for storage, and converting the old log into the second write-ahead log and storing it in the slow disk; Synchronously sending the log entry to all slave replicas corresponding to the master replica, so that the slave replicas synchronously update the corresponding first write-ahead log; A commit operation is performed on the log entry of the target key and the value.

4. The distributed key-value storage system according to claim 3, characterized in that: The submit operation includes: Retrieve whether the first index corresponding to the primary copy stores the storage information of the target key in the fast disk; If the storage information of the target key is retrieved according to the first index, determine whether the data size of the value is suitable for the slab file where the target key is located; If appropriate, update the slab file in situ according to the value; If not suitable, deleting the target key and the corresponding original value from the slab file, locating the target key and the value to a suitable slab file to store the target key and the value, and associating the positioning information into the first index; If the storage information of the target key is not retrieved in the first index, locate the target key and the value in a suitable slab file according to the target key and the value, and associate the location information into the first index; After writing the target key and the value, a tracking table update operation is initiated.

5. The distributed key-value storage system according to claim 2 or 4, characterized in that: The tracking table update operation includes: If the target key is retrieved in the tracking table, the clock value corresponding to the target key is set to the maximum value; If the target key is not found in the tracking table, adding the key and its corresponding clock value to the tracking table, and setting the clock value to a maximum value; If the tracking table is full, remove the key with a clock value of 0 from the tracking table; If there is no key with a clock value of 0 in the tracking table, the clock values ​​of all keys in the tracking table are gradually reduced in steps of 1 until a key with a clock value of 0 appears in the tracking table after a certain step, and then the key with a clock value of 0 is removed from the tracking table.

6. The distributed key-value storage system according to claim 1, characterized in that: The distributed key-value storage system is configured to include: When the cluster node crashes and restarts, a first index is constructed according to the first data set in the fast disk and stored in the memory; At the same time, the data shards of the cluster nodes are added to the corresponding replica group and marked as slave replicas in the replica group; the corresponding master replica is determined according to the slave replica, and the log entries of the master replica are obtained and synchronized; the last log entry that completes the commit operation is obtained; After the first index is constructed and stored in the memory, the commit operations of all log entries generated after the last log entry that completed the commit operation are executed one by one.

7. The distributed key-value storage system according to claim 1, characterized in that: The distributed key-value storage system is configured to include: The fast disk and the tracking table are monitored, and when the usage rate of the fast disk exceeds the occupancy rate threshold, it is defined as meeting the write trigger timing; when the hot key ratio of the tracking table is lower than a preset ratio threshold, it is defined as meeting the read trigger timing, and the hot key ratio is the ratio of the number of keys in the tracking table belonging to the fast disk to the number of keys in the tracking table; Performing a data compression operation according to the write trigger timing; or performing multiple rounds of the data compression operation according to the read trigger timing until the difference between the hot key ratio of the current round of the tracking table and the hot key ratio of the previous round of the tracking table is lower than a preset boost threshold; The data compression operation includes: Determine a target compression interval; determine a hot key and a cold key in the target compression interval according to the write trigger timing or the read trigger timing; The cold key is removed from the slab file where it is located and merged and sorted with the SST file containing the cold key within the key range of the slow disk to generate a new SST file to replace the original SST file; after the cold key is moved to the second data set, the clock value of the cold key is maintained unchanged; If the hot key is not in the fast disk, a suitable slab file is selected according to the value of the hot key, and the hot key value pair is written into the selected slab file; otherwise, the hot key value pair is only deleted from the SST file where it is located, and the clock value of the hot key is set to the maximum value; The transfer of the cold key or the hot key causes the first data set and the second data set to change, and the first index is updated according to the changed first data set, and the second index is updated according to the changed second data set.

8. The distributed key-value storage system according to claim 7, characterized in that: Determining the target compression interval includes: Divide the entire key value space into multiple continuous key spaces, the continuous key spaces are called compression intervals, and the compression intervals are determined by the key ranges of a preset fixed number of continuous SST files in the slow disk; Randomly selecting a preset number of compression intervals as candidate compression intervals, and calculating the scores of the candidate compression intervals; The candidate compression interval with the highest score is selected as the target compression interval.

9. The distributed key-value storage system according to claim 8, characterized in that: The calculating the score of the candidate compression interval includes: If the key is in the tracking table, the coldness of the key is set to 1 / (the clock value of the key+1); if the key is not in the tracking table, the coldness of the key is set to 1; the coldness of all keys in the candidate compression interval is summarized and summed, and the sum value is marked as the interval benefit; Obtaining an interval cost of the candidate compression interval, where the interval cost is the sum of the read and write costs of the compression operation to be performed in the slow disk; The score is determined according to a ratio of the interval benefit to the interval cost.

10. The distributed key-value storage system according to claim 7, characterized in that: The step of respectively determining a hot key and a cold key in the target compression interval according to the write trigger timing or the read trigger timing includes: At the read triggering time, from all the keys in the target compression interval that are simultaneously stored in the slow disk and the tracking table, a key with the largest clock value of a first preset percentage is selected and marked as the hot key; At the read triggering time, marking the key with a clock value of 0 stored in both the fast disk and the tracking table in the target compression interval as the cold key; At the write triggering time, marking the key with the maximum clock value stored in the slow disk and the tracking table in the target compression interval as the hot key; At the write trigger timing, from all the keys in the target compression interval that are simultaneously stored in the fast disk and the tracking table, the keys with the smallest clock values ​​in a second preset percentage are selected and marked as the cold keys, and the second preset percentage is used to ensure that the cold keys are more than the hot keys.

Images