Frequency-based storage and differentiation management methods

Abstract

This invention presents a frequency-based storage and differentiated management method. Based on the frequency of key-value pairs, this method categorizes them into high-frequency, medium-frequency, and low-frequency key-value pairs. It designs a new frequency-based key-value format to allow for the selection of different key-value formats for data storage at different frequencies. Furthermore, it proposes differentiated management based on key-value pair type. This significantly reduces the amount of data in the log structure merge tree (LSM-tree), further mitigating write amplification and improving read / write performance. It achieves fast read / write access for high-frequency key-value pairs. The values ​​and key addresses of medium-frequency key-value pairs, as well as the entire low-frequency key-value pair, are stored in a value log (ValueLog). Since range queries require random reads, prefetching values ​​from the Value Log improves range query performance.

Description

Technical Field

[0001] This invention belongs to the field of key-value storage, specifically a frequency-based storage and differentiation management method. Background Technology

[0002] With the explosive growth of data volume, the storage and processing of massive amounts of data has become a challenge. Traditional relational databases, due to their poor concurrent read / write capabilities, low processing efficiency, and poor scalability, are unable to meet the demands of processing massive amounts of data. Furthermore, because big data has multiple dimensions and inconsistent fields within each row, storing it in a relational database would result in significant space waste. Against this backdrop, key-value stores, offering advantages such as high performance, strong scalability, and support for tiered deployment, have emerged. Due to the excellent write performance of LSM-trees (Log Structure Merged Trees), key-value stores based on LSM-trees are gradually becoming the mainstream persistent key-value stores. However, LSM-trees can suffer from read / write amplification, impacting the efficiency of data retrieval for users.

[0003] To address the read / write performance issues of LSM-tree log structures, a popularity-based key-value pairing technique has emerged to optimize their performance. However, traditional popularity-based key-value pairing ignores the actual highly skewed workloads, such as the daily trending topics lists generated by social media platforms like Facebook, YouTube, TikTok, and Weibo. Compared to ordinary data, trending topics lists have a significantly higher access frequency, and the access frequency of the top and bottom trending topics also differs greatly. Simply dividing all data into hot and cold data does not account for the significant frequency differences between different data pairs, resulting in less efficient read / write performance under large-scale data storage. Therefore, to better adapt to the needs of real-world production and daily life, it is necessary to maximize the performance of key-value storage systems, building upon existing read / write performance optimizations to ensure that data of different frequencies achieves its optimal performance. For example, high-frequency data should have increasingly efficient read / write access performance, medium-frequency data should maintain a certain level of read / write access performance while offering good range query performance, and low-frequency data should have the most efficient range query performance compared to the other two data types. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a frequency-based storage and differentiated management method. Based on the frequency of key-value pairs, key-value pairs are categorized into high-frequency, medium-frequency, and low-frequency pairs. A new frequency-based key-value format is designed to allow for the selection of different key-value formats for data storage at different frequencies. Furthermore, differentiated management based on key-value pair type is proposed. This significantly reduces the amount of data in the log structure merge tree (LSM-tree), further mitigating write amplification and improving read / write performance. High-frequency key-value pairs can achieve fast read / write access. The values ​​and key addresses of medium-frequency key-value pairs, as well as the entire low-frequency key-value pair, are stored in a value log. Since range queries require random reads, prefetching values ​​from the value log improves range query performance.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A frequency-based storage and differentiation management method, characterized by comprising the following steps:

[0007] 1) Classify key-value pairs based on frequency into high-frequency, medium-frequency, and low-frequency key-value pairs:

[0008] 2) Design corresponding key-value formats for different categories based on key-value pairs:

[0009] For high-frequency key-value pairs, the key-value format only needs to add the key-value pair type field H, without making any other changes;

[0010] For medium-frequency key-value pairs, their key-value formats are designed separately. The key format contains a key-value pair type field M, a key field, and a value address field value addr. The value format contains four fields: key-value pair type field M, segment number segment_no, offset, and actual value. The value address value addr represents the address of the actual value value in the value log, the segment number segment_no represents which segment the actual value value is in the value log, and the offset represents the offset value of the actual value value within the segment.

[0011] For low-frequency key-value pairs, the key-value pair format is: key-value pair type field L, segment number segment_no, offset field, key and value;

[0012] 3) Implement differentiated management for key-value pairs of different categories:

[0013] High-frequency key-value pairs are stored in the log structure merge tree (LSM-tree); medium-frequency key-value pairs are separated into key and value, with the key and value addresses stored in the LSM-tree and the actual value stored in the value log; low-frequency key-value pairs are placed directly in the value log. When reading low-frequency key-value pairs, the parallel I / O characteristics of the solid-state drive are utilized to prefetch the value from the value log during range queries.

[0014] In step 1), for a sequence of key-value pairs in a block of data, the frequency of each key-value pair is calculated, the corresponding confidence level is obtained, and the pairs are sorted. The top 10% of key-value pairs by confidence level are defined as high-frequency key-value pairs; the top 11% to 40% of key-value pairs by confidence level are defined as medium-frequency key-value pairs; and the remaining 60% of key-value pairs are defined as low-frequency key-value pairs. The specific process is as follows:

[0015] 11) Input data block: key-value pair sequence (KV1, KV2, ..., KV...) n ), where n is the number of key-value pairs in the key-value pair sequence;

[0016] 12) Calculate the frequency k of each key-value pair per unit time. i And based on frequency k i Sort the key-value pairs:

[0017]

[0018] Among them: PR i and PW i Key-value pairs (KV) i The read frequency and write frequency, where T is the time interval;

[0019] 13) Calculate the frequency gap d i :

[0020] d i =k i+1 -k i

[0021] 14) Calculate the confidence level Δ i :

[0022]

[0023] 15) Use simple selection sort to sort the confidence scores of the key-value pairs in ascending order. The sorted confidence score sequence is as follows:

[0024] Δ′1,Δ′2,…,Δ′ n-1

[0025] 16) Divide the key-value pair sequence into three branches according to a confidence ratio of 6:3:1:

[0026] The sorted confidence sequence is divided into three branches according to the confidence level, including the high-frequency branch [0, Δ']. 0.1(n-1) ], Mid-frequency branch [Δ' 0.2(n-1) ,Δ' 0.4(n-1) ] and low-frequency branches [Δ' 0.5(n-1) ,Δ' (n-1) Find the key-value pairs corresponding to the sorted confidence levels of the three branches, and store the key-value pairs in order of confidence level to obtain the high-frequency key-value pair sequence KV. H Mid-frequency key-value pair sequences KV M and low-frequency key-value pair sequences KV L ,

[0027] Low frequency:

[0028] Mid-frequency:

[0029] High frequency:

[0030] Where l ranges from 1 to L, where L is the number of sequences in the low-frequency branch; m ranges from 1 to M, where M is the number of sequences in the mid-frequency branch; h ranges from 1 to H, where H is the number of sequences in the high-frequency branch; and KV l For low-frequency key-value pair sequences KV L Key-value pairs in KV m For mid-frequency key-value pair sequences KV M Key-value pairs in KV h For high-frequency key-value pair sequences KV H Key-value pairs in;

[0031] [0, Δ' 0.1(n-1) [] represents the top 10% of the sorted confidence score sequence. The subscript n-1 indicates the length of the confidence score sequence, and the subscripts 0.1, 0.2, 0.4, and 0.5 indicate the relative position of the current confidence score in the sorted confidence score sequence.

[0032] The specific process of step 3) is as follows:

[0033] 31) Store high-frequency key-value pairs in the sorted string table SST file of the log structure merge tree LSM-tree. The high-frequency key-value pairs are sorted in order of confidence in the sorted string table SST file.

[0034] 32) Store the keys and values ​​of medium-frequency key-value pairs separately:

[0035] Stored values: The actual values ​​of medium-frequency key-value pairs are stored in the Value Log, which contains the key-value pair type field M; the Value Log file uses a segmented storage method, so it contains the segment number (segment_no) and offset (offset) of the key-value pair stored in the Value Log, as well as the actual value (value).

[0036] Storage Key: The key and value addresses are stored in the sorted string table (SST) file of the log structure merge tree (LSM-tree). In the SST file, mid-frequency and high-frequency key-value pairs are both ordered by confidence level. These mid-frequency and high-frequency key-value pairs are stored separately in different SST files, and these SST files are distinguished by their respective SST values. H This represents a sorted string table SST containing high-frequency key-value pairs, named SST. M SST represents a sorted string table storing mid-frequency key-value pairs, so that they can be quickly identified when querying key-value pairs;

[0037] In the log structure merge tree (LSM-tree), each level places high-frequency key-value pairs and medium-frequency key-value pairs in a 1:2 ratio. The first level (C0) contains one high-frequency key-value pair and two medium-frequency key-value pairs, the second level (C2) contains two high-frequency key-value pairs and four medium-frequency key-value pairs, and each subsequent level contains three more key-value pairs than the previous level. When the previous level is full, the key-value pairs are merged into the next level.

[0038] 33) Low-frequency key-value pairs only require range read performance and do not require access speed. Store the key values ​​of low-frequency key-value pairs in the Value Log and separate them from the value storage of medium-frequency key-value pairs.

[0039] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0040] 1) This invention calculates confidence levels based on read / write frequency to sort key-value pairs. It uses a highly efficient three-branch model to calculate frequency gaps and confidence levels, and employs a simple selection sorting method to sort frequency, confidence, and key-value pairs. This approach is applicable to large-scale key-value pair sequences, yielding more accurate and detailed sorting results. By using confidence level as the classification criterion, it considers the differences in read / write frequency between key-value pairs, resulting in more accurate classification.

[0041] 2) This invention categorizes key-value pairs into 10% high-frequency key-value pairs, 30% medium-frequency key-value pairs, and 60% low-frequency key-value pairs. Studying workloads helps us optimize key-value storage systems more effectively, thereby improving the performance of the storage system under specific workloads. This invention is more adaptable to actual production and living environments, meeting the key characteristics of workloads in industrial production environments: firstly, low-frequency key-value pairs occupy a significant portion of the space; secondly, 3 / 10 of the medium-frequency key-value pairs account for 30% of access requests; and thirdly, 70% of access requests are concentrated on a small portion of high-frequency data.

[0042] 3) This invention designs corresponding key-value formats for different types of key-value pairs. Traditional key-value pairs use the same key-value format, which requires more effort when searching for frequently accessed key-value pairs. Therefore, this invention designs different key-value formats to help accelerate access to frequently accessed key-value pairs.

[0043] 4) This invention employs differentiated management for key-value pairs of varying frequencies. High-frequency key-value pairs are stored in the Log Structure Merging Tree (LSM-tree); medium-frequency key-value pairs are separated into key and value pairs, with the key and value address stored in the LSM-tree and the actual value stored in the Value Log; low-frequency key-value pairs are stored in the Value Log. In the LSM-tree, the key and value addresses of high-frequency and medium-frequency key-value pairs are stored separately and marked for differentiation. Similarly, in the Value Log, the key and value addresses of low-frequency and medium-frequency key-value pairs are stored separately and differentiated. Therefore, this differentiated storage method helps reduce the amount of data in the LSM-tree and improves the read / write access speed of high-frequency key-value pairs. Furthermore, the separation of key and value for medium-frequency key-value pairs satisfies both read / write access speed and range query requirements. Finally, placing low-frequency key-value pairs in the Value Log improves the performance of range queries. Attached Figure Description

[0044] Figure 1 Overall architecture diagram of frequency-based storage and differentiated management methods.

[0045] Figure 2 Performance impact diagram of key-value pair types.

[0046] Figure 3 High-frequency key-value pair key-value format diagram.

[0047] Figure 4 Key-value format diagram of medium-frequency key-value pairs.

[0048] Figure 5Low-frequency key-value pair key-value format diagram.

[0049] Figure 6 Log structure merge tree (LSM-tree) structure diagram.

[0050] Figure 7 Value Log structure diagram. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of protection of this application.

[0052] This invention designs a storage and differentiation management method with high read / write performance and fast access based on key-value pair frequency, including the following steps:

[0053] 1) Classify key-value pairs based on frequency into high-frequency, medium-frequency, and low-frequency key-value pairs:

[0054] For a sequence of key-value pairs in a block of data, calculate the frequency of each key-value pair, obtain the corresponding confidence level, and sort them. Define the top 10% of key-value pairs by confidence level as high-frequency key-value pairs. The read / write frequency of these key-value pairs increases exponentially per unit time, and memory and CPU resources should be allocated to them. Define the top 11% to 40% of key-value pairs by confidence level as medium-frequency key-value pairs. The read / write frequency of these key-value pairs increases linearly per unit time, and resource allocation should be allocated to them moderately. Define the remaining 60% of key-value pairs as low-frequency key-value pairs. The read / write frequency of these key-value pairs remains basically constant per unit time.

[0055] 2) Design corresponding key-value formats for different categories based on key-value pairs:

[0056] For high-frequency key-value pairs, since high-speed read and write access efficiency is required, their key-value format only needs to add a key-value pair type field without any other changes. For medium-frequency key-value pairs, it is necessary to ensure both fast read and write access speed and a certain range query performance, so their key-value formats are designed separately. For low-frequency key-value pairs, there are no requirements for read and write speed, only the range query performance needs to be ensured, so their key-value format is similar to that of high-frequency key-value pairs.

[0057] 3) Implement differentiated management for key-value pairs of different categories:

[0058] Since high-frequency key-value pairs have the highest read / write frequency, and fast read / write access efficiency needs to be ensured, their key-value pairs are stored in the Log Structure Merged Tree (LSM-tree). To reduce the data volume of the LSM-tree, a key-value separation operation is performed on medium-frequency key-value pairs, storing the key and value addresses in the LSM-tree and the actual values ​​in the Value Log. For low-frequency key-value pairs, the key-value pairs are directly placed in the Value Log. To improve the efficiency of range queries, when reading low-frequency key-value pairs, the parallel I / O characteristics of the solid-state drive are utilized to prefetch values ​​from the Value Log during range queries.

[0059] Further, step 1) includes:

[0060] 11) Input data block: key-value pair sequence (KV1, KV2, ..., KV...) n ), where n is the number of key-value pairs in the key-value pair sequence.

[0061] 12) Calculate the frequency k of each key-value pair per unit time. i And based on frequency k i Sort the key-value pairs:

[0062]

[0063] Among them: PR i and PW i Key-value pairs (KV) i The read frequency and write frequency, where T is the time interval.

[0064] 14) Calculate the frequency gap d i :

[0065] d i =k i+1 -k i

[0066] 14) Calculate the confidence level Δ i :

[0067]

[0068] 15) Since the key-value pair sequence contains a large amount of information, a simple selection sort is used to sort the key-value pair confidence levels from smallest to largest. In this case, the time complexity is O(n^2). 2 ), where n is the problem size, which is the length of the sequence, and the sorted confidence sequence is:

[0069] Δ′1,Δ′2,…,Δ′ n-1

[0070] 16) Based on the efficient three-branch model, the key-value pair sequence is divided into three branches according to the confidence ratio of 6:3:1: the sorted confidence sequence is divided into three branches according to the confidence level, including the high-frequency branch [0, Δ' 0.1(n-1) ], Mid-frequency branch [Δ' 0.2(n-1) ,Δ' 0.4(n-1) ] and low-frequency branches [Δ' 0.5(n-1) ,Δ' (n-1) Find the key-value pairs corresponding to the sorted confidence levels of the three branches, and store the key-value pairs in order of confidence level to obtain the high-frequency key-value pair sequence KV. H Mid-frequency key-value pair sequences KV M and low-frequency key-value pair sequences KV L ,

[0071] Low frequency:

[0072] Mid-frequency:

[0073] High frequency:

[0074] Where l ranges from 1 to L, where L is the number of sequences in the low-frequency branch; m ranges from 1 to M, where M is the number of sequences in the mid-frequency branch; h ranges from 1 to H, where H is the number of sequences in the high-frequency branch; and KV l For low-frequency key-value pair sequences KV L Key-value pairs in KV m For mid-frequency key-value pair sequences KV M Key-value pairs in KV h For high-frequency key-value pair sequences KV H Key-value pairs in; [0, Δ' 0.1(n-1) [] represents the top 10% of the sorted confidence sequence. The subscript n-1 indicates the length of the confidence sequence, and the subscripts 0.1, 0.2, 0.4, and 0.5 indicate the relative position of the current confidence level in the sorted confidence sequence.

[0075] This key-value pair classification method categorizes key-value pairs into three types by analyzing the confidence level of key-value pair read / write frequency per unit time. (See also...) Figure 2The importance of read / write access speed and range query performance for high, medium, and low frequency key-value pairs is represented by the thickness of the arrows, with thicker arrows indicating greater importance. High-frequency key-value pairs correspond to real-time activities in daily life and production (such as online promotions and breaking news), attracting billions of visits in a short period. Therefore, high-speed read / write access is crucial for these high-frequency key-value pairs. Medium-frequency key-value pairs do not have the same high speed requirements as high-frequency pairs, but their read / write speed must be higher than that of low-frequency pairs. Their overall read / write speed falls between high and low frequency key-value pairs, and they have certain requirements for range queries. Low-frequency key-value pairs have a very low read / write frequency per unit time, so their read / write speed is not critical, but range query performance for low-frequency key-value pairs needs to be guaranteed and improved.

[0076] Furthermore, step 2) includes:

[0077] 21) Design a high-frequency key-value pair format. (See reference...) Figure 3 High-frequency key-value pairs require high-speed read and write access efficiency. Therefore, for high-frequency key-value pairs, it is only necessary to add a key-value pair type field H to the key-value pair. The key-value pair type field includes H (representing high-frequency key-value pairs), M (representing medium-frequency key-value pairs), and L (representing low-frequency key-value pairs).

[0078] 22) Design a key-value pair format for medium-frequency key-value pairs. (See also...) Figure 4 The read / write efficiency and range query performance requirements for medium-frequency key-value pairs fall between those of high-frequency and low-frequency key-value pairs. Therefore, the key and value formats need to be designed separately. The key format contains three fields: key-value pair type (type), key, and value address (addr). The value format contains four fields: key-value pair type (M), segment number (segment_no), offset, and value. The value address (value addr) represents the address of the actual value (value) in the value log. The segment number (segment_no) represents the segment in the value log where the value (value) is located. The offset (offset) represents the offset of the value (value) within that segment.

[0079] 23) Design a low-frequency key-value pair format. Low-frequency key-value pairs only require high performance for range queries; read / write access speed is not a concern. Therefore, refer to [reference needed]. Figure 5To add a key-value pair type field L, as well as a segment number (segment_no) and offset field in the Value Log, to the low-frequency key-value pair key-value pair format, the low-frequency key-value pair key-value pair key-value pair format is: key-value pair type field L, segment number (segment_no), offset field, key, and value.

[0080] Furthermore, step 3) includes:

[0081] 31) See Figure 6 High-frequency key-value pairs are stored in the sorted string table SST file of the log structure merge tree (LSM-tree). These high-frequency key-value pairs are ordered by confidence level within the SST file. The LSM-tree offers excellent read / write performance, meeting the high speed requirements for accessing high-frequency key-value pairs; therefore, both the keys and values ​​of these pairs are stored within the LSM-tree. The LSM-tree in the diagram has levels C0, C1, ~C... N N+1 represents the total number of levels in the log structure merge tree (LSM-tree). In each level, high-frequency and medium-frequency key-value pairs are placed in a 1:2 ratio. In the first level C0, high-frequency and medium-frequency key-value pairs are placed in a 1:2 ratio, i.e., one high-frequency key-value pair and two medium-frequency key-value pairs. In the first level C1, high-frequency and medium-frequency key-value pairs are placed in a 1:2 ratio, i.e., two high-frequency key-value pairs and four medium-frequency key-value pairs. The next level stores three more key-value pairs than the previous level. When the previous level is full, the key-value pairs are merged into the next level.

[0082] 32) To simultaneously meet the requirements of both read / write access speed and range query performance for medium-frequency key-value pairs, the keys and values ​​of medium-frequency key-value pairs are stored separately:

[0083] Stored value: See Figure 7 The actual values ​​of medium-frequency key-value pairs are stored in the Value Log, which contains the key-value pair type field M. The Value Log file uses segmented storage, so it contains the segment number (segment_no), offset, and actual value of the key-value pair stored in the Value Log.

[0084] Storage key: See Figure 6The key and value addresses are stored in the sorted string table (SST) file of the log structure merge tree (LSM-tree). In the SST file, mid-frequency and high-frequency key-value pairs are both ordered by confidence level. The mid-frequency and high-frequency key-value pairs are stored separately in different SST files, and these SST files are distinguished by their respective names. H This represents a sorted string table SST containing high-frequency key-value pairs, named SST. M This represents the sorted string table SST, which stores the key-value pairs with medium frequency, so that they can be quickly identified when querying key-value pairs.

[0085] 33) See Figure 7 Low-frequency key-value pairs only require range read performance and are not required to access speed. Therefore, the key values ​​of low-frequency key-value pairs are stored in the Value Log and are stored separately from the value storage of medium-frequency key-value pairs.

[0086] Example 1

[0087] Appendix Figure 1 This embodiment presents a schematic diagram of the architecture of the frequency-based storage and differentiation management method, including frequency-based key-value pair sequence classification, key-value pair type-based key-value pair format design, and key-value pair differentiation management. Specifically, it includes the following steps:

[0088] Step 1: Frequency-based key-value pair sequence classification

[0089] 1) Input block data: key-value pair sequence (KV1, KV2, ..., KV...) 10 And the corresponding read / write frequency within 10 seconds:

[0090] ((34,35),(62,88),(58,6),(98,94),(42,12),(15,59),(74,76),(2,44),(87,61),(48,20))

[0091] 2) Calculate frequency At this time, T = 10s, and the corresponding frequency sequence is:

[0092] (6.9, 15, 6.4, 19.2, 5.4, 7.4, 15, 4.6, 14.8, 6.8)

[0093] 3) Calculate the frequency gap d i =k i+1 -k i The corresponding frequency gap sequence is:

[0094] (8.1, -8.6, 12.8, -13.8, 2, 7.6, -10.4, 10.2, -8)

[0095] 4) Calculate the confidence level The corresponding confidence sequence is:

[0096] (1.174, -0.573, 2, -0.719, 0.370, 1.027, -0.693, 2.217, -0.541)

[0097] 5) Sort the confidence sequences in ascending order using simple selection sort:

[0098] (-0.719, -0.693, -0.573, -0.541, 0.370, 1.027, 1.174, 2, 2.217)

[0099] 6) Divide the confidence sequence into three parts according to a ratio of 1:3:6.

[0100] A: (-0.719)

[0101] B:(-0.693, -0.573, -0.541)

[0102] C: (0.370, 1.027, 1.174, 2, 2.217)

[0103] 7) Based on the confidence level, find the corresponding frequency, and thus find the corresponding key-value pairs.

[0104] A:

[0105] B:

[0106] C:

[0107] Sequence A is a high-frequency key-value pair sequence, sequence B is a medium-frequency key-value pair sequence, and sequence C is a low-frequency key-value pair sequence.

[0108] Step 2: Design key-value format based on key-value pair type

[0109] Different key-value storage formats are designed for key-value pairs with different frequencies to adapt to different performance requirements.

[0110] 1) See Figure 3 Add a key-value pair type field H to high-frequency key-value pairs (KV4).

[0111] 2) See Figure 4The key and value of the mid-frequency key-value pairs (KV7, KV3, KV9) are designed separately. The key format adds a key-value pair type field M and a value address information field value addr. The value format contains a key-value pair type field M, the segment number (segment_no) of the actual value in the value log, the offset, and the actual value value.

[0112] 3) See Figure 5 Low-frequency key-value pairs (KV5, KV6, KV1, KV3, KV8, KV) 10 It also includes the key-value pair type field L, the segment number of the key-value pair in the value log, offset, key, and value.

[0113] Step 3: Key-value pair differentiation management

[0114] Because different types of key-value pairs have different frequencies and different requirements for read / write access speed and range query performance, a differentiated approach is used to manage different types of key-value pairs.

[0115] 1) See Figure 6 High-frequency key-value pairs (KV4) data have high requirements for read and write access speed. Therefore, storing high-frequency key-value pairs (KV4) in the sorted string table SST file of the log structure merge tree (LSM-tree) can achieve fast read and write access efficiency to adapt to the rapid growth of hot data access in a short period of time.

[0116] 2) To simultaneously meet the requirements of read / write access speed and range query performance for medium-frequency key-value pairs, the keys and values ​​of these pairs are stored separately. (See also...) Figure 6 The keys (key7, key3, key9) are also stored in the sorted string table SST file of the LSM-tree log structure to maintain a certain read / write access speed and to separate it from the sorted string table SST file storing high-frequency key-value pairs. See also Figure 7 The actual values ​​(value7, value3, value9) are stored in different segments of the Value Log, thus meeting the performance requirements for range queries of medium-frequency key-value pairs.

[0117] 3) Low-frequency key-value pairs only require range query performance, and access speed is not a requirement. Therefore, the key values ​​of low-frequency key-value pairs (KV5, KV6, KV1, KV3, KV8, KV) are... 10 It is stored in the Value Log, see [link / reference] Figure 7 It is also stored separately from the value storage of mid-frequency key-value pairs.

[0118] In this embodiment, by calculating the frequency of different key-value pairs, a highly efficient three-branch model (utility three-branch decision model) is used to divide the key-value pair data into high-frequency, medium-frequency, and low-frequency key-value pairs in a 1:3:6 ratio. Corresponding key-value formats are designed based on different types of key-value pairs, and a differentiated approach is used to manage different key-value pairs. This ensures that high-frequency key-value pairs have fast read / write access speeds, medium-frequency key-value pairs simultaneously meet the requirements for read / write access speed and range query performance, and low-frequency key-value pairs have high range query performance. Therefore, this invention is more adaptable to actual workloads and has high efficiency compared to traditional key-value separation methods.

[0119] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A frequency-based storage and differentiation management method, characterized in that, The method includes the following steps: 1) Classify key-value pairs based on frequency into high-frequency, medium-frequency, and low-frequency key-value pairs: 2) Design corresponding key-value formats for different categories based on key-value pairs: For high-frequency key-value pairs, the key-value format only needs to add the key-value pair type field H, without making any other changes; For medium-frequency key-value pairs, their key-value formats are designed separately. The key format contains a key-value pair type field M, a key field, and a value address field value addr. The value format contains four fields: key-value pair type field M, segment number segment_no, offset, and actual value. The value address value addr represents the address of the actual value value in the value log, the segment number segment_no represents which segment the actual value value is in the value log, and the offset represents the offset value of the actual value value within the segment. For low-frequency key-value pairs, the key-value pair format is: key-value pair type field L, segment number segment_no, offset field, key and value; 3) Implement differentiated management for key-value pairs of different categories: High-frequency key-value pairs are stored in the log structure merge tree (LSM-tree); medium-frequency key-value pairs are separated into key and value, with the key and value addresses stored in the LSM-tree and the actual value stored in the value log; low-frequency key-value pairs are placed directly in the value log. When reading low-frequency key-value pairs, the parallel I / O characteristics of the solid-state drive are utilized to prefetch the value from the value log during range queries.

2. The frequency-based storage and differentiated management method according to claim 1, characterized in that, In step 1), for a sequence of key-value pairs in a block of data, the frequency of each key-value pair is calculated, the corresponding confidence level is obtained, and the pairs are sorted. The top 10% of key-value pairs by confidence level are defined as high-frequency key-value pairs; the top 11% to 40% of key-value pairs by confidence level are defined as medium-frequency key-value pairs; and the remaining 60% of key-value pairs are defined as low-frequency key-value pairs. The specific process is as follows: 11) Input data block: key-value pair sequence (KV1, KV2, ..., KV...) n ), where n is the number of key-value pairs in the key-value pair sequence; 12) Calculate the frequency k of each key-value pair per unit time. i And based on frequency k i Sort the key-value pairs: Among them: PR i and PW i Key-value pairs (KV) i The read frequency and write frequency, where T is the time interval; 13) Calculate the frequency gap d i : d i =k i+1 -k i 14) Calculate the confidence level Δ i : 15) Use simple selection sort to sort the confidence scores of the key-value pairs in ascending order. The sorted confidence score sequence is as follows: D′1,D′2,…,D′ n-1 16) Divide the key-value pair sequence into three branches according to a confidence ratio of 6:3:1: The sorted confidence sequence is divided into three branches according to the confidence level, including the high-frequency branch [0, Δ']. 0.1(n-1) ], Mid-frequency branch [Δ' 0.2(n-1) ,Δ' 0.4(n-1) ] and low-frequency branches [Δ' 0.5(n-1) ,Δ' (n-1) Find the key-value pairs corresponding to the sorted confidence levels of the three branches, and store the key-value pairs in order of confidence level to obtain the high-frequency key-value pair sequence KV. H Mid-frequency key-value pair sequences KV M and low-frequency key-value pair sequences KV L , Low frequency: Mid-frequency: High frequency: Where l ranges from 1 to L, where L is the number of sequences in the low-frequency branch; m ranges from 1 to M, where M is the number of sequences in the mid-frequency branch; h ranges from 1 to H, where H is the number of sequences in the high-frequency branch; and KV l For low-frequency key-value pair sequences KV L Key-value pairs in KV m For mid-frequency key-value pair sequences KV M Key-value pairs in KV h For high-frequency key-value pair sequences KV H Key-value pairs in; [0, Δ' 0.1(n-1) [] represents the top 10% of the sorted confidence score sequence. The subscript n-1 indicates the length of the confidence score sequence, and the subscripts 0.1, 0.2, 0.4, and 0.5 indicate the relative position of the current confidence score in the sorted confidence score sequence.

3. The frequency-based storage and differentiated management method according to claim 1, characterized in that, The specific process of step 3) is as follows: 31) Store high-frequency key-value pairs in the sorted string table SST file of the log structure merge tree LSM-tree. The high-frequency key-value pairs are sorted in order of confidence in the sorted string table SST file. 32) Store the keys and values ​​of medium-frequency key-value pairs separately: Stored values: The actual values ​​of medium-frequency key-value pairs are stored in the Value Log, which contains the key-value pair type field M; the Value Log file uses a segmented storage method, so it contains the segment number (segment_no) and offset (offset) of the key-value pair stored in the Value Log, as well as the actual value (value). Storage Key: The key and value addresses are stored in the sorted string table (SST) file of the log structure merge tree (LSM-tree). In the SST file, mid-frequency and high-frequency key-value pairs are both ordered by confidence level. These mid-frequency and high-frequency key-value pairs are stored separately in different SST files, and these SST files are distinguished by their respective SST values. H This represents a sorted string table SST containing high-frequency key-value pairs, named SST. M SST represents a sorted string table storing mid-frequency key-value pairs, so that they can be quickly identified when querying key-value pairs; In the log structure merge tree (LSM-tree), each level places high-frequency key-value pairs and medium-frequency key-value pairs in a 1:2 ratio. The first level (C0) contains one high-frequency key-value pair and two medium-frequency key-value pairs, the second level (C2) contains two high-frequency key-value pairs and four medium-frequency key-value pairs, and each subsequent level contains three more key-value pairs than the previous level. When the previous level is full, the key-value pairs are merged into the next level. 33) Low-frequency key-value pairs only require range read performance and do not require access speed. Store the key values ​​of low-frequency key-value pairs in the Value Log and separate them from the value storage of medium-frequency key-value pairs.

Images