Key value storage system

Abstract

The invention provides a key value storage system. The key value storage system comprises a data pre-reading module, a cache management module and an index adjustment module. And the data pre-reading module is used for predicting the data blocks which can be accessed in the preset time period and loading the data blocks to the cache in advance so as to reduce reading delay and disk I / O (Input / Output) operation, so that the reading efficiency is improved. The cache management module is used for monitoring the running state of the key value storage system in real time and dynamically adjusting the cache space based on the running state, it is ensured that the cache meets the high hit rate, memory use can be reasonably controlled, resource waste is avoided, and the reading performance and the response speed are further improved. The index adjusting module is used for dynamically adjusting an index structure according to the access frequency of the key value pair, reducing disk I / O operation during reading, improving the searching speed and reducing the searching cost. Therefore, according to the key value storage system, the negative influence caused by read amplification can be reduced, the response speed of the system is improved, and faster and more efficient data access experience is provided for a user.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a key-value storage system. Background Art

[0002] In the era of big data and cloud computing, key-value storage systems play a key role as basic components in many fields, especially in scenarios that require efficient data read and write operations. Key-value storage systems based on LSM-Tree (Log-Structured MergeTree, a hierarchical, ordered, disk-oriented data structure) perform well in processing large-scale data sets due to their efficient write performance and ordered data management capabilities.

[0003] However, existing key-value storage systems often suffer from a "read amplification" phenomenon when reading data, that is, the amount of data actually read is much larger than the amount of data requested. Summary of the invention

[0004] In view of this, the present application provides a key-value storage system to solve the problem in the prior art that the amount of data actually read when reading data is much larger than the amount of data requested.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] The first aspect of the present application discloses a key-value storage system, comprising:

[0007] Data pre-reading module, cache management module and index adjustment module;

[0008] The data pre-reading module is used to predict the data blocks that will be accessed within a preset time period and load the data blocks into the cache in advance;

[0009] The cache management module is used to monitor the operating status of the key-value storage system in real time and dynamically adjust the cache space based on the operating status;

[0010] The index adjustment module is used to dynamically adjust the index structure according to the access frequency of the key-value pair.

[0011] Optionally, in the above system, the data pre-reading module includes:

[0012] The data collection submodule is used to collect log data of each storage node;

[0013] A feature engineering submodule is used to construct a feature vector of each data block; the feature vector includes at least a recent access time feature vector, an access frequency feature vector and an access interval feature vector;

[0014] A heat score submodule, used to calculate a heat score of each data block based on the feature vector;

[0015] The model training submodule is used to build the data pre-reading initial model and use cross-validation to train the model to obtain the data pre-reading model;

[0016] The pre-reading decision submodule is used to generate a pre-reading data block list by using the data pre-reading model to predict and perform data pre-reading according to the pre-reading data block list.

[0017] Optionally, in the above system, when the heat score submodule calculates the heat score of each data block, it is used to:

[0018] Determining a weight factor for each of the eigenvectors;

[0019] For each data block, based on each of the feature vectors and the weight factors corresponding to each of the feature vectors, a heat score of each data block is calculated using a preset heat score calculation formula.

[0020] Optionally, in the above system, the cache management module includes:

[0021] A monitoring submodule, used to monitor the load data of the key-value storage system; wherein the load data includes memory usage, cache hit rate, CPU usage and disk I / O status;

[0022] An adjustment submodule, used for adjusting the cache space of the key-value storage system according to the load data;

[0023] The cleaning submodule is used to trigger the cache cleaning process and release inactive data blocks when the cache usage reaches a preset upper limit.

[0024] Optionally, in the above system, when the adjustment submodule adjusts the cache space of the key-value storage system according to the load data, it is used to:

[0025] Get the current system load value, the maximum expected system load value, the minimum cache value, and the maximum cache value;

[0026] Based on the maximum expected value of the system load, the system load data, the minimum cache value and the maximum cache value, a target cache value is calculated by a preset cache calculation formula;

[0027] The size of the cache space is adjusted according to the target cache value.

[0028] Optionally, in the above system, the index adjustment module includes:

[0029] The statistics submodule is used to count the access frequency of each key-value pair;

[0030] An index structure adjustment submodule, used to adjust the index structure based on the access frequency;

[0031] Maintenance submodule, used to regularly update the index and optimize index storage.

[0032] Optionally, in the above system, when the index structure adjustment submodule adjusts the index structure based on the access frequency, it is used to:

[0033] Comparing the access frequency of each key-value pair with a preset threshold value; the preset threshold value is calculated based on the average access frequency of all key-value pairs and the standard deviation of the access frequency of all key-value pairs;

[0034] The key-value pairs whose access frequency is greater than the threshold are screened out, and an additional index layer is created for each of the screened key-value pairs.

[0035] Optionally, in the above system, the data pre-reading module further includes:

[0036] The preprocessing submodule is used to clean the log data.

[0037] Optionally, in the above system, the data pre-reading module further includes:

[0038] The optimization submodule is used to collect access data of the pre-read data blocks and optimize the data pre-reading module.

[0039] Optionally, in the above system, the cache management module further includes:

[0040] The evaluation submodule is used to evaluate the effectiveness of the cache strategy of the cache management module according to a preset time period.

[0041] It can be seen from the above technical solutions that a key-value storage system provided by the present application includes a data pre-reading module, a cache management module and an index adjustment module. The data pre-reading module is used to predict the data blocks that will be accessed within a preset time period, and load the data blocks into the cache in advance to reduce read latency and disk I / O operations, thereby improving read efficiency. The cache management module is used to monitor the operating status of the key-value storage system in real time, and dynamically adjust the cache space based on the operating status to ensure that the cache can both meet the high hit rate and reasonably control the use of memory, avoid resource waste, and further improve the reading performance and response speed. The index adjustment module is used to dynamically adjust the index structure according to the access frequency of the key-value pair, reduce the disk I / O operations during reading, improve the search speed and reduce the search cost. It can be seen that the key-value storage system of the present application can reduce the negative impact of read amplification, improve the system response speed, provide users with a faster and more efficient data access experience, and solve the problem in the prior art that the amount of data actually read when reading data is much larger than the amount of data requested. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a key-value storage system disclosed in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a data pre-reading module disclosed in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a cache management module disclosed in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of an index adjustment module disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0049] Furthermore, in this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0050] As can be seen from the background technology, existing key-value storage systems usually have a "read amplification" phenomenon when reading data, that is, the amount of data actually read is much larger than the amount of data requested.

[0051] In view of this, the present application provides a key-value storage system to solve the problem in the prior art that the amount of data actually read when reading data is much larger than the amount of data requested.

[0052] The present application embodiment provides a key-value storage system, such as Figure 1 As shown, specifically including:

[0053] A data pre-reading module 101 , a cache management module 102 and an index adjustment module 103 .

[0054] The data pre-reading module 101 is used to predict data blocks that will be accessed within a preset time period and load the data blocks into the cache in advance.

[0055] The cache management module 102 is used to monitor the operating status of the key-value storage system in real time and dynamically adjust the cache space based on the operating status.

[0056] The index adjustment module 103 is used to dynamically adjust the index structure according to the access frequency of the key-value pairs.

[0057] It should be noted that the data pre-reading module 101 can predict the data blocks that will be accessed within a preset time period, for example, (1) customers concentrate on creating / deleting object data blocks corresponding to backup and snapshot policies within a preset time period. (2) The storage internal object data block index regularly detected within a preset time period. (3) The hot data index frequently read and written within a preset time period, etc. The data pre-reading module 101 reads such data for analysis, predicts the data blocks that will be accessed within the preset time period, and loads the data blocks into the cache in advance to perform data pre-reading.

[0058] The cache management module 102 monitors the operating status of the key-value storage system in real time and dynamically adjusts the cache space based on the operating status, thereby ensuring a high cache hit rate and reasonably controlling the memory usage of the storage node to avoid wasting resources.

[0059] The index adjustment module 103 can obtain the access frequency of each key-value pair and dynamically adjust the index structure according to the access frequency of the key-value pair to reduce the disk I / O operation during reading. For key-value pairs with high frequency of access, a finer-grained index is created to reduce the search range and increase the reading speed.

[0060] The present application provides a key-value storage system, including a data pre-reading module, a cache management module and an index adjustment module. The data pre-reading module is used to predict the data blocks that will be accessed within a preset time period, and load the data blocks into the cache in advance to reduce read latency and disk I / O operations, thereby improving read efficiency. The cache management module is used to monitor the operating status of the key-value storage system in real time, and dynamically adjust the cache space based on the operating status to ensure that the cache can both meet the high hit rate and reasonably control the use of memory, avoid resource waste, and further improve the reading performance and response speed. The index adjustment module is used to dynamically adjust the index structure according to the access frequency of the key-value pair, reduce the disk I / O operations during reading, increase the search speed and reduce the search cost. It can be seen that the key-value storage system of the present application can reduce the negative impact of read amplification, improve the system response speed, provide users with a faster and more efficient data access experience, and solve the problem in the prior art that the amount of data actually read when reading data is much larger than the amount of data requested.

[0061] Optionally, in another embodiment of the present application, the data pre-reading module 101 is as follows: Figure 2 As shown, it may include:

[0062] The data collection submodule 201 is used to collect log data of each storage node.

[0063] The feature engineering submodule 202 is used to construct a feature vector of each data block; the feature vector at least includes a recent access time feature vector, an access frequency feature vector and an access interval feature vector.

[0064] The heat score submodule 203 is used to calculate the heat score of each data block based on the feature vector.

[0065] The model training submodule 204 is used to construct an initial data pre-reading model and use cross-validation to train the model to obtain a data pre-reading model.

[0066] The pre-reading decision submodule 205 is used to generate a pre-reading data block list by using the data pre-reading model prediction, and perform data pre-reading according to the pre-reading data block list.

[0067] It should be noted that the data collection submodule 201 collects log data for a period of time at each storage node. The log data includes the timestamp of the read request, the key-value pair identifier, the placement group to which the request object data block belongs, the hard disk to which the access object data block belongs, the read result and the response time. The feature engineering submodule 202 constructs a feature vector of the data block for each data block; the feature vector at least includes the most recent access time feature vector, the access frequency feature vector and the access interval feature vector, and in some embodiments, it may also include feature vectors such as the time mode (such as peak hours, periodic access), the key value of the access object, etc. The heat scoring submodule 203 calculates the heat score of each data block based on the feature vector. The model training submodule 204 selects an appropriate machine learning model, such as random forest, gradient boosting tree (GBDT), and long short-term memory network (LSTM), according to the actual storage requirements of each production environment. After selecting the model, it constructs the data pre-reading initial model, and uses cross-validation training model to obtain the data pre-reading model. The pre-read decision submodule 205 generates a pre-read data block list using the data pre-read model prediction, which includes data blocks that may be accessed in various time periods, such as data blocks that may be accessed in the next 5 minutes / 10 minutes / 1 hour. These data blocks are then loaded into the storage node cache in advance based on the pre-read data block list.

[0068] Optionally, in another embodiment of the present application, the heat score submodule 203 calculates a heat score of each data block in one implementation, including:

[0069] Determine the weight factor for each eigenvector.

[0070] For each data block, based on each feature vector and the weight factor corresponding to each feature vector, the heat score of each data block is calculated using a preset heat score calculation formula.

[0071] It should be noted that the weight factor corresponding to each feature vector is first determined, and then the heat score of each data block is calculated using the following heat score calculation formula:

[0072] Hscore(t)=w1·Trecent(t)+w2·Ffreq(t)-w3·Iinterval(t)

[0073] Among them, Trecent is the most recent access time, Ffreq is the access frequency, Iinterval is the access interval, Hscore is the heat score, w1, w2, w3 are weight factors used to adjust the importance of different features, Trecent(t) is the normalized value of the most recent access time, Ffreq(t) is the normalized value of the access frequency, and Iinterval(t) is the normalized value of the access interval.

[0074] Optionally, in another embodiment of the present application, the data pre-reading module 101 may further include:

[0075] The preprocessing submodule is used to clean the log data.

[0076] It should be noted that after collecting the log data, the preprocessing submodule groups the data into time windows, each window contains a continuous sequence of read requests, and cleans the data to remove outliers and invalid requests.

[0077] Optionally, in another embodiment of the present application, the data pre-reading module 101 may further include:

[0078] The optimization submodule is used to collect access data of pre-read data blocks and optimize the data pre-read module.

[0079] It should be noted that the access data of the pre-read data block is collected to track the access status of the pre-read data block and determine whether the pre-read data block is accessed according to the prediction result, so as to optimize the subsequent data pre-reading module.

[0080] Optionally, in another embodiment of the present application, the cache management module 102 is as follows: Figure 3 As shown, it may include:

[0081] The monitoring submodule 301 is used to monitor the load data of the key-value storage system; wherein the load data includes memory usage, cache hit rate, CPU usage and disk I / O status.

[0082] The adjustment submodule 302 is used to adjust the cache space of the key-value storage system according to the load data.

[0083] The cleaning submodule 303 is used to trigger a cache cleaning process and release inactive data blocks when the cache usage reaches a preset upper limit.

[0084] It should be noted that the memory usage, cache hit rate, CPU usage and disk I / O status of the key-value storage system are monitored by the monitoring submodule 301 to understand the load of the key-value storage system. Then the adjustment submodule 302 adjusts the cache space of the key-value storage system according to the load data to ensure that the cache can meet the performance requirements without excessively occupying memory resources. Then the cleaning submodule 303 implements an adaptive LRU replacement strategy, combines the access frequency and timestamp information of the data blocks in the cache, and intelligently manages the data in the cache. When the cache usage reaches the preset upper limit, the cache cleaning process is triggered to release inactive data blocks.

[0085] Optionally, in another embodiment of the present application, an implementation manner of the above adjustment submodule adjusting the cache space of the key-value storage system according to the load data includes:

[0086] Get the current system load value, the maximum expected system load value, the minimum cache value, and the maximum cache value.

[0087] Based on the maximum expected value of the system load, the system load data, the minimum cache value and the maximum cache value, the target cache value is calculated using a preset cache calculation formula.

[0088] Adjust the size of the cache threshold according to the target cache value.

[0089] It should be noted that the system load measurement value, the maximum expected system load value, the cache minimum value and the cache maximum value at the current moment are first obtained, wherein the system load measurement value is any item in the load data (for example, CPU usage, I / O request rate, etc.), the maximum expected system load value is the maximum value of the pre-set system load, and the cache minimum value and the cache maximum value are the lower limit and upper limit of the pre-set cache space. Then, the system load measurement value, the maximum expected system load value, the cache minimum value and the cache maximum value are substituted into the following target cache value calculation formula:

[0090]

[0091] Among them, Cmin is the minimum cache value, Cmax is the maximum cache value, Mcurrent(t) is the system load measurement value at the current moment, and Mmax is the maximum expected value of the system load. The target cache value is calculated, and then the size of the cache space is adjusted according to the target cache value.

[0092] Optionally, in another embodiment of the present application, the cache management module 102 may further include:

[0093] The evaluation submodule is used to evaluate the effectiveness of the cache strategy of the cache management module according to a preset time period.

[0094] It should be noted that the cache strategy of the cache management module is evaluated for effectiveness regularly according to a preset time period, and the cache strategy is continuously optimized to improve cache efficiency and system response speed.

[0095] Optionally, in another embodiment of the present application, the index adjustment module 103 is as follows: Figure 4 As shown, it may include:

[0096] The statistics submodule 401 is used to count the access frequency of each key-value pair.

[0097] The index structure adjustment submodule 402 is used to adjust the index structure based on the access frequency.

[0098] The maintenance submodule 403 is used to regularly update the index and optimize the index storage.

[0099] It should be noted that the statistical submodule 401 counts the access frequency of each key-value pair. The higher the access frequency, the more popular the data is, so as to identify which are hot data. Then the index structure adjustment submodule 402 adjusts the index structure based on the access frequency to reduce the search path length. For key-value pairs whose access frequency reaches a certain threshold, a smaller and finer-grained secondary index is created. The maintenance submodule 403 regularly updates the index to reflect the latest access mode, and optimizes the index storage to reduce the storage overhead of the index itself.

[0100] Optionally, in another embodiment of the present application, an implementation manner in which the index structure adjustment submodule 402 adjusts the index structure based on the access frequency may include:

[0101] The access frequency of each key-value pair is compared with a preset threshold value; the preset threshold value is calculated based on the average access frequency of all key-value pairs and the standard deviation of the access frequency of all key-value pairs.

[0102] Filter out the key-value pairs whose access frequency is greater than the threshold, and create an additional index layer for each filtered key-value pair.

[0103] It should be noted that, assuming there are multiple levels of indexes, it is possible to decide whether to create an additional index layer for a key-value pair based on the access frequency F. For example, if the access frequency of a key-value pair exceeds a preset threshold, an additional index layer is created for it, such as creating a smaller and finer-grained secondary index. Therefore, the access frequency of each key-value pair is compared with the preset threshold; the preset threshold is calculated based on the average access frequency of all key-value pairs and the standard deviation of the access frequency of all key-value pairs, and the formula is as follows:

[0104] Fthreshold(t)=α·Favg(t)+β·Fstd(t)

[0105] Among them, Favg(t) is the average access frequency of all key-value pairs; Fstd(t) is the standard deviation of the access frequency of all key-value pairs; α and β are adjustment factors used to adjust the sensitivity of the threshold.

[0106] Then, the key-value pairs with access frequency greater than the threshold are filtered out, and for each filtered key-value pair, an additional index layer is created to optimize the search path.

[0107] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0108] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0109] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A key-value storage system, characterized in that: include: Data pre-reading module, cache management module and index adjustment module; The data pre-reading module is used to predict the data blocks that will be accessed within a preset time period and load the data blocks into the cache in advance; The cache management module is used to monitor the operating status of the key-value storage system in real time and dynamically adjust the cache space based on the operating status; The index adjustment module is used to dynamically adjust the index structure according to the access frequency of the key-value pair.

2. The system according to claim 1, characterized in that The data pre-reading module comprises: The data collection submodule is used to collect log data of each storage node; A feature engineering submodule is used to construct a feature vector of each data block; the feature vector includes at least a recent access time feature vector, an access frequency feature vector and an access interval feature vector; A heat score submodule, used to calculate a heat score of each data block based on the feature vector; The model training submodule is used to build the data pre-reading initial model and use cross-validation to train the model to obtain the data pre-reading model; The pre-reading decision submodule is used to generate a pre-reading data block list by using the data pre-reading model to predict and perform data pre-reading according to the pre-reading data block list.

3. The system according to claim 2, characterized in that The heat score submodule calculates the heat score of each data block to: Determining a weight factor for each of the eigenvectors; For each data block, based on each of the feature vectors and the weight factors corresponding to each of the feature vectors, a heat score of each data block is calculated using a preset heat score calculation formula.

4. The system according to claim 1, characterized in that The cache management module includes: A monitoring submodule, used to monitor the load data of the key-value storage system; wherein the load data includes memory usage, cache hit rate, CPU usage and disk I / O status; An adjustment submodule, used for adjusting the cache space of the key-value storage system according to the load data; The cleaning submodule is used to trigger the cache cleaning process and release inactive data blocks when the cache usage reaches a preset upper limit.

5. The system according to claim 4, characterized in that When the adjusting submodule adjusts the cache space of the key-value storage system according to the load data, it is used to: Get the current system load value, the maximum expected system load value, the minimum cache value, and the maximum cache value; Based on the maximum expected value of the system load, the system load data, the minimum cache value and the maximum cache value, a target cache value is calculated by a preset cache calculation formula; The size of the cache space is adjusted according to the target cache value.

6. The system according to claim 1, characterized in that The index adjustment module includes: The statistics submodule is used to count the access frequency of each key-value pair; An index structure adjustment submodule, used to adjust the index structure based on the access frequency; Maintenance submodule, used to regularly update the index and optimize index storage.

7. The system according to claim 6, characterized in that When the index structure adjustment submodule adjusts the index structure based on the access frequency, it is used to: Compare the access frequency of each key-value pair with the preset threshold; The preset threshold is calculated based on the average access frequency of all key-value pairs and the standard deviation of the access frequency of all key-value pairs; The key-value pairs whose access frequency is greater than the threshold are screened out, and an additional index layer is created for each of the screened key-value pairs.

8. The system according to claim 2, characterized in that The data pre-reading module further includes: The preprocessing submodule is used to clean the log data.

9. The system according to claim 2, characterized in that The data pre-reading module further includes: The optimization submodule is used to collect access data of the pre-read data blocks and optimize the data pre-reading module.

10. The system according to claim 2, characterized in that The cache management module further includes: The evaluation submodule is used to evaluate the effectiveness of the cache strategy of the cache management module according to a preset time period.

Images