Time series data storage method and system based on LSM tree key-value separation

Abstract

The present invention discloses a time series data storage method and system based on LSM tree key-value separation, and relates to the technical field of data storage. The method utilizes a key-value separate storage method to store keys in an ordered index based on an LSM tree, and to store values ​​in a key-value log file based on sequential addition, thereby realizing a function of fast range query, solving the problems of high delay in read and write operations of the existing key-value storage system, and read and write and space amplification generated when the data volume is large. At the same time, a detection-based compression algorithm is used to efficiently clean up expired indexes in the LSM tree, thereby avoiding various performance problems caused by index accumulation due to low compression efficiency of the LSM tree in the existing system. Furthermore, according to a constructed full prefix deletion statistics table, the prefix of the index key in the table and the deletion time defined in the metadata can be used to reduce random queries on the ordered index when cleaning up garbage data in the key-value log file, thereby solving the problems of low efficiency in cleaning up garbage data in the existing storage system and not consuming more resources.

Description

Technical Field

[0001] The invention discloses a method and a system, and relates to the technical field of data storage, and in particular to a method and a system for storing time series data based on LSM tree key-value separation. Background Art

[0002] In the field of IoT, time series data has the characteristics of high frequency of collection, large data volume, and structure. The time series data collected in the IoT system usually has multiple attributes, and the time series data storage system needs to be able to quickly complete data search according to various query requirements of users.

[0003] Currently, LSM trees can be used as the basic data storage structure. Since indexes and data are stored separately, the cleanup of expired indexes can be completed in the compression of LSM trees. However, there are defects in the compression algorithm of LSM trees, which makes LSM trees prone to excessive expansion. Although the compression algorithm of LSM trees has been improved in subsequent versions, it is still easy to accumulate a large number of expired indexes, resulting in a significant decrease in query performance. At the same time, the cleanup of expired data in distributed databases requires the help of indexes, which further leads to low cleaning efficiency. In addition, the system generates a lot of garbage data when it runs for a long time, which takes up a lot of disk space. Summary of the invention

[0004] In view of the problems in the prior art, the present invention provides a method and system for storing time series data based on LSM tree key-value separation. The specific solution proposed by the present invention is:

[0005] A time series data storage method based on LSM tree key-value separation, based on the LSM tree key-value separation method, encapsulates the key-value data written into the distributed database into key-value entries according to the write request, wherein the key-value entries include index keys and metadata, and the metadata include timestamps and tags.

[0006] Determine whether the key-value entry is a deletion mark of the index key according to the label in the metadata. If so, form a pair of key values ​​with the prefix of the index key and the deletion time defined in the metadata, put them into the full prefix deletion statistics table, write the key-value entry into the key-value storage log, and construct an index entry using the index key, the metadata and the address of the key-value entry in the key-value storage log, write the index entry into the sorted string table of the skip table, merge the skip tables of adjacent layers according to the overlapping data of the sorted string tables, determine whether the index entry stored in the sorted string table of the skip table with a smaller level in the adjacent layer skip table has a deletion mark, and determine whether the skip table with a larger level is the deepest level that is not empty. If both are yes, discard the index entry;

[0007] According to the index key and timestamp in the read request, query the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp has been deleted. If it has not been deleted, search for the index entry. According to the found index entry, determine whether to return the key value entry according to the read request.

[0008] According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and according to the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained, and the key-value storage log is cleaned up or rewritten into the distributed database according to the determination result.

[0009] Furthermore, in the time series data storage method based on LSM tree key value separation, it is determined whether the index entry stored in the sorting string table of the smaller level jump table in the adjacent layer jump table has a deletion mark, if not, the index entry is retained, and the index entry is written into the new sorting string table,

[0010] If so, it is determined whether the skip table with a larger level is the deepest level that is not empty. If not, the index entry is retained and the index entry is written into a new sorted string table.

[0011] Furthermore, in the time series data storage method based on LSM tree key-value separation, the key-value pair consisting of the index key prefix and the deletion time is put into the full prefix deletion statistics table, including:

[0012] Write the key-value pair consisting of the index key prefix and deletion time into the prefix deletion statistics table.

[0013] After all key-value entries are written to the key-value storage log according to the write request, the prefix deletion statistics table is merged with the full prefix deletion statistics table.

[0014] Furthermore, the querying of the full prefix deletion statistics table in the time series data storage method based on LSM tree key value separation to determine whether the index key corresponding to the timestamp is deleted includes:

[0015] According to the full prefix deletion statistics table, confirm that the index key corresponding to the timestamp has been deleted, and return null to the upper-level application.

[0016] According to the full prefix deletion statistics table, if the index key corresponding to the timestamp has not been deleted, the index entry is searched in the skip table storing the index entry.

[0017] Furthermore, in the time series data storage method based on LSM tree key-value separation, the step of determining whether the oldest key-value storage log needs to be retained includes:

[0018] According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log does not need to be retained, and then the key-value storage log is cleaned up.

[0019] According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log needs to be retained, and then the key-value storage log is rewritten into the distributed database.

[0020] The present invention also provides a time series data storage system based on LSM tree key-value separation, including a front-end statistics module, a key-value storage module and an ordered index module.

[0021] The key-value storage module is based on the LSM tree key-value separation method. According to the write request, the key-value data written by the distributed database is encapsulated into a key-value entry. The key-value entry contains an index key and metadata. The metadata contains a timestamp and a tag.

[0022] The pre-statistics module determines whether the key-value entry is a deletion mark of the index key according to the tag in the metadata. If so, the prefix of the index key and the deletion time defined in the metadata form a pair of key values ​​and put them into the full prefix deletion statistics table. The key-value storage module writes the key-value entry into the key-value storage log.

[0023] The ordered index module constructs an index entry using the index key, the metadata, and the address of the key-value entry in the key-value storage log, writes the index entry into the sorted string table of the jump table, merges the jump tables of adjacent layers according to the overlapping data of the sorted string tables, determines whether the index entry stored in the sorted string table of the jump table with a smaller level in the adjacent jump table has a deletion mark, and determines whether the jump table with a larger level is the deepest level that is not empty, and if both are true, discards the index entry;

[0024] The front-end statistics module queries the full prefix deletion statistics table based on the index key and timestamp in the read request to determine whether the index key corresponding to the timestamp has been deleted. If it has not been deleted, it searches for the index entry and determines whether to return the key value entry according to the read request based on the found index entry.

[0025] According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and based on the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained. Based on the judgment result, the key-value storage module cleans the key-value storage log or rewrites the key-value storage log into the distributed database.

[0026] Further, the ordered index module in the time series data storage system based on LSM tree key value separation determines whether the index entry stored in the sorted string table of the jump table with a smaller level in the adjacent layer jump table has a deletion mark, if not, retains the index entry, and writes the index entry into the new sorted string table.

[0027] If so, it is determined whether the skip table with a larger level is the deepest level that is not empty. If not, the index entry is retained and the index entry is written into a new sorted string table.

[0028] Furthermore, the pre-statistics module in the time series data storage system based on LSM tree key-value separation puts the key-value pair consisting of the index key prefix and the deletion time into the full prefix deletion statistics table, including:

[0029] Write the key-value pair consisting of the index key prefix and deletion time into the prefix deletion statistics table.

[0030] After all key-value entries are written to the key-value storage log according to the write request, the prefix deletion statistics table is merged with the full prefix deletion statistics table.

[0031] Furthermore, the front-end statistics module in the time series data storage system based on LSM tree key value separation queries the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp is deleted, including:

[0032] According to the full prefix deletion statistics table, confirm that the index key corresponding to the timestamp has been deleted, and return null to the upper-level application.

[0033] According to the full prefix deletion statistics table, if the index key corresponding to the timestamp has not been deleted, the index entry is searched in the skip table storing the index entry.

[0034] Further, the front-end statistical module in the time series data storage system based on LSM tree key-value separation determines whether the oldest key-value storage log needs to be retained, including:

[0035] According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log does not need to be retained, and then the key-value storage module cleans up the key-value storage log.

[0036] According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log needs to be retained, and then the key-value storage module rewrites the key-value storage log into the distributed database.

[0037] The benefits of the present invention are:

[0038] The present invention provides a time series data storage method based on LSM tree key value separation, and realizes storage optimization of distributed database system. By using the key value separate storage mode, the key is stored in an ordered index based on the LSM tree, and the value (data) is stored in an ordered index based on sequential addition to the key value log file to realize the function of fast range query, and the sequential addition realizes the transformation from random writing to sequential writing, solves the problems of high delay in reading and writing operations of the existing key value storage system, reading, writing and space amplification generated when the data volume is large, and at the same time, the detection-based compression algorithm can be used to efficiently and quickly clean up the expired index in the LSM tree, solve the various performance problems caused by the low compression efficiency of the LSM tree in the existing system and the accumulation of indexes, and according to the constructed full prefix deletion statistics table, the prefix of the index key in the table and the deletion time defined in the metadata can reduce the random query of the ordered index when cleaning up the garbage data in the key value log file, solve the problem of low efficiency of garbage data cleaning in the existing storage system and will not consume more resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of writing data using the present invention.

[0040] Figure 2 It is a schematic diagram of a method of storing data using the present invention.

[0041] Figure 3 It is a schematic diagram of the data reading process of the system of the present invention.

[0042] Figure 4 The present invention is a flow chart of merging index entry data.

[0043] Figure 5 It is a schematic diagram of the data merging form of the jump table in the present invention.

[0044] Figure 6 The present invention is used to perform data cleaning flow chart. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0046] The method of the present invention is based on a time series data storage method for LSM tree key-value separation. Based on the LSM tree key-value separation mode, the key-value data written into the distributed database is encapsulated into a key-value entry according to a write request. The key-value entry includes an index key and metadata. The metadata includes a timestamp and a tag.

[0047] Determine whether the key-value entry is a deletion mark of the index key according to the label in the metadata. If so, form a pair of key values ​​with the prefix of the index key and the deletion time defined in the metadata, put them into the full prefix deletion statistics table, write the key-value entry into the key-value storage log, and construct an index entry using the index key, the metadata and the address of the key-value entry in the key-value storage log, write the index entry into the sorted string table of the skip table, merge the skip tables of adjacent layers according to the overlapping data of the sorted string tables, determine whether the index entry stored in the sorted string table of the skip table with a smaller level in the adjacent layer skip table has a deletion mark, and determine whether the skip table with a larger level is the deepest level that is not empty. If both are yes, discard the index entry;

[0048] According to the index key and timestamp in the read request, query the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp has been deleted. If it has not been deleted, search for the index entry. According to the found index entry, determine whether to return the key value entry according to the read request.

[0049] According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and according to the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained, and the key-value storage log is cleaned up or rewritten into the distributed database according to the determination result.

[0050] The method of the present invention utilizes a key-value separate storage method to store the key in an ordered index based on an LSM tree, and stores the value in a key-value log file based on a sequential addition, thereby realizing a function of fast range query, solving the problems of high delay in read and write operations of the existing key-value storage system, and read and write and space amplification generated when the data volume is large, and at the same time, using a detection-based compression algorithm to efficiently clean up expired indexes in the LSM tree, thereby avoiding various performance problems caused by index accumulation due to low compression efficiency of the LSM tree in the existing system, and according to the constructed full-volume prefix deletion statistics table, using the prefix of the index key in the table and the deletion time defined in the metadata, it is possible to reduce random queries on the ordered index when cleaning up garbage data in the key-value log file, thereby solving the problems of low efficiency in cleaning up garbage data in the existing storage system without consuming more resources.

[0051] In some embodiments of the method of the present invention, when storing time series data based on LSM tree key value separation, refer to Figure 1In the writing process, the upper-layer application sends a write request to the key-value storage database system through the interface layer. The database system encapsulates the written key-value data into a key-value entry. The key-value entry contains the index key, index value and metadata. The metadata contains timestamps, tags and other user-defined attributes. The database system determines whether the entry is a deletion mark for the index key based on the tag in the metadata of the key-value entry. If it is a deletion mark, it is necessary to extract the prefix of the index key in the key-value entry and the user-defined deletion time point in the metadata to form a pair of key values ​​and write them to the full prefix deletion statistics table. When the write or update operation in the full prefix deletion statistics table reaches the set threshold, the entire full prefix deletion statistics table is persistently stored.

[0052] After the key-value entry is written to the key-value storage log, an index entry is constructed using the index key, metadata, and the address of the key-value entry in the key-value storage log, and then the index entry is written to the skip list.

[0053] Furthermore, the key-value pairs consisting of the prefix of the index key and the deletion time point defined by the user in the metadata can be written into the prefix deletion statistics table of the dictionary structure. When all the key-value entries contained in this write operation are successfully written into the key-value storage log in an appended manner, the corresponding prefix deletion statistics table is immediately sent to the asynchronous processing queue, and the thread takes out the prefix deletion statistics table and merges it with the full prefix deletion statistics table. When the write or update operation in the full prefix deletion statistics table reaches the set threshold, the entire full prefix deletion statistics table is persistently stored.

[0054] Furthermore, the index entries are written into the skip list, and when the skip list exceeds the set size, it can be converted into a read-only skip list; when the read-only skip lists in the memory exceed the set number, the data in the read-only skip list is written into the sorted string table of the 0-layer skip list in lexicographic order; when the sorted string tables of the 0-layer skip list exceed the set number, the multiple sorted string tables of the 0-layer are merged to generate a larger sorted string table, which is written to the 1-layer skip list and written to the disk. Similarly, when the sorted string table of the 1-layer skip list exceeds the set size, the multiple sorted string tables of the 1-layer skip list are merged to generate a larger sorted string table, which is written to the 2-layer and written to the disk, and the merging operation of higher layers is similar. Only the data between the sorted string tables of the 0-layer skip list will overlap, and the data between the sorted string tables of the same layer of the 1-layer or higher layers will not overlap.

[0055] The key value storage and index entry storage structures mentioned above are as follows: Figure 2 As shown in the figure, both the key-value log file and the index entry file will be recorded and synchronously written to the disk to avoid data loss due to failure or downtime.

[0056] Furthermore, when merging the data of index entries through the database system during key-value storage, the process is as follows: Figure 4 and Figure 5 , where the default number of sorting string tables in layer 0 is 5, and the size of layer 0 is not limited. The default size of layer 1 is 64MB, and each layer is expanded 10 times thereafter. The maximum capacity can be set to more than 6TB. Data merging is performed by layer. Before the merging begins, the ratio of the total actual size of the sorting string tables in each layer to the rated size of each layer is calculated. The layers with a ratio exceeding 1 participate in the merging. First, a sorting string table with the highest overlap with the tables in the current layer is selected from the next layer (such as layer 2) as the lower layer table, and then all sorting string tables overlapping with the lower layer table are selected from the current layer (such as layer 1) as the upper layer table, and then the upper layer table and the lower layer table are merged using merge sort. In this process, the latest version of the key value entry corresponding to each index key is retained. If an index entry in the upper layer table is found to have a deletion mark during the merging process, and there is no sorting string table on a higher layer of the layer where the lower layer table is located, the index entry is discarded. The above merging process can efficiently clean up the deleted index data.

[0057] When storing time series data based on LSM tree key-value separation, refer to Figure 3 In the reading process, the upper-layer application sends a read request to the database system of the key-value storage through the interface layer. The database system queries the global prefix deletion table based on the index key and timestamp in the read request. If it is determined that the index key corresponding to the timestamp has been deleted, it directly returns null to the upper-layer application; if it is determined that the index key corresponding to the timestamp has not been deleted, an index entry search is performed. The index entry is searched in the skip table in the memory. If it is found in the skip table, it is directly returned to the upper-layer application. If it is not found in the skip table, it continues to search in the sorted string table of layer 0, and so on until the index entry to be queried is found. After finding the index entry, determine whether the key value entry needs to be returned based on the detailed content of the read request. If necessary, read the key value entry from the corresponding key value log file according to the address in the index entry, and return the data after successful reading.

[0058] When storing time series data based on LSM tree key-value separation, refer to Figure 6In the data cleaning process, the upper-layer application sends the cleaning request to the key-value storage database system through the interface layer. The database system first copies the full prefix deletion statistics table, and then finds the key-value log file with the smallest (oldest) file number. All key-value entries in the file are traversed in turn. According to the prefix of the table recorded in the full prefix deletion statistics table and the corresponding deletion timestamp, it is determined whether the log entry needs to be retained. If not, it is skipped directly. If necessary, the writing process is re-executed. In addition, the key-value log file sequence number is globally incremented and will not be reused. After the key-value log file is cleaned, the operation process and result records can be persistently stored to avoid data loss.

[0059] The present invention also provides a time series data storage device based on LSM tree key-value separation, including a front-end statistics module, a key-value storage module and an ordered index module.

[0060] The key-value storage module is based on the LSM tree key-value separation method. According to the write request, the key-value data written by the distributed database is encapsulated into a key-value entry. The key-value entry contains an index key and metadata. The metadata contains a timestamp and a tag.

[0061] The pre-statistics module determines whether the key-value entry is a deletion mark of the index key according to the tag in the metadata. If so, the prefix of the index key and the deletion time defined in the metadata form a pair of key values ​​and put them into the full prefix deletion statistics table. The key-value storage module writes the key-value entry into the key-value storage log.

[0062] The ordered index module constructs an index entry using the index key, the metadata, and the address of the key-value entry in the key-value storage log, writes the index entry into the sorted string table of the jump table, merges the jump tables of adjacent layers according to the overlapping data of the sorted string tables, determines whether the index entry stored in the sorted string table of the jump table with a smaller level in the adjacent jump table has a deletion mark, and determines whether the jump table with a larger level is the deepest level that is not empty, and if both are true, discards the index entry;

[0063] The front-end statistics module queries the full prefix deletion statistics table based on the index key and timestamp in the read request to determine whether the index key corresponding to the timestamp has been deleted. If it has not been deleted, it searches for the index entry and determines whether to return the key value entry according to the read request based on the found index entry.

[0064] According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and based on the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained. Based on the judgment result, the key-value storage module cleans the key-value storage log or rewrites the key-value storage log.

[0065] The information interaction, execution process and other contents between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0066] Similarly, the system of the present invention realizes storage optimization of distributed database system. By using the method of storing keys and values ​​separately, the keys are stored in the ordered index based on LSM tree, and the values ​​(data) are stored in the ordered index based on sequential addition to the key value log file to realize the function of fast range query, and the sequential addition realizes the transformation from random writing to sequential writing, which solves the problems of high delay in reading and writing operations of the existing key value storage system, reading, writing and space amplification generated when the data volume is large, and the detection-based compression algorithm can be used to efficiently and quickly clean up the expired indexes in the LSM tree, solve the various performance problems caused by the low compression efficiency of the LSM tree in the existing system and the accumulation of indexes, and according to the constructed full prefix deletion statistics table, the prefix of the index key in the table and the deletion time defined in the metadata can reduce the random query of the ordered index when cleaning up the garbage data in the key value log file, solve the problem of low efficiency of garbage data cleaning in the existing storage system and will not consume more resources.

[0067] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0068] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A time series data storage method based on LSM tree key-value separation, characterized by Based on the LSM tree key-value separation method, the written key-value data is encapsulated into key-value entries through the distributed database according to the write request. The key-value entries include index keys and metadata. The metadata includes timestamps and tags. Determine whether the key-value entry has a deletion mark for the index key according to the tag in the metadata. If so, form a pair of key values ​​with the prefix of the index key and the deletion time defined in the metadata and put them into the full prefix deletion statistics table. Writing the key-value entry into a key-value storage log, constructing an index entry using the index key, the metadata, and the address of the key-value entry in the key-value storage log, writing the index entry into a sorted string table of a skip table, merging the skip tables of adjacent layers according to overlapping data in the sorted string tables, determining whether an index entry stored in a sorted string table of a skip table with a smaller level in an adjacent layer of skip tables has a deletion mark, and determining whether a skip table with a larger level is a non-empty deepest level, and if both are true, discarding the index entry; According to the index key and timestamp in the read request, query the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp in the read request has been deleted. If it has not been deleted, search for the index entry and determine whether to return the key value entry according to the read request based on the found index entry; According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and according to the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained, and the key-value storage log is cleaned up or rewritten into the distributed database according to the determination result.

2. The method for storing time series data based on LSM tree key-value separation according to claim 1 is characterized in that The step of determining whether an index entry stored in a sorted string table of a smaller-level skip table in an adjacent layer skip table has a deletion mark, if not, retaining the index entry, and writing the index entry into a new sorted string table, If so, it is determined whether the skip table with a larger level is the deepest level that is not empty. If not, the index entry is retained and the index entry is written into a new sorted string table.

3. The method for storing time series data based on LSM tree key-value separation according to claim 1 is characterized in that The key-value pairs consisting of the index key prefix and deletion time are put into the full prefix deletion statistics table, including: Write the key-value pair consisting of the index key prefix and deletion time into the prefix deletion statistics table. After all key-value entries are written to the key-value storage log according to the write request, the prefix deletion statistics table is merged with the full prefix deletion statistics table.

4. The method for storing time series data based on LSM tree key-value separation according to claim 1 is characterized in that The querying of the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp is deleted includes: According to the full prefix deletion statistics table, confirm that the index key corresponding to the timestamp has been deleted, and return null to the upper-level application. According to the full prefix deletion statistics table, if the index key corresponding to the timestamp has not been deleted, the index entry is searched in the skip table storing the index entry.

5. The method for storing time series data based on LSM tree key-value separation according to claim 1 is characterized in that The determining whether the oldest key-value storage log needs to be retained includes: According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log does not need to be retained, and then the key-value storage log is cleaned up. According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log needs to be retained, and then the key-value storage log is rewritten into the distributed database.

6. A time series data storage system based on LSM tree key-value separation, characterized by It includes the front-end statistics module, key-value storage module and ordered index module. The key-value storage module is based on the LSM tree key-value separation method. According to the write request, the key-value data written by the distributed database is encapsulated into a key-value entry. The key-value entry contains an index key and metadata. The metadata contains a timestamp and a tag. The pre-statistics module determines whether the key-value entry is a deletion mark of the index key according to the tag in the metadata. If so, the prefix of the index key and the deletion time defined in the metadata form a pair of key values ​​and put them into the full prefix deletion statistics table. The key-value storage module writes the key-value entry into the key-value storage log. The ordered index module constructs an index entry using the index key, the metadata, and the address of the key-value entry in the key-value storage log, writes the index entry into the sorted string table of the jump table, merges the jump tables of adjacent layers according to the overlapping data of the sorted string tables, determines whether the index entry stored in the sorted string table of the jump table with a smaller level in the adjacent jump table has a deletion mark, and determines whether the jump table with a larger level is the deepest level that is not empty, and if both are true, discards the index entry; The front-end statistics module queries the full prefix deletion statistics table based on the index key and timestamp in the read request to determine whether the index key corresponding to the timestamp has been deleted. If it has not been deleted, it searches for the index entry and determines whether to return the key value entry according to the read request based on the found index entry. According to the cleanup request, all key-value entries of the oldest key-value storage log are traversed, and based on the prefix and deletion time of the index key recorded in the full prefix deletion statistics table, it is determined whether the oldest key-value storage log needs to be retained. Based on the judgment result, the key-value storage module cleans the key-value storage log or rewrites the key-value storage log into the distributed database.

7. The time series data storage system based on LSM tree key-value separation according to claim 6 is characterized by: The ordered index module determines whether an index entry stored in the sorted string table of the smaller-level skip table in the adjacent layer skip table has a deletion mark, if not, retains the index entry, and writes the index entry into a new sorted string table. If so, it is determined whether the skip table with a larger level is the deepest level that is not empty. If not, the index entry is retained and the index entry is written into a new sorted string table.

8. The time series data storage system based on LSM tree key-value separation according to claim 6 is characterized by: The pre-statistics module puts the key-value pair consisting of the index key prefix and deletion time into the full prefix deletion statistics table, including: Write the key-value pair consisting of the index key prefix and deletion time into the prefix deletion statistics table. After all key-value entries are written to the key-value storage log according to the write request, the prefix deletion statistics table is merged with the full prefix deletion statistics table.

9. The time series data storage system based on LSM tree key-value separation according to claim 6 is characterized by: The front-end statistics module queries the full prefix deletion statistics table to determine whether the index key corresponding to the timestamp is deleted, including: According to the full prefix deletion statistics table, confirm that the index key corresponding to the timestamp has been deleted, and return null to the upper-level application. According to the full prefix deletion statistics table, if the index key corresponding to the timestamp has not been deleted, the index entry is searched in the skip table storing the index entry.

10. The time series data storage system based on LSM tree key-value separation according to claim 6 is characterized in that The front-end statistics module determines whether the oldest key-value storage log needs to be retained, including: According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log does not need to be retained, and then the key-value storage module cleans up the key-value storage log. According to the prefix and deletion time of the index key, it is confirmed that the oldest key-value storage log needs to be retained, and then the key-value storage module rewrites the key-value storage log into the distributed database.

Images