A method and system for merging time-series data files optimized for solid-state drives
By controlling the amount of data to be merged by recording the data size in the LSM-tree, and by using a parallel approach to merge files, the write amplification and performance fluctuation issues of SSDs in time-series data storage systems are resolved, extending the lifespan of SSDs and reducing tail latency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF SOFTWARE - CHINESE ACAD OF SCI
- Filing Date
- 2026-02-03
- Publication Date
- 2026-06-02
AI Technical Summary
Existing time-series data storage systems are optimized for mechanical hard drives based on the LSM-tree model, which leads to problems such as severe write amplification, high storage costs, system performance fluctuations, and severe tail latency when solid-state drives (SSDs) are written.
By parsing time-series data into time-series keys and values, controlling the amount of data to be merged using the size of the data recorded in the log structure merge tree (LSM-tree), and performing file merging in parallel, the amount of data to be merged is reduced by duplicate writing and resource consumption.
It effectively reduces write amplification, extends the lifespan of the SSD, reduces system performance fluctuations and tail latency, and improves system stability and efficiency.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of information technology, specifically relating to a method and system for merging time-series data files optimized for solid-state drives. Background Technology
[0002] With the rapid development of the Internet of Things, time-series data has begun to grow explosively, which has brought challenges to time-series data storage systems. As a result, some time-series databases (TSDBs) have emerged that are specifically designed for time-series data storage. Hard disk drives (HDDs) typically dominate time-series data storage. HDDs consist of read / write heads and platters. During operation, the platters rotate at thousands of revolutions per minute, while the heads move along the radius of the platter to write and read data at designated locations. Therefore, most current time-series databases are optimized for these physical characteristics of HDDs, achieving higher performance through sequential writes and reads. However, with the decreasing price of solid-state drives (SSDs) based on flash memory technology, data centers are increasingly adopting them. SSDs are entirely composed of semiconductor chips with no moving parts. During data writing and reading, there is no movement of mechanical parts, making them insensitive to data access patterns and capable of parallel data processing. Furthermore, SSDs have a limited lifespan. Therefore, most optimization methods designed for HDDs are not applicable to SSDs.
[0003] Time-series data is a set of data indexed according to the time dimension, such as monitoring data of server memory metrics. Log Structurally Merged Tree (LSM-tree) structures are widely used in time-series data storage systems, such as... Figure 1 As shown, an LSM-tree consists of three parts: a Write-ahead Log (WAL), a Cache, and a Sorted Strings Table (SSTable) file. In the LSM-tree model, data is first written... Figure 1 The `Writer Data` function in the cache writes the received data to the cache, then writes the data to the Write-Ahead Log (WAL). Once the write to the WAL is successful, the write request is complete. When the cached data reaches a certain size threshold, the data is refreshed. Figure 1The data is flushed to the SSTables at Level 1 on disk, then the cache is cleared, awaiting the next batch of data to be written. When the file size in Level 1 reaches a threshold, the SSTables files in Level 1 are merged (compacted). Files with overlapping keys are read and merged into new SSTable files without duplicate keys, which are then placed in the next level, Level 2. At the same time, the files in Level 1 that participated in the merge are deleted. The same operation is performed in Level 1, Level 2, ..., Level 1Max.
[0004] WiscKey is a key-value store architecture optimized for SSDs. It leverages the rich parallelism inherent in SSDs, storing values in VLog files on the SSD and storing the addresses of keys and values within these files in an LSM-tree. During data retrieval, the system first retrieves the address of the corresponding value in the SSD from the LSM-tree based on the SeriesKey. Then, it queries the VLog file on the SSD in a highly parallel manner, fully utilizing the SSD's random data read capabilities. WiscKey does not perform file merging; instead, it uses a simple garbage collection method to reclaim expired values in the files, ensuring that valid values are stored contiguously in the VLog file. One end of the VLog file becomes the head, where new values are appended; the other end is the tail, which is the starting point for garbage collection. During garbage collection, a value is read from the tail, and then the LSM-tree is checked for a corresponding key. If the key exists, the value is considered valid and written to the head; otherwise, the space occupied by the value just read from the tail is released.
[0005] Tail latency, also known as "high percentage latency," is the latency of a relatively small but long-responding portion of requests. It is a high latency that is rarely seen by the client and is usually caused by system performance fluctuations.
[0006] Existing time-series data storage systems (such as InfluxDB, IotDB, and other time-series databases) are based on LSM-trees. LSM-trees are data structures optimized for mechanical hard drives, and this optimization is ineffective or even harmful to SSDs. 1) Severe Write Amplification and High Storage Costs: "Write amplification" refers to a situation where the total I / O volume performed by the system exceeds the actual amount of data written by the user. Existing time-series data storage systems using the LSM-tree write model store each time series data point as a regular key-value pair. Particularly when writing to SSTables, the characteristics of time-series data are not considered. When Level_1 triggers file compaction, data in Level_1 is read, sorted, and written to Level_2. Similarly, data in Level_2 is also read, sorted, and written to Level_3. The same data is repeatedly read, sorted, and written until the highest level, causing severe write amplification. Each SSD storage unit has a limited erase lifespan and a finite capacity lifespan; write amplification reduces the lifespan of the SSD, increasing storage costs.
[0007] 2) Traditional file merging is based on multiple files, merging multiple files into a new file. It is impossible to control the amount of data merged each time, which will consume a lot of resources (such as CPU and disk I / O), resulting in reduced system write performance and fluctuations. Summary of the Invention
[0008] This invention analyzes the characteristics of time-series data and, based on the special read / write characteristics of SSDs, utilizes the internal parallel data processing capabilities of SSDs to solve the problems of severe write amplification and reduced SSD lifespan when merging files in time-series data storage systems based on the traditional LSM-tree model; as well as the problems of uncontrollable CPU and disk I / O resource consumption and severe tail latency caused by system performance fluctuations due to the inability to control the size of each merge.
[0009] The technical solution adopted in this invention is as follows: A method for merging time-series data files optimized for solid-state drives includes the following steps: The time series data is parsed into time series keys and values. The values are written to the time series data log file (TSLog file) of the solid-state drive (SSD). The address index of the value in the time series data log file and the time series key are written to the log structure merge tree (LSM-tree) in the form of key-value data. The time-series data log files are merged. During the merging process, the amount of data involved in the merging is controlled by the amount of data recorded in each key-value data in the time-series data log file in the log structure merging tree.
[0010] Furthermore, the time series data is parsed into time series keys and values, wherein the time series key consists of a measurement indicator, a tag set, and a measurement field, and the value consists of a measurement value and a timestamp, and the time series key and value are written into a cache.
[0011] Furthermore, the step of writing the values to the time-series data log file of the solid-state drive includes: when the data in the cache reaches the maximum cache threshold or exceeds the maximum cache time, sorting each value according to the timestamp, compressing all the values using a compression algorithm, and writing the compressed values to the time-series data log file of the solid-state drive.
[0012] Furthermore, the step of writing the address index and time series key in the time series data log file into the log structure merge tree in the form of key-value data includes: using the minimum timestamp, maximum timestamp, file number, offset, data block size, and aggregation calculation information as values, combining them with the time series key to form key-value data, and writing the key-value data into the log structure merge tree.
[0013] Furthermore, the merging of time-series data log files includes: Query the log structure merge tree to find the time series keys that exist in all time series data log files from Level 1 to Level 2 (N), as well as the amount of data for each time series key in Level 1 to Level 2 (N), and then sort the time series keys based on the amount of data. The selection criteria for time series keys to be merged are: the total amount of data for all time series keys to be merged in the time series data log file does not exceed the set threshold. The time series data log files corresponding to the time series keys involved in the merging are counted. Each time series data log file corresponds to a file number. Then, they are grouped according to the file number. All the data corresponding to the time series keys in each group are in the same time series data log file. Create a new thread for each time series data log file to read the data in parallel, then sort the data and write it to a new time series data log file in the N+1 level. Record the size of the data involved in the merging of each time series data log file, as well as the size of the remaining unmerged data. If the time series data log file has been completely merged, delete it; otherwise, mark it as partially merged and do not delete it. After the files are successfully merged, the merged indexes are removed from the log structure merge tree, and the indexes of the time series keys in the new time series data log files are written into the log structure merge tree.
[0014] Furthermore, when merging time-series log files, the time series keys from the first to the Nth level are directly obtained from the log structure merging tree when selecting data to participate in the Nth level file merging, and the data is written from the first to the Nth level to the N+1th level through a single file merging.
[0015] Furthermore, the merging of time-series data log files controls the amount of data involved in the merging by controlling the number of time-series keys involved in the merging, thereby controlling the CPU and disk I / O resources used.
[0016] A time-series data file merging system optimized for solid-state drives, comprising: The data writing module is used to parse time-series data into time-series keys and values, write the values to the time-series data log file of the solid-state drive, and write the address index of the value in the time-series data log file and the time-series key into the log structure merge tree in the form of key-value data. The file merging module is used to merge time-series data log files. During the merging process, the amount of data involved in the merging is controlled by the amount of data recorded in each key-value data in the time-series data log file in the log structure merging tree.
[0017] The present invention also provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.
[0018] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the above-described method.
[0019] The beneficial effects of this invention are as follows: Existing time-series data storage systems based on LSM-trees have two unavoidable drawbacks during file merging: 1) Data in the cache, after being written to disk, is at Level_1. It is then read, sorted, and written to Level_2 through file merging at Level_1, and then repeatedly read, sorted, and written through file merging at Level_2, ..., Level_Max-1, finally being written to Level_Max. The same data is repeatedly written Max times starting from Level_1, leading to severe write amplification and affecting the lifespan of the SSD. 2) Traditional file merging algorithms can only read, sort, and write all selected files to the next level in each merge. If these files are large, they will consume a lot of CPU and disk I / O resources, significantly impacting the system's foreground writes and queries, causing system performance fluctuations and severe tail latency.
[0020] The file merging algorithm proposed in this invention directly obtains the SeriesKeys from Level_1 to Level_N from the LSM-tree when selecting data to participate in Level_N. It can write data from Level_1, Level_2, ..., Level_N to Level_N+1 in one file merging, avoiding the repeated writing of traditional file merging algorithms and reducing write amplification.
[0021] The file merging algorithm proposed in this invention first obtains the SeriesKeys from Level 1 to Level 9 from the LSM-tree when selecting data to participate in Level 9. This allows us to determine the data size of each SeriesKey in the TSLog file. Therefore, by controlling the number of SeriesKeys to participate in the merging, we can control the amount of data to be merged, thereby controlling the CPU and disk I / O resources used, reducing the impact on the front-end system's write and query operations, reducing system performance fluctuations, and reducing tail latency. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of a log structure merge tree (LSM-tree).
[0023] Figure 2 This is a flowchart of the steps involved in merging time-series data files optimized for SSDs.
[0024] Figure 3 This is a schematic diagram of the timing data writing process.
[0025] Figure 4 This is a schematic diagram of a precise file merging process with controllable flow.
[0026] Figure 5 This is the flowchart of the Level_N merging algorithm.
[0027] Figure 6 This is a module composition diagram of a time-series data file merging system optimized for SSDs. Detailed Implementation
[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0029] HDDs use high-speed rotating platters, allowing the read / write head to move along the platter's radius to perform data writing and reading operations at designated locations. This converts random writes and reads into batch sequential writes and reads, reducing HDD jitter and accelerating read / write speeds. SSDs, on the other hand, are composed of semiconductor chips and lack a moving robotic arm. Internally, SSDs are divided into multiple channels, each capable of independent and simultaneous operation. Some SSDs incorporate multiple error-correcting code engines and flash controllers for each channel to achieve high-performance read / write. SSDs are insensitive to data access patterns and feature random read and write capabilities.
[0030] Based on the above analysis of HDDs and SSDs, it is clear that batch sequential write and batch sequential read modes are HDD-friendly, reducing robotic arm jitter and accelerating read / write efficiency. Current time-series data storage systems based on traditional LSM-trees suffer from write amplification issues due to file merging, severely impacting SSD lifespan and causing system performance fluctuations and significant tail latency. This invention utilizes the internal parallel performance of SSDs, combined with a key-value separation storage architecture, to design a traffic-controllable and precise file merging method.
[0031] Figure 2 This is a flowchart of the steps of the present invention, including: Step 1. Data writing process: Parse the time series data into time series keys and values, write the values to the time series data log file TSLog on the solid-state drive, and write the address index of the value in the time series data log file and the time series key into the log structure merge tree LSM-tree in the form of key-value data; Step 2. Merging Time-Series Data Log Files: Merge the time-series data log files. During the merging process, the amount of data involved in the merging is controlled by the size of each key-value pair recorded in the log structure's merge tree within the time-series data log file. In one embodiment, the data writing process of step 1 of the present invention is as follows: Figure 3 As shown, it includes the following steps: 1.1) Parse the received time-series data into a key-value model and write it to the cache. The key is the time-series key (SeriesKey), which consists of <measurement metrics, tag sets, and measurement fields>, and the value is the value, which consists of <measurement value and timestamp>. SeriesKey=measurement, tag_sets, field_key value=[<value1,timestamp1> ,<value2,timestamp2> ,…] 1.2) When the data in the cache reaches the threshold `max_cache_threshold` (maximum cache threshold) or exceeds `max_cache_cold_duration` (maximum cache duration), each value is first sorted according to its timestamp. All values are then compressed using a compression algorithm (an existing compression algorithm can be used). Finally, the compressed timestamps and values are written to the SSD's Time-Series Log (TSLog) file. The TSLog file written from the cache to the SSD is at Level 1, and the offset is also recorded in the TSLog file.
[0032] The format of the encoded values in TSLog is: [size,crc32,type,count,encoded_values], where size is the size of the data block, crc32 is the checksum encoding, type is the type of the value, count is the number of values, and encoded_values is all the values after compression encoding.
[0033] 1.3) Record the minimum timestamp min_ts and the maximum timestamp max_ts. Construct key-value data using the minimum timestamp, maximum timestamp, file number file_id, offset, data block size size, and aggregation information. The key is SeriesKey and the value is (min_ts, max_ts, file_id, offset, size, aggregation).
[0034] In this context, aggregation refers to the maximum, minimum, average, and other values of multiple data points contained in a compressed data block. For example, if a compressed data block contains 1000 data points, the aggregation calculations are performed on these 1000 data points to obtain the maximum, minimum, average, and other results. This allows users to directly retrieve the results when querying, without needing to load the compressed data block into memory, decompress it, and then perform the calculations, thus improving performance.
[0035] 1.4) Write the key-value data obtained in step 1.3) into the LSM-tree.
[0036] 1.5) As the number of TSLog files in the SSD increases, the merging of TSLog files will be triggered.
[0037] Step 2 of this invention employs a time-series data log file (TSLog) merging algorithm to merge TSLog files. During the merging process, the amount of data involved in the merging is controlled by the amount of data recorded in each key-value pair in the LSM-tree within the TSLog file.
[0038] As time-series data is continuously written, the number of TSLog files increases rapidly. The data (value) for each SeriesKey may be distributed across multiple TSLog files. File merging operations are needed to combine the SeriesKey values from multiple TSLog files and write them into a single TSLog file, making the data more ordered and accelerating queries. This invention proposes a traffic-controllable, precise file merging algorithm that can reduce write amplification caused by file merging and improve SSD lifespan.
[0039] Taking the TSLog file system, which has four levels, as an example... Figure 4 As shown in Level_1 to Level_4, the TSLog files written by the cache are in Level_1. Every certain period of time (e.g., every second), the number of files in Level_1 to Level_4 is checked. If the number of files in a certain level exceeds the set threshold, the TSLog files in that level are merged. The merged new file is written to the next level, and the TSLog files that participated in the merge are deleted.
[0040] The specific merging algorithm for Level_N is explained below. The algorithm's flow is as follows: Figure 5 As shown, the specific steps include: 2.1) First, query the LSM-tree to find all SeriesKeys present in all TSLog files from Level_1 to Level_N, and the data size of each SeriesKey in Level_1 to Level_N. Then, sort the SeriesKeys based on their data size. This method finds the SeriesKey with the largest data size distributed in the TSLog files from Level_1 to Level_N, and prioritizes merging the SeriesKeys with the largest data size.
[0041] The "data volume size" here differs from the "data block size" mentioned earlier. A Level contains multiple TSLog files, and each TSLog file stores data for multiple SeriesKeys. "Data volume size" refers to the total size of the SeriesKey data across all TSLog files from Level_1 to Level_N when a merge is triggered at Level_N. When SeriesKey data is stored in a single TSLog file, it is stored in data blocks, and "data block size" refers to the size of this data block.
[0042] 2.2) Select the SeriesKeys to be merged. The selection is based on the fact that the total amount of data of all SeriesKeys to be merged in TSLog does not exceed the set threshold, which controls the flow of each file merge.
[0043] 2.3) Count the TSLog files corresponding to the SeriesKeys participating in the merging. Each TSLog file corresponds to a file_id. Then, group them according to the file_id. All data corresponding to the SeriesKeys in each group are in the same TSLog file (with the same file_id).
[0044] 2.4) Create a new thread for each TSLog file to read the data in parallel, sort the data, and write it to a new TSLog file in Level_N+1.
[0045] 2.5) Record the size of the data involved in the merging of each TSLog file, as well as the size of the remaining unmerged data. If the TSLog file has been completely merged, delete it; otherwise, mark it as partially merged and do not delete it.
[0046] 2.6) After the files are successfully merged, first delete the index of the merged SeriesKey from the LSM-tree, and then write the index of SeriesKey in the new TSLog file (i.e., SeriesKey and (min_ts, max_ts, file_id, offset, size, aggregation)) into the LSM-tree.
[0047] The key point of this invention is: When file merging is triggered at Level_N, this invention controls the amount of data corresponding to the SeriesKey participating in the merging by setting a threshold. By controlling the amount of data in each file merging, the impact of file merging on system performance is reduced, and tail latency is decreased.
[0048] When selecting the SeriesKey to participate in the Level_N file merging, this invention starts from Level_1 and continues up to Level_N. The data in Level_1, Level_2, ..., Level_N-1 is then directly written to Level_N+1 through this file merging, skipping intermediate levels. This avoids the problem of traditional file merging only writing the data in the current Level to the next Level, which significantly reduces duplicate data writing, write amplification, and improves the lifespan of the SSD.
[0049] Another embodiment of the present invention provides a timing data file merging system optimized for SSDs, such as... Figure 6 As shown, it includes: The data writing module is used to parse time-series data into keys and values, write the values to the SSD's TSLog file, and write the address index of the value in the TSLog and the key to the LSM-tree in the form of key-value data. The file merging module is used to merge TSLog files. During the merging process, the amount of data involved in the merging is controlled by the amount of data in the TSLog file for each key-value pair recorded in the LSM-tree.
[0050] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding processes in the aforementioned method embodiments.
[0051] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.
[0052] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.
[0053] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.
[0054] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.
Claims
1. A method for merging time-series data files optimized for solid-state drives, characterized in that, Includes the following steps: The time-series data is parsed into time-series keys and values. The values are written to the time-series data log file of the solid-state drive. The address index of the value in the time-series data log file and the time-series key are written to the log structure merge tree in the form of key-value data. The time-series data log files are merged. During the merging process, the amount of data involved in the merging is controlled by the amount of data recorded in each key-value data in the time-series data log file in the log structure merging tree.
2. The method according to claim 1, characterized in that, The process involves parsing time-series data into time-series keys and values. The time-series keys consist of measurement indicators, a set of labels, and measurement fields, while the values consist of measurement values and timestamps. The time-series keys and values are then written into a cache.
3. The method according to claim 2, characterized in that, The step of writing values to the time-series data log file of the solid-state drive includes: when the data in the cache reaches the maximum cache threshold or exceeds the maximum cache time, sorting each value according to the timestamp, compressing all values using a compression algorithm, and writing the compressed values to the time-series data log file of the solid-state drive.
4. The method according to claim 3, characterized in that, The step of writing the address index and time series key in the time series data log file into the log structure merge tree in the form of key-value data includes: using the minimum timestamp, maximum timestamp, file number, offset, data block size, and aggregation calculation information as values, combining them with the time series key to form key-value data, and writing the key-value data into the log structure merge tree.
5. The method according to claim 4, characterized in that, The merging of time-series data log files includes: Query the time series keys that exist in all time series data log files from the first level to the Nth level from the log structure merge tree, and the amount of data for each time series key in the first level to the Nth level. Then sort the time series keys based on the amount of data. The selection criteria for time series keys to be merged are: the total amount of data for all time series keys to be merged in the time series data log file does not exceed the set threshold. The time series data log files corresponding to the time series keys involved in the merging are counted. Each time series data log file corresponds to a file number. Then, they are grouped according to the file number. All the data corresponding to the time series keys in each group are in the same time series data log file. Create a new thread for each time series data log file to read the data in parallel, then sort the data and write it to a new time series data log file in the N+1 level; Record the size of the data involved in the merging of each time series data log file, as well as the size of the remaining unmerged data. If the time series data log file has been completely merged, delete it; otherwise, mark it as partially merged and do not delete it. After the files are successfully merged, the merged indexes are removed from the log structure merge tree, and the indexes of the time series keys in the new time series data log files are written into the log structure merge tree.
6. The method according to claim 5, characterized in that, The process of merging time-series log files involves directly obtaining the time series keys from the first to the Nth level from the log structure merging tree when selecting data to participate in the Nth level file merging, and writing the data from the first to the Nth level to the N+1th level through a single file merging.
7. The method according to claim 5, characterized in that, The process of merging time-series data log files controls the amount of data involved in the merging process by controlling the number of time-series keys involved, thereby controlling the CPU and disk I / O resources used.
8. A time-series data file merging system optimized for solid-state drives, characterized in that, include: The data writing module is used to parse time-series data into time-series keys and values, write the values to the time-series data log file of the solid-state drive, and write the address index of the value in the time-series data log file and the time-series key into the log structure merge tree in the form of key-value data. The file merging module is used to merge time-series data log files. During the merging process, the amount of data involved in the merging is controlled by the amount of data recorded in each key-value data in the time-series data log file in the log structure merging tree.
9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.