A method for implementing an expiration schedule for a distributed relational database based on rocksdb

Abstract

The invention discloses a distributed relational database expiration schedule implementation method based on rocksdb, which belongs to the field of distributed relational database, database kernel and storage engine design, and specifically comprises: creating a TTL table, a server layer calculating the data range of the TTL table, and registering a TTL cleaning task with an engine layer; then the server layer starts a scheduled task, and pushes up cleanTs according to the TTL calculation, while the engine layer receives the TTL task, registers a CompactionFilter in rocksdb, and regularly pulls cleanTs of the current task data from a computing node; adopts a snapshot consistency reading method of cleanTs, so that RocksDB executes Compaction, and completes data cleaning for data within a range by calling CompactionFilter; then determines whether the server layer needs active Compaction based on statistical data on a regular basis, and if necessary, initiates CompactionRange and CompactionFiles instructions; after the engine layer receives the active Compaction instruction, RocksDB performs Compaction, and deletes data according to CompactionFilter to complete data cleaning.

Description

Technical Field

[0001] The invention belongs to the field of distributed relational database, database kernel and storage engine design, and specifically is a method for realizing an expiration schedule of a distributed relational database based on rocksdb. Background Art

[0002] RocksDB is a KV storage engine with an LSM-Tree architecture. It greatly improves write performance with the advantage of sequential writes. In recent years, as a persistent and high-performance storage engine, it has been widely used in the field of distributed relational databases. TiDB and CockroachDB both use RocksDB as their storage engine.

[0003] Time to live (TTL) is a table with a life cycle of data. It can set an expiration time for the data. After the expiration time, the data will be automatically deleted. TiDB's TTL is a row-granularity table. Through the background thread, the TTL table data is scanned regularly, and the data rows that have exceeded the expiration time are deleted. PolarDB-X uses the solution of treating TTL tables as partition tables, performing range partitioning according to time periods, and deleting partitions after data expires. It is a partition table solution with time partition granularity.

[0004] For example, a Chinese patent with authorization announcement number CN111400331B discloses a processing method and device based on the TiDB database, which is characterized by receiving a data operation request, the data operation request is used to indicate a first operation on first data; performing the first operation on the first data according to the data operation request; determining whether the first operation fails; if so, retrying the first operation at the application level. The processing method and device based on the TiDB database can retry the first operation at the application level after the data operation fails, thereby increasing the probability of successful execution of the first operation and reducing the possibility of failure to update hot data using the TiDB database under high concurrency conditions; and because the retry of the first operation is performed at the application level, the entire process is imperceptible to the user, thereby achieving an increase in the success rate of data operations without the user's perception, and optimizing the user experience.

[0005] For example, the Chinese patent application with the publication number CN116107806A discloses a database backup management method, system, device and storage medium. The present invention includes: sorting out the database clusters that need to be backed up; classifying the database clusters and formulating backup strategies for different types of database clusters; deploying backup software and database backup command scripts on all node servers of the database; completing the configuration of the backup software and the backup command script; and using the unified scheduling function of the backup software to achieve the management goal of database backup integration. Compared with the prior art, the present invention solves the problem that the original backup command of the PolarDB database cannot achieve integrated management, and can achieve integrated management of multiple database instances, greatly reducing the daily operation and maintenance workload of the backup administrator.

[0006] The above existing technologies all have the following problems: 1) Whether it is the TTL table at the partition granularity of PolarDB-X or the TTL table at the row granularity of TiDB, there is a lack of an easy-to-use method or an efficient solution for cleaning up expired data; 2) For distributed databases using the rocksdb engine, there is no solution for using rocksdb's compaction to clean up TTL. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes a method for implementing an expiration schedule for a distributed relational database based on rocksdb. It adopts an active triggering method, stores TTL table data through rocksdb, uses rocksdbcompaction to clean up expired data, and introduces snapshot consistency reading of cleanTs to achieve the purpose of realizing efficient TTL table function.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A method for implementing an expiration schedule of a distributed relational database based on rocksdb, comprising:

[0010] Step S1: After the client connects to the database, a TTL table is created;

[0011] Step S2: The server layer calculates the TTL table data range and registers the TTL cleanup task to the engine layer;

[0012] Step S3: The server layer starts the scheduled task and pushes up the cleanTs according to the TTL calculation. At the same time, the engine layer receives the TTL task, registers the CompactionFilter in rocksdb, and periodically pulls the cleanTs of the current task data from the computing node;

[0013] Step S4: Use the snapshot consistency read method of cleanTs to enable RocksDB to perform compaction and clean up the data within the range by calling CompactionFilter;

[0014] Step S5: The server layer periodically checks whether active compaction is needed based on statistical data. If necessary, the CompactionRange and CompactionFiles instructions are initiated;

[0015] Step S6: After the engine layer receives the active compaction instruction, RocksDB performs compaction and deletes data according to the CompactionFilter to complete data cleanup.

[0016] Specifically, the information of the TTL table in step S1 includes: TTL type, data expiration time, and expiration time column.

[0017] Specifically, the specific steps of step S2 include:

[0018] Step S201: The server layer completes table creation, calculates the key range after TTL table data encoding, and writes the start time of the data into the key of the key-value pair;

[0019] Step S202: registering a cleanup task at the engine layer according to the TTL table data range;

[0020] Step S203: Send an instruction to the engine table that the data in the keys range belongs to the TTL table and needs to be cleaned up by compaction.

[0021] Specifically, the specific steps of step S3 include:

[0022] Step S301: define data model and TTL;

[0023] Step S302: Start a scheduled task at the server layer and periodically increase cleanTs according to TTL calculation;

[0024] Step S303: Through the background thread, on each computing node, the cleanTs of the current task data is periodically pulled;

[0025] Step S304: On each computing node, upon receiving a new TTL task, a CompactionFilter is registered in RocksDB;

[0026] Step S305: When the data expires, the registered CompactionFilter will be triggered.

[0027] Specifically, the snapshot consistency reading method of cleanTs in step S4 includes:

[0028] Step S401: When new data is inserted or existing data is updated, TTL is used as part of the key and stored together with the data in RocksDB;

[0029] Step S402: When the cleanTs value reaches or exceeds the set expiration time, a data cleaning operation is performed;

[0030] Step S403: merging data stored in multiple layers into one layer, and deleting expired or no longer needed data;

[0031] Step S404: Create a class that inherits from CompactionFilter and rewrite the Filter method to determine whether each key-value pair is expired;

[0032] Step S405: Start the database, register and use the custom CompactionFilter, and automatically call the custom Filter method

[0033] Step S406: Use the snapshot function of RocksDB to obtain a timestamp before performing the read operation, and use this timestamp to obtain a data snapshot. If expired data is detected, call the delete API of RocksDB to delete the expired data.

[0034] Specifically, the factors for determining whether active compaction is needed in step S5 include: data volume statistics exceeding a threshold, large data expiration time statistics, and frequent read and write operations with large data volumes.

[0035] Specifically, the steps of initiating the CompactionRange and CompactionFiles instructions in step S5 include:

[0036] Step S501: Call the CompactionRange and CompactionFiles functions of RocksDB and specify the range to be operated;

[0037] Step S502: Obtain relevant range files and a list of files that need to perform compaction operations by reading data files and index files in RocksDB;

[0038] Step S503: Initiate the CompactionRange and CompactionFiles instructions to pass the file to the Compaction function of RocksDB for processing.

[0039] Specifically, the specific steps of RocksDB performing compaction in step S6 include:

[0040] Step S601: Determine the order of compaction operations based on the scores of each layer, start compaction and wait for thread scheduling.

[0041] Step S602: RocksDB divides the compaction into multiple subcompactions, processes them through child threads and puts them into an input array to form an SST file.

[0042] Step S603: traverse the multi-path SST files through mergeIterator, sort the keys in the multi-path files using the minimum heap method, and then take out the top element of the heap each time;

[0043] Step S604: Create an output file, process subcompaction in a child thread, and merge the results into the final output file;

[0044] Step S605: Add a task to the thread pool and wait for scheduling. After all subcompactions are processed, the entire compaction process is completed.

[0045] Specifically, a method for implementing an expiration schedule of a distributed relational database based on rocksdb includes: a data storage module, an expiration time management module, a compaction module, a consistency reading module, and a monitoring and logging module.

[0046] The data storage module uses RocksDB as the backend storage engine and is responsible for storing and reading data;

[0047] The expiration time management module is used to set the expiration policy for each data and manage the expiration time of the data;

[0048] The compaction module is used to perform the compaction operation of RocksDB;

[0049] The consistent read module uses the snapshot function of RocksDB to process data pre-fetching and caching;

[0050] The monitoring and logging module is used to monitor the system's CPU usage and disk I / O indicators in real time, and to detect and handle problems in a timely manner.

[0051] Specifically, the compaction module includes: a compaction trigger unit, a compaction scheduler unit, a compaction worker unit, a compaction filter unit, and a compaction progress monitor unit.

[0052] The compaction trigger unit is used to detect the data expiration time and trigger the compaction operation;

[0053] The compaction scheduler unit is used to reasonably arrange the time and priority of compaction tasks according to the system load and data expiration policy;

[0054] The CompactionWorker unit is responsible for executing the actual compaction operation, obtaining the data files to be merged, and performing the merge and delete operations;

[0055] The CompactionFilter unit is used to check the expiration timestamp of the data, identify and delete the expired data, and retain the unexpired data;

[0056] The Compaction Progress Monitor unit is responsible for monitoring and reporting the progress of the compaction task, and regularly updating the status and progress information of the task.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention proposes a method for implementing an expiration schedule for a distributed relational database based on RocksDB, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple processes, low investment and operation costs, and low production work costs. The purpose of clearing expired data is achieved by using the compaction filter function of RocksDB.

[0059] 2. The present invention proposes a method for implementing an expiration schedule of a distributed relational database based on rocksdb, which adopts the solution of rocksdb compaction to clean up expired data. The data is cleaned up when rocksdb performs compaction. There is no need to query the expired data and then delete it, and there is no need to perform the operation of deleting data. There is no need to maintain the expiration time column index or partition problem. When the distributed database is consistent with multiple copies through the consensus algorithm, the consensus log is reduced, thereby greatly reducing the use of disk IO and reducing the complexity of using TTL tables.

[0060] 3. The present invention proposes a method for implementing an expiration schedule of a distributed relational database based on rocksdb. It proposes to use the rocksdb feature to actively trigger compaction to clean up data based on statistical data for different scenarios of range and sst files. At the same time, in order to solve the problem of asynchronous compaction execution of each rocksdb, a method of using cleanTs to see consistent data from a relational database is used, which optimizes the user experience of the TTL table and has the characteristics of simple use, high efficiency in cleaning up expired data, and low use of system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a method for implementing an expiration schedule of a distributed relational database based on rocksdb in the present invention;

[0062] Figure 2 This is a flow chart of a compaction filter filtering and cleaning expired data based on a rocksdb distributed relational database expiration schedule implementation method of the present invention;

[0063] Figure 3 Add a cleanTs consistency flow chart to the distributed relational database expiration schedule implementation method based on rocksdb of the present invention;

[0064] Figure 4 This is a TTL architecture diagram of clearing expired data based on a rocksdb distributed relational database expiration schedule implementation method of the present invention;

[0065] Figure 5 This is a system architecture diagram of a method for implementing an expiration schedule of a distributed relational database based on rocksdb in the present invention. DETAILED DESCRIPTION

[0066] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, in the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate the orientation or position relationship based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "No. 1", "No. 2" and "No. 3" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The present invention is further explained below in conjunction with specific embodiments.

[0067] Example 1

[0068] See also Figure 1-Figure 4 , an embodiment provided by the present invention: a method for implementing an expiration schedule of a distributed relational database based on rocksdb, comprising the following steps:

[0069] Step S1: After the client connects to the database, a TTL table is created;

[0070] TTL is a mechanism for setting the lifetime of a key in a key-value pair storage. It refers to the survival time of data in the database. When a key-value pair is created, an expiration time can be set for it. After this time, the data will be considered expired and the database will automatically delete the key-value pair.

[0071] Common methods for implementing database expiration schedules are as follows:

[0072] (1) Use date type: Use date type to store expiration time in the database, and judge the expiration time by comparing date values;

[0073] (2) Set the key lifetime: Set the key lifetime in the key-value pair storage. When the key expires, the database will automatically delete the key-value pair.

[0074] (3) Use scheduled tasks: Use scheduled tasks to regularly check whether the data in the database is expired and delete expired data;

[0075] (4) Use triggers: Use triggers in the database to monitor data changes. When data changes, the trigger determines whether to delete expired data based on preset rules.

[0076] (5) Use embedded cache: Combine the cache with the database and set the expiration time of the cache to delete the data when it expires;

[0077] (6) Use distributed locks: Use distributed locks to control data access and deletion. When data expires, use distributed locks to ensure timely deletion of data.

[0078] The present invention uses the TTL of the setting key to implement the database expiration schedule, which is simple and easy to use. There is no need to write additional logic in the application to detect whether the data is expired, and there is no need to maintain a complex expiration schedule in the database, thereby achieving the effect of reducing the burden on the application and database and strengthening predictability.

[0079] Step S2: The server layer calculates the TTL table data range and registers the TTL cleanup task to the engine layer;

[0080] Step S3: The server layer starts the scheduled task and pushes up the cleanTs according to the TTL calculation. At the same time, the engine layer receives the TTL task, registers the CompactionFilter in rocksdb, and periodically pulls the cleanTs of the current task data from the computing node;

[0081] RocksDB is an efficient, high-performance, single-point database engine developed by Facebook based on Google's open source KV storage LevelDB. It uses a log-structured database engine. RocksDB is suitable for tuning in different production environments. It can directly use memory, Flash, hard disk or HDFS, and supports different compression algorithms. It has a complete set of tools for production and debugging. It is also an embedded, persistent storage engine with high performance, fast storage and adaptability.

[0082] CompactionFilter is a pluggable filter that can be attached to the compaction operation, allowing additional processing of data during the compaction process, such as deleting expired data.

[0083] Step S4: Use the snapshot consistency read method of cleanTs to enable RocksDB to perform compaction and clean up the data within the range by calling CompactionFilter;

[0084] Snapshot consistency read refers to reading a consistent view of data at a certain point in time to ensure that the read data is logically correct, which usually involves pre-fetching and caching of data. The present invention adopts the snapshot consistency read method of cleanTs to achieve the effect of ensuring read consistency, improving read performance, reducing network transmission overhead and improving database availability.

[0085] In RocksDB, compaction is an important persistence operation that is responsible for organizing and compressing data stored on disk to free up space and improve query performance. The compaction operation usually scans all data on the disk and organizes it into a continuous data layer. Each layer contains a timestamp, and expired data will be located below newer data, making them easy to identify and clean up.

[0086] Step S5: The server layer periodically checks whether active compaction is needed based on statistical data. If necessary, the CompactionRange and CompactionFiles instructions are initiated;

[0087] Step S6: After the engine layer receives the active compaction instruction, RocksDB performs compaction and deletes data according to the CompactionFilter to complete data cleanup.

[0088] RocksDB compaction uses two methods: 1) a general trigger method with general configuration; 2) an active trigger method for TTL table data. The general trigger method is based on LSM-Tree in storage, and will trigger the compaction process for data when the compaction conditions are met; the active trigger method is a method that is added relative to non-TTL tables and only triggers compaction for TTL tables. In the general compaction method, the compaction conditions may not be met, resulting in data not being deleted. The present invention uses the manual compaction provided by RocksDB to actively trigger the compaction process of the TTL table, and based on statistical information, achieves refined compaction effects, which can take into account the balance between functional requirements and performance. Manual compaction can use two methods: CompactRange and compactFiles.

[0089] The information of the TTL table in step S1 includes: TTL type, data expiration time, and expiration time column.

[0090] The specific steps of step S2 include:

[0091] Step S201: The server layer completes table creation, calculates the key range after TTL table data encoding, and writes the start time of the data into the key of the key-value pair;

[0092] Step S202: registering a cleanup task at the engine layer according to the TTL table data range;

[0093] Step S203: Send an instruction to the engine table that the data in the keys range belongs to the TTL table and needs to be cleaned up by compaction.

[0094] The specific steps of step S3 include:

[0095] Step S301: define data model and TTL;

[0096] Step S302: Start a scheduled task at the server layer and periodically increase cleanTs according to TTL calculation;

[0097] Pushing up cleanTs means calculating and pushing up a specific cleanTs based on the TTL of data items, which is used to mark and delete expired data items. In a distributed relational database, each computing node needs to periodically pull the cleanTs of the current task data and register a CompactionFilter in RocksDB. When the data expires, the registered CompactionFilter will be triggered to perform specific operations, such as deleting or updating the expired data. The method of pushing up cleanTs adopted in the present invention can ensure that the data items in the database can be cleaned and updated in time after expiration, thereby ensuring the efficiency and performance of the database.

[0098] Step S303: Through a background thread, on each computing node, periodically pull the cleanTs of the current task data;

[0099] Step S304: On each computing node, when a new TTL task is received, register a CompactionFilter in RocksDB;

[0100] Step S305: When the data expires, the registered CompactionFilter will be triggered.

[0101] The steps of triggering the CompactionFilter to filter and delete the expired data include: 1) Parse the TTL information from the keys; 2) Compare the TTL of the data with the cleanTs; 3) If TTL < cleanTs, then filter and delete the key-value filter.

[0102] The snapshot consistency read method introducing cleanTs in step S4 includes:

[0103] Step S401: When new data is inserted or existing data is updated, store the TTL as part of the key together with the data in RocksDB;

[0104] Step S402: When the cleanTs value reaches or exceeds the set expiration time, perform data cleaning operations;

[0105] Step S403: Merge the data stored in multiple layers into one layer and delete the expired or no longer needed data;

[0106] Step S404: Create a class inherited from CompactionFilter and override the Filter method to determine whether each key-value pair has expired;

[0107] Step S405: Start the database, register and use the custom CompactionFilter, and automatically call the custom Filter method

[0108] Step S406: Use the snapshot function of RocksDB to obtain a timestamp before performing the read operation, and use this timestamp to obtain a data snapshot. If expired data is detected, call the delete API of RocksDB to delete the expired data.

[0109] The significance of cleanTs is that data before this time is in a cleanable state. As for when to clean up, it is determined by the compaction containing the current data triggered by the storage engine rocksdb. Therefore, data before cleanTs may have been cleaned up by rocksdb, or rocksdb may not have compacted yet and thus has not been cleaned up for the time being.

[0110] In the snapshot consistency read of cleanTs, the conditions for judging whether the current TTL can be read are:

[0111] 1) Based on readTs, the current data TTL can be read. readTs represents the node read by the current snapshot.

[0112] 2) TTL is equal to or later than cleanTs;

[0113] Therefore, the earliest readable range of the current TTL table is [cleanTs, readTs]. As for the inconsistency of row records and index data mentioned above, they will not be read because TTL is earlier than cleanTs, thus achieving consistent reading.

[0114] The factors for determining whether active compaction is needed in step S5 include: the data volume statistics exceed the threshold, the data expiration time statistics are large, and the read and write operations are frequent and the data volume is large.

[0115] The steps of initiating the CompactionRange and CompactionFiles instructions in step S5 include:

[0116] Step S501: Call the CompactionRange and CompactionFiles functions of RocksDB and specify the range to be operated;

[0117] CompactFiles is a granular update method for triggering compaction compared to CompactRange. It is a supplement to CompactRange and is lightweight compared to CompactRange, enabling finer control over compaction.

[0118] The CompactionRange function accepts two parameters, the start key and the end key. Within this range, the files that need to perform the compaction operation will be selected.

[0119] Step S502: Obtain relevant range files and a list of files that need to perform compaction operations by reading data files and index files in RocksDB;

[0120] Step S503: Initiate the CompactionRange and CompactionFiles instructions to pass the file to the Compaction function of RocksDB for processing.

[0121] The CompactRange trigger can be based on TTL table statistics and database statistics, such as by periodically scanning TTL data keys. The conditions for triggering the CompactRange can be customized:

[0122] 1) The number of keys to be deleted in the scan range reaches a certain threshold;

[0123] 2) Within the scan range, the number of keys to be deleted exceeds a certain ratio of the total number of keys;

[0124] 3) Server-level statistics: data exceeding a certain threshold in the past period of time is deleted;

[0125] 4) Set periodic long time intervals and timed triggers.

[0126] The specific steps of RocksDB performing compaction in step S6 include:

[0127] Step S601: Determine the order of compaction operations based on the scores of each layer, start compaction and wait for thread scheduling.

[0128] Step S602: RocksDB divides the compaction into multiple subcompactions, processes them through child threads and puts them into an input array to form an SST file.

[0129] In the implementation method of the expiration schedule of a distributed relational database based on RocksDB, the SST file is a file used to persist database data. The SST file has multiple formats, and the default is BlockBasedTable, which, as the name implies, is stored based on data blocks.

[0130] The advantages of using the SST file format in the present invention are: providing persistent storage, sequential writing, binary search, optimized storage space, and support for multiple data structures and auxiliary data blocks, which helps to improve the performance and availability of the database.

[0131] Step S603: traverse the multi-path SST files through mergeIterator, sort the keys in the multi-path files using the minimum heap method, and then take out the top element of the heap each time;

[0132] Step S604: Create an output file, process subcompaction in a child thread, and merge the results into the final output file;

[0133] Step S605: Add a task to the thread pool and wait for scheduling. After all subcompactions are processed, the entire compaction process is completed.

[0134] Example 2

[0135] See also Figure 5 Another embodiment provided by the present invention is a method for implementing an expiration schedule of a distributed relational database based on rocksdb, comprising:

[0136] Data storage module, expiration time management module, compaction module, consistent reading module, monitoring and log module,

[0137] The data storage module uses RocksDB as the backend storage engine and is responsible for storing and reading data;

[0138] The expiration time management module is used to set the expiration policy for each data and manage the expiration time of the data;

[0139] The compaction module is used to perform the compaction operation of RocksDB;

[0140] The consistent read module uses the snapshot function of RocksDB to process data pre-fetching and caching;

[0141] The monitoring and logging module is used to monitor the system's CPU usage and disk I / O indicators in real time, and to detect and handle problems in a timely manner.

[0142] The compaction module includes: a compaction trigger unit, a compaction scheduler unit, a compaction worker unit, a compaction filter unit, and a compaction progress monitor unit.

[0143] The compaction trigger unit is used to detect the data expiration time and trigger the compaction operation;

[0144] The compaction scheduler unit is used to reasonably arrange the time and priority of compaction tasks according to the system load and data expiration policy;

[0145] The Compaction Worker unit is responsible for performing the actual compaction operation, obtaining the data files to be merged, and performing the merge and delete operations;

[0146] The Compaction Filter unit is used to check the expiration timestamp of the data, identify and delete the expired data, and retain the unexpired data;

[0147] The Compaction Progress Monitor unit is responsible for monitoring and reporting the progress of the compaction task, and regularly updating the status and progress information of the task.

[0148] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are within the protection of the present invention.

Claims

1. A method for implementing an expiration schedule for a distributed relational database based on rocksdb. It is characterized in that include: Step S1: After the client connects to the database, a TTL table is created; Step S2: The server layer calculates the TTL table data range and registers the TTL cleanup task to the engine layer; Step S3: The server layer starts the scheduled task and pushes up the cleanTs according to the TTL calculation. At the same time, the engine layer receives the TTL task, registers the CompactionFilter in rocksdb, and periodically pulls the cleanTs of the current task data from the computing node; Step S4: Use the snapshot consistency read method of cleanTs to enable RocksDB to perform compaction and clean up the data within the range by calling CompactionFilter; Step S5: The server layer periodically checks whether active compaction is needed based on statistical data. If necessary, the CompactionRange and CompactionFiles instructions are initiated; Step S6: After the engine layer receives the active compaction instruction, RocksDB performs compaction and deletes data according to the CompactionFilter to complete data cleanup; The snapshot consistency reading method of cleanTs in step S4 includes: Step S401: When new data is inserted or existing data is updated, TTL is used as part of the key and stored together with the data in RocksDB; Step S402: When the cleanTs value reaches or exceeds the set expiration time, a data cleaning operation is performed; Step S403: merging data stored in multiple layers into one layer, and deleting expired or no longer needed data; Step S404: Create a class that inherits from CompactionFilter and rewrite the Filter method to determine whether each key-value pair is expired; Step S405: Start the database, register and use the custom CompactionFilter, and automatically call the custom Filter method Step S406: Use the snapshot function of RocksDB to obtain a timestamp before performing the read operation, and use this timestamp to obtain a data snapshot. If expired data is detected, call the delete API of RocksDB to delete the expired data.

2. A method for implementing an expiration schedule of a distributed relational database based on rocksdb as claimed in claim 1, It is characterized in that The information of the TTL table in step S1 includes: TTL type, data expiration time, and expiration time column.

3. A method for implementing a distributed relational database expiration schedule based on rocksdb as claimed in claim 2, It is characterized in that The specific steps of step S2 include: Step S201: The server layer completes table creation, calculates the key range after TTL table data encoding, and writes the start time of the data into the key of the key-value pair; Step S202: registering a cleanup task at the engine layer according to the TTL table data range; Step S203: Send an instruction to the engine table that the data in the keys range belongs to the TTL table and needs to be cleaned up by compaction.

4. A method for implementing an expiration schedule of a distributed relational database based on rocksdb as claimed in claim 3, It is characterized in that The specific steps of step S3 include: Step S301: define data model and TTL; Step S302: Start a scheduled task at the server layer and periodically increase cleanTs according to TTL calculation; Step S303: Through the background thread, on each computing node, the cleanTs of the current task data is periodically pulled; Step S304: On each computing node, upon receiving a new TTL task, a CompactionFilter is registered in RocksDB; Step S305: When the data expires, the registered CompactionFilter will be triggered.

5. A method for implementing a distributed relational database expiration schedule based on rocksdb as claimed in claim 4, It is characterized in that The factors for determining whether active compaction is needed in step S5 include: the data volume statistics exceed the threshold, the data expiration time statistics are large, and the read and write operations are frequent and the data volume is large.

6. A method for implementing a distributed relational database expiration schedule based on rocksdb as claimed in claim 5, It is characterized in that The steps of initiating the CompactionRange and CompactionFiles instructions in step S5 include: Step S501: Call the CompactionRange and CompactionFiles functions of RocksDB and specify the range to be operated; Step S502: Obtain relevant range files and a list of files that need to perform compaction operations by reading data files and index files in RocksDB; Step S503: Initiate the CompactionRange and CompactionFiles instructions to pass the file to the Compaction function of RocksDB for processing.

7. A method for implementing a distributed relational database expiration schedule based on rocksdb as claimed in claim 6, It is characterized in that The specific steps of RocksDB performing compaction in step S6 include: Step S601: Determine the order of compaction operations based on the scores of each layer, start compaction and wait for thread scheduling. Step S602: RocksDB divides the compaction into multiple subcompactions, processes them through child threads and puts them into an input array to form an SST file. Step S603: traverse the multi-path SST files through mergeIterator, sort the keys in the multi-path files using the minimum heap method, and then take out the top element of the heap each time; Step S604: Create an output file, process subcompaction in a child thread, and merge the results into the final output file; Step S605: Add a task to the thread pool and wait for scheduling. After all subcompactions are processed, the entire compaction process is completed.

8. A method for implementing an expiration schedule of a distributed relational database based on rocksdb as claimed in claim 7, It is characterized in that include: Data storage module, expiration time management module, compaction module, consistent reading module, monitoring and log module, The data storage module uses RocksDB as the backend storage engine and is responsible for storing and reading data; The expiration time management module is used to set the expiration policy for each data and manage the expiration time of the data; The compaction module is used to perform the compaction operation of RocksDB; The consistent read module uses the snapshot function of RocksDB to process data pre-fetching and caching; The monitoring and logging module is used to monitor the system's CPU usage and disk I / O indicators in real time, and to detect and handle problems in a timely manner.

9. A method for implementing a distributed relational database expiration schedule based on rocksdb as claimed in claim 8, It is characterized in that The compaction module includes: a compaction trigger unit, a compaction scheduler unit, a compaction worker unit, a compaction filter unit, and a compaction progress monitor unit. The compaction trigger unit is used to detect the data expiration time and trigger the compaction operation; The compaction scheduler unit is used to reasonably arrange the time and priority of compaction tasks according to the system load and data expiration policy; The Compaction Worker unit is responsible for performing the actual compaction operation, obtaining the data files to be merged, and performing the merge and delete operations; The Compaction Filter unit is used to check the expiration timestamp of the data, identify and delete the expired data, and retain the unexpired data; The Compaction Progress Monitor unit is responsible for monitoring and reporting the progress of the compaction task, and regularly updating the status and progress information of the task.

Images