TTL control method, system and device based on RocksDB and medium

Abstract

The invention discloses a TTL (Transistor-Transistor Logic) control method, system and equipment based on RocksDB and a medium. The method comprises the following steps: inserting a timestamp in front of a value part of each key value pair; when the put operation is executed, TTL is calculated according to provided preset expiration time, and the TTL obtained through calculation is stored in the first eight bytes of the value; when data are read, the timestamp in the value is firstly analyzed, the analyzed timestamp is compared with the current time, when the analyzed timestamp is earlier than the current time, it is considered that the read data are expired, the read data are asynchronously deleted, and when the analyzed timestamp is later than the current time, the read data are output. According to the method, the system, the equipment and the medium, the TTL of the data can be flexibly and accurately controlled, and meanwhile, the expired data can be timely processed.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data and relates to a TTL control method, system, device and medium based on RocksDB. Background Art

[0002] With the rapid development of big data technology, various application scenarios have put forward higher requirements for data management, especially in situations where the life cycle of data needs to be precisely controlled, such as cache systems, message queues, session management, etc. The TTL (Time-To-Live) mechanism is a common method for managing these time-sensitive data. It allows data to automatically expire and be deleted from the storage system after a specified time, thereby reducing unnecessary data storage and retrieval burdens.

[0003] As an efficient key-value storage engine, RocksDB is widely used in scenarios that need to process large amounts of data. However, the existing RocksDB has some limitations in TTL control. First, RocksDB cannot precisely control the expiration time of each key-value pair, but can only set the same TTL for all key-value pairs in the entire column family. This means that in a column family, all key-value pairs share the same expiration policy, and cannot flexibly handle the specific needs of a single key-value pair. This limitation is particularly inconvenient when you need to process data with different life cycles.

[0004] Secondly, RocksDB does not implement instant deletion for each key-value pair when it expires. Although RocksDB supports setting global TTL, it does not immediately clear the data when the key-value pair expires, but relies on the subsequent compaction process to clean up. Therefore, even if a key-value pair has expired, the user may still read the expired data when executing a query. For scenarios that need to ensure data accuracy, this undoubtedly increases the complexity of the system and the difficulty of data processing.

[0005] In addition, the existing RocksDB does not provide a mechanism to shield expired data during the reading process of TTL data, which means that even if the data is no longer valid, users may still obtain these expired information when querying, which will affect the correctness and reliability of the application. This inconvenience not only increases the additional processing cost of developers, but may also cause some unexpected errors and problems.

[0006] Therefore, the shortcomings of the existing technology are that RocksDB cannot flexibly and accurately control the TTL of data, and the processing of expired data is not timely enough, which affects the efficiency of data management and the reliability of the system. In order to solve these problems, a new mechanism is urgently needed to achieve precise TTL control of each key-value pair in RocksDB and process the data immediately after it expires, so as to optimize the performance and user experience of the storage system. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a TTL control method, system, device and medium based on RocksDB, which can flexibly and accurately control the TTL of data and can process expired data in a timely manner.

[0008] To achieve the above object, the present invention discloses a TTL control method based on RocksDB, comprising:

[0009] Insert a timestamp before the value part of each key-value pair;

[0010] When performing a put operation, the TTL is calculated based on the preset expiration time provided, and the calculated TTL is stored in the first 8 bytes of value;

[0011] When reading data, first parse the timestamp in the value, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired, and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

[0012] A further improvement of the RocksDB-based TTL control method of the present invention is:

[0013] Furthermore, the timestamp is 8 bytes, and the timestamp is used to mark the expiration time of the data.

[0014] Furthermore, when performing a put operation, the TTL is calculated based on the provided preset expiration time, and the calculated TTL is stored in the first 8 bytes of value. When the preset expiration time is not provided or the provided preset expiration time is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data never expires.

[0015] Furthermore, the read data is asynchronously deleted through the Compaction Filter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

[0016] The present invention discloses a TTL control system based on RocksDB, which is characterized by comprising:

[0017] Insert module, used to insert a timestamp before the value part of each key-value pair;

[0018] A calculation module, used for calculating TTL according to the provided preset expiration time when performing a put operation, and storing the calculated TTL in the first 8 bytes of value;

[0019] The reading module is used to parse the timestamp in the value when reading data, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

[0020] As a further improvement of the TTL control system based on RocksDB described in the present invention,

[0021] Furthermore, the timestamp is 8 bytes, and the timestamp is used to mark the expiration time of the data.

[0022] Furthermore, when performing a put operation, the TTL is calculated based on the provided preset expiration time, and the calculated TTL is stored in the first 8 bytes of value. When the preset expiration time is not provided or the provided preset expiration time is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data never expires.

[0023] Furthermore, the read data is asynchronously deleted through the Compaction Filter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

[0024] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the RocksDB-based TTL control method are implemented.

[0025] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the RocksDB-based TTL control method are implemented.

[0026] The present invention has the following beneficial effects:

[0027] The RocksDB-based TTL control method, system, device and medium of the present invention can realize independent TTL control of each key-value pair by inserting a timestamp in front of the value part of each key-value pair during specific operation, so that different data entries can set different expiration times according to their respective needs. Compared with the traditional global TTL strategy, this method is more flexible and can adapt to complex data management needs. At the same time, the expired data is asynchronously deleted, so that the expired data is processed in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0032] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0034] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0035] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0037] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0038] Embodiment 1

[0039] refer to Figure 1 The RocksDB-based TTL control method of the present invention comprises the following steps:

[0040] 1) Data structure design;

[0041] A timestamp is inserted before the value part of each key-value pair. The timestamp is 8 bytes and is used to mark the expiration time of the data.

[0042] 2) TTL management of put operations;

[0043] When performing a put operation, the TTL is calculated based on the preset expiration time, and the calculated TTL is stored in the first 8 bytes of value; wherein, when the preset expiration time is not provided or the provided preset expiration time is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data never expires.

[0044] 3) TTL check for get and exist operations;

[0045] When reading data, first parse the timestamp in the value, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired, and the read data is not returned, and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

[0046] The process of asynchronously deleting the read data is as follows:

[0047] The read data is asynchronously deleted through the Compaction Filter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

[0048] It should be noted that

[0049] An 8-byte timestamp is added before the value part of each key-value pair to store the expiration time of the data.

[0050] Format example:

[0051] Original data structure: [actual data]

[0052] The data structure after adding TTL: [TTL timestamp (8 bytes)] + [actual data]

[0053] TTL timestamp means:

[0054] Use long data to represent UNIX timestamps, indicating the future expiration time of the data. For example, the current time plus the timestamp after the expiration date.

[0055] Specific implementation of put operation:

[0056] The put operation is responsible for writing data to RocksDB and calculating and storing the TTL timestamp.

[0057] pseudocode:

[0058]

[0059]

[0060] Key parameter explanation:

[0061] ttlInSeconds is the validity period of the data, in seconds.

[0062] expirationTime calculates the future expiration time of the data in milliseconds. valueWithTTL is the actual stored data including the TTL timestamp.

[0063] TTL check for get and exist operations:

[0064] The get and exist operations need to check the TTL timestamp in the value to decide whether to return the data.

[0065] pseudocode:

[0066]

[0067]

[0068] Key parameter explanation:

[0069] The expirationTime is parsed from the first 8 bytes of the value and is used to determine whether the data is expired.

[0070] If the data has expired, the key-value pair will be deleted asynchronously and null will be returned.

[0071] actualValue is the actual data extracted from the data containing the TTL timestamp.

[0072] Implementation of custom Compaction Filter:

[0073] The custom Compaction Filter is used to automatically clean up expired data during the data compression process.

[0074] pseudocode:

[0075]

[0076]

[0077] Explanation of key steps:

[0078] The filter method is called at each compaction and checks the TTL timestamp of each key-value pair.

[0079] If the data has expired, it returns true, indicating that the record is deleted.

[0080]

[0081]

[0082] RocksDB configuration and initialization:

[0083] When initializing RocksDB, set the custom Compaction Filter Factory to the RocksDB configuration.

[0084] pseudocode:

[0085]

[0086] Explanation of key steps:

[0087] Apply the custom CompactionFilter Factory to the RocksDB instance through the options.setCompactionFilterFactory method.

[0088] The RocksDB.open method opens the database and applies the configuration.

[0089] It should be noted that the present invention introduces a precise TTL control strategy in RocksDB to solve many problems in the prior art. First, by inserting an 8-byte timestamp in front of the value part of each key-value pair, the present invention can achieve independent TTL control for each key-value pair, so that different data entries can set different expiration times according to their respective needs. Compared with the traditional global TTL strategy, this method is more flexible and can adapt to complex data management needs.

[0090] Secondly, the present invention introduces a TTL check mechanism in the get and exist operations to ensure that expired data is automatically filtered out during query, and supports asynchronous deletion, thereby avoiding the interference of expired data on query results. The accuracy and reliability of the system are greatly improved, especially in scenarios that require real-time performance and data accuracy.

[0091] Thirdly, by customizing the Compaction Filter, the present invention can automatically clear expired data during the compaction process of RocksDB, further reducing the storage space occupied and optimizing storage efficiency. This automatic cleaning mechanism not only reduces the burden on developers, but also effectively avoids the problem of system performance degradation due to the accumulation of expired data.

[0092] The present invention is applicable to a wide range of application scenarios, especially in systems that require precise TTL control of large amounts of data. For example, in a distributed cache system, different cached data may have different life cycles, and the traditional global TTL strategy cannot meet this requirement. The present invention can set TTL for each cache entry separately, effectively improving the flexibility and efficiency of cache management.

[0093] Embodiment 2

[0094] The RocksDB-based TTL control system of the present invention comprises:

[0095] Insert module, used to insert a timestamp before the value part of each key-value pair;

[0096] A calculation module, used for calculating TTL according to the provided preset expiration time when performing a put operation, and storing the calculated TTL in the first 8 bytes of value;

[0097] The reading module is used to parse the timestamp in the value when reading data, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

[0098] In one implementation manner of the present invention, the timestamp is 8 bytes, and the timestamp is used to mark the expiration time of the data.

[0099] In one embodiment of the present invention, when performing a put operation, TTL is calculated according to a provided preset expiration time, and the calculated TTL is stored in the first 8 bytes of value. When a preset expiration time is not provided or the provided preset expiration time is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data never expires.

[0100] In one implementation of the present invention, the read data is asynchronously deleted through a Compaction Filter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

[0101] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0102] Embodiment 3

[0103] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the RocksDB-based TTL control method are implemented, for example, including: inserting a timestamp before the value part of each key-value pair; when performing a put operation, calculating the TTL according to the preset expiration time provided, and storing the calculated TTL in the first 8 bytes of the value; when reading data, first parse the timestamp in the value, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, it is considered that the read data has expired, and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard structure bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs, specifically, the programs may include program codes, and the program codes include computer operation instructions. The memory may include internal memory and non-volatile memory, and provide instructions and data to the processor.

[0104] Embodiment 4

[0105] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the RocksDB-based TTL control method are implemented, for example, including: inserting a timestamp before the value part of each key-value pair; when performing a put operation, calculating the TTL according to the preset expiration time provided, and storing the calculated TTL in the first 8 bytes of the value; when reading data, first parse the timestamp in the value, and compare the parsed timestamp with the current time, when the parsed timestamp is earlier than the current time, it is considered that the read data has expired, and the read data is asynchronously deleted, and when the parsed timestamp is later than the current time, the read data is output. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0111] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0112] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A TTL control method based on RocksDB, characterized in that: include: Insert a timestamp before the value part of each key-value pair; When performing a put operation, the TTL is calculated based on the preset expiration time provided, and the calculated TTL is stored in the first 8 bytes of value; When reading data, first parse the timestamp in the value, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired, and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

2. The RocksDB-based TTL control method according to claim 1, characterized in that: The timestamp is 8 bytes and is used to mark the expiration time of the data.

3. The RocksDB-based TTL control method according to claim 1, characterized in that: When performing a put operation, the TTL is calculated based on the preset expiration time provided, and the calculated TTL is stored in the first 8 bytes of value. If the preset expiration time is not provided or the preset expiration time provided is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data will never expire.

4. The RocksDB-based TTL control method according to claim 1, characterized in that: The read data is asynchronously deleted through CompactionFilter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

5. A TTL control system based on RocksDB, characterized in that: include: Insert module, used to insert a timestamp before the value part of each key-value pair; A calculation module, used for calculating TTL according to the provided preset expiration time when performing a put operation, and storing the calculated TTL in the first 8 bytes of value; The reading module is used to parse the timestamp in the value when reading data, and compare the parsed timestamp with the current time. When the parsed timestamp is earlier than the current time, the read data is considered expired and the read data is asynchronously deleted. When the parsed timestamp is later than the current time, the read data is output.

6. The RocksDB-based TTL control system according to claim 4, characterized in that: The timestamp is 8 bytes and is used to mark the expiration time of the data.

7. The RocksDB-based TTL control system according to claim 4, characterized in that: When performing a put operation, the TTL is calculated based on the preset expiration time provided, and the calculated TTL is stored in the first 8 bytes of value. If the preset expiration time is not provided or the preset expiration time provided is less than or equal to 0, the maximum value of the long type is stored in the first 8 bytes of value, indicating that the data will never expire.

8. The RocksDB-based TTL control system according to claim 4, characterized in that: The read data is asynchronously deleted through CompactionFilter, wherein the Compaction Filter is used to check and delete expired data during the compaction process of RocksDB.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the RocksDB-based TTL control method are implemented as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the RocksDB-based TTL control method are implemented as described in any one of claims 1 to 4.

Images