Database-based compression processing method and device, equipment, medium and product
By counting the number of valid and deleted key values during database traversal, determining whether compression conditions are met, and then performing compression, the performance issues caused by tombstone marking are resolved, improving database read performance and availability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-04-10
AI Technical Summary
The frequent skipping of tombstone markers caused by numerous deletion operations in the existing database reduces read performance, increases unnecessary low-level traversal overhead, and leads to read amplification and performance degradation issues.
During database traversal, the system records the number of valid and deleted key values, determines whether compression conditions are met based on the ratio, and performs compression when the conditions are met. Compression tasks are managed through a pre-defined linked list.
This reduces the number of deleted key-value pairs in the database, improves read performance, reduces underlying traversal overhead, avoids read amplification issues, and enhances the overall performance and availability of the database.
Smart Images
Figure CN121833696A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, and in particular to database-based compression processing methods, apparatus, devices, media, and products. Background Technology
[0002] Currently, databases (such as RocksDB, an embedded key-value database) can optimize data writing by using the LSM-Tree (Log-Structured Merge-Tree) storage structure for high-throughput, data-intensive applications.
[0003] In existing technology, the deletion operation in RocksDB is performed by writing a tombstone marker to indicate that a key-value pair has been deleted. The key-value pair is not immediately removed from the disk, but is cleaned up by a subsequent compression process.
[0004] However, when there are a large number of delete operations, the database will have a large number of tombstone markers. This means that during traversal or query operations, invalid tombstone markers need to be frequently skipped, thus reducing read performance, increasing unnecessary low-level traversal overhead, and causing read amplification. This not only increases system load but also leads to database performance degradation, ultimately causing availability issues. Summary of the Invention
[0005] In view of the above problems, a database-based compression processing method, apparatus, device, medium, and product are proposed to overcome or at least partially solve the above problems, including: A database-based compression processing method, the method comprising: During the traversal of the database, the first count of valid key values and the second count of deleted key values are recorded; wherein the deleted key values are marked with a tombstone. Based on the first statistical count and the second statistical count, determine whether the traversal key value range corresponding to the current traversal process meets the preset compression conditions; When the key-value range corresponding to the current traversal process meets the preset compression conditions, the key-value range corresponding to the current traversal process is compressed.
[0006] Optionally, it is characterized in that, When the key-value range corresponding to the current traversal process meets the preset compression conditions, the key-value range corresponding to the current traversal process is compressed, including: When the key-value interval corresponding to the current traversal process meets the preset compression conditions, add the identifier of the key-value interval corresponding to the current traversal process to the preset linked list. When performing the compression task, the traversal key value range recorded in the preset linked list is compressed.
[0007] Optionally, during the compression task, the traversal key-value range recorded in the preset linked list is compressed, including: When performing a compression task, the target compression key value range is determined from the traversed key value range recorded in the preset linked list, and the target compression key value range is compressed.
[0008] Optionally, when performing a compression task, determining the target compression key value range from the traversed key value ranges recorded in the preset linked list includes: When performing a compression task, determine the current range of compression key values; Determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. Delete the head traversal key value interval and adopt a new head traversal key value interval. Perform a judgment to determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. If there is no record in the preset linked list, or if the current compressed key value interval does not intersect with the head traversal key value interval, the current compressed key value interval is determined as the target compressed key value interval.
[0009] Optionally, it also includes: When the current compressed key value range is the initial value, obtain the head traversal key value range located at the head of the preset linked list, and update the current compressed key value range to the head traversal key value range; wherein, the initial value of the current compressed key value range is empty.
[0010] Optionally, the identifier of the traversed key value range includes the traversal start key value and the traversal end key value of the traversed key value range.
[0011] Optionally, the compression task is characterized by being executed periodically by creating a thread.
[0012] Optionally, based on the first statistical count and the second statistical count, it is determined whether the traversal key-value range corresponding to the current traversal process meets the preset compression conditions, including: Determine the ratio of the second statistical quantity to the sum of the first and second statistical quantities; Determine whether the ratio is greater than a threshold; If the ratio is greater than the threshold, it is determined that the traversal key value range corresponding to the current traversal process meets the preset compression conditions.
[0013] Optionally, before recording the first count of valid key values encountered and the second count of deleted key values during the traversal of the database, the method further includes: In response to a deletion operation on a key value in the database, a tombstone marker is set for the deleted key value.
[0014] Optionally, the feature is that, during the traversal of the database, recording a first statistical count of valid key values encountered and a second statistical count of deleted key values includes: During the traversal of the database, the first statistical count of valid key values and the second statistical count of deleted key values are determined based on the tombstone marker settings of the key values.
[0015] Optionally, the database uses a hierarchical structure to store data, and the database has multiple hierarchical files.
[0016] A database-based compression processing apparatus, the apparatus comprising: The quantity statistics module is used to record the first statistical count of valid key values and the second statistical count of deleted key values during the traversal of the database; wherein the deleted key values are marked with a tombstone. The compression condition judgment module is used to determine whether the traversal key value range corresponding to the current traversal process meets the preset compression condition based on the first statistical quantity and the second statistical quantity. The compression module is used to compress the traversal key-value range corresponding to the current traversal process when the preset compression conditions are met.
[0017] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.
[0018] A computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method described above.
[0019] A computer program product includes a computer program that, when executed by a processor, implements the method described above.
[0020] The embodiments of the present invention have the following advantages: In this embodiment of the invention, during the traversal of the database, a first statistical count of valid key values and a second statistical count of deleted key values are recorded; wherein deleted key values are marked with tombstones; based on the first and second statistical counts, it is determined whether the traversed key value interval corresponding to the current traversal process meets a preset compression condition; when the traversed key value interval corresponding to the current traversal process meets the preset compression condition, the traversed key value interval corresponding to the current traversal process is compressed. This achieves compression of the traversed key value interval based on the statistical count of valid key values and the statistical count of deleted key values marked with tombstones, thereby reducing the number of deleted key values in the database, improving the database read performance, reducing the underlying traversal overhead, avoiding read amplification problems, thereby reducing the system load, improving the overall performance of the database, and enhancing the availability of the database. Attached Figure Description
[0021] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart of a database-based compression processing method provided in some embodiments of the present invention; Figure 2 This is a flowchart of a database-based compression processing method II provided in some embodiments of the present invention; Figure 3 This is a structural block diagram of a database-based compression processing device provided in some embodiments of the present invention. Detailed Implementation
[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0024] In related technologies, databases (such as Rocksdb) can optimize data writing. Deletion operations in a database are performed by writing a tombstone marker to indicate that a key-value pair has been deleted; the key-value pair is not immediately removed from the disk.
[0025] Newly written data is initially placed in the layer with the lowest sequence number. Through compression, it gradually moves towards layers with higher sequence numbers. Reading begins from the layer with the lowest sequence number. If the data is not found in layer 0, it will proceed to layer 1, and so on, until the data is found or all layers have been searched. When reading deleted key-value pairs, if a tombstone marker for the corresponding key is found in a layer with a lower sequence number, it indicates that the key-value pair has been deleted, and the search will not continue to layers with higher sequence numbers.
[0026] The mechanism of writing and deleting tombstone markers can impact read performance in certain scenarios. For example, after deleting a large number of key-value pairs, reading, writing, or traversing the database may result in encountering a large number of tombstone markers during the read process. Since the keys of the tombstone markers have been deleted, there will be a large number of skipping invalid tombstones during the traversal, causing read amplification and inefficiency in read operations.
[0027] To address the performance issues caused by tombstone marking, compression can be used to eliminate it. Full compression of the database can eliminate tombstone marking, but this leads to significant write amplification, and full compression will also consume a considerable amount of time when the database contains a large amount of data.
[0028] Based on this, this embodiment of the invention proposes to calculate whether a specific range of key-value pair records need to be compressed by counting the number of tombstone markers during traversal operations, so as to trigger compression actions in a timely manner for a large number of deletion operations and avoid negative impacts on the availability of data and the system.
[0029] When a database (such as RocksDB) experiences a large number of delete operations, it generates numerous tombstone markers. This necessitates frequent skipping of invalid tombstone markers during traversal or query operations, reducing read performance, increasing unnecessary low-level traversal overhead, and causing read amplification. This not only increases system load but also degrades database performance, ultimately leading to availability issues. Therefore, after a large number of delete operations, data compression should be performed as soon as possible to reduce unnecessary resource overhead.
[0030] Data compression can include two aspects: one is determining the conditions that trigger compression, and the other is compressing a range of key-value pairs.
[0031] The present invention will be further described below with reference to the accompanying drawings: Reference Figure 1The diagram illustrates a flowchart of a database-based compression processing method provided by some embodiments of the present invention, which is applied to a database.
[0032] As some examples, the database could be Rocksdb (an embedded key-value database).
[0033] RocksDB is a high-performance embedded key-value database primarily designed for data-intensive applications requiring high throughput and low latency access. It optimizes write performance through its LSM-Tree storage structure, making it particularly suitable for write-heavy, read-light scenarios.
[0034] RocksDB supports features such as compression, transactions, column families, snapshots, and write-ahead log (WAL). It offers a wealth of configuration options to adapt to different hardware environments, such as disks, SSDs (Solid State Drives), and memory.
[0035] Specifically, this may include the following steps: Step 101: During the process of traversing the database, record the first statistical count of valid key values traversed and the second statistical count of deleted key values; wherein the deleted key values are marked with tombstones.
[0036] In some embodiments of the present invention, the database uses a hierarchical structure to store data, and the database has multiple hierarchical files.
[0037] As examples, a database hierarchy can include multiple hierarchical files with different sequence numbers. For example, level 0, level 1, and so on up to level N. When data is written to the database, it first enters the level with the smallest sequence number, i.e., level 0. Then, as subsequent data is written and compression operations are performed, the data gradually moves from the level with the smallest sequence number to the level with the largest sequence number.
[0038] When traversing the database, you can examine the key-value pairs in each level of the file. For each key-value pair, you can determine whether it is a valid key-value pair or a deleted key-value pair based on the tombstone marker. For example, if it is a valid key-value pair, increment the first count by 1; if it is a deleted key-value pair (i.e., a key-value pair with a tombstone marker), increment the second count by 1.
[0039] By traversing multiple levels of files in the database, the first count of valid key values and the second count of deleted key values can be determined.
[0040] In some examples, during traversal, each level of the RocksDB database has a cursor that reads key-value pairs from an SST (Sorted String Table File) file. RocksDB can also traverse the entire database by forming a min-heap from the cursors at each level and moving the cursor at the top of the heap backward while maintaining the min-heap.
[0041] In RocksDB database traversal, a cursor is an abstract data structure / tool used to locate and read key-value pairs in an SST file; it is equivalent to a file read pointer, which can indicate the position of the current key-value pair in the corresponding SST file, and can also quickly jump to the next key-value pair according to the sorting rules.
[0042] During the movement of the cursor, valid key values and deleted key values can be traversed. Therefore, the number of valid key values traversed in each traversal operation can be counted as key_num (the first count of valid key values), and the number of deleted key values can be counted as del_num (the second count of deleted key values).
[0043] In some embodiments of the present invention, during the traversal of the database, recording a first statistical count of valid key values encountered and a second statistical count of deleted key values includes: During the traversal of the database, the first statistical count of valid key values and the second statistical count of deleted key values are determined based on the tombstone marker settings of the key values.
[0044] In practical applications, key-value pairs can be scanned layer by layer according to the hierarchical structure of the database, and tombstone markers can be used to distinguish between valid and deleted key-value pairs.
[0045] For example, when a key-value pair is scanned, it is checked whether a tombstone marker is set. If not, it means that the key-value pair is valid, and the first count of valid key-value pairs (key_num) is incremented by 1. If a tombstone marker is set, it means that the key-value pair has been deleted, and the second count of deleted key-value pairs (del_num) is incremented by 1.
[0046] In some examples, after each traversal of the rocksdb database, a tuple iter_tuple (traversal information tuple) can be generated, such as (start_key, end_key, key_num, del_num), which can correspond to the start position of the traversal, the end position of the traversal, the first count of valid keys, and the second count of deleted keys, respectively.
[0047] In some embodiments of the present invention, before recording the first statistical count of valid key values and the second statistical count of deleted key values during the traversal of the database, the method further includes: In response to a deletion operation on a key value in the database, a tombstone marker is set for the deleted key value.
[0048] As examples, a deletion operation can be a user-initiated deletion command or an automatic deletion operation triggered by predefined rules (such as data expiration, data cleanup strategies, etc.).
[0049] In practical applications, when a deletion command for a key-value pair is received, a tombstone marker can be set at the location corresponding to the key-value pair to indicate that the key-value pair has been deleted, but the data of the key-value pair will not be removed from the physical storage immediately.
[0050] Step 102: Based on the first statistical quantity and the second statistical quantity, determine whether the traversal key value range corresponding to the current traversal process meets the preset compression conditions.
[0051] As examples, traversing a key-value range refers to sequentially traversing the range of key values covered during database traversal. A key-value range can correspond to a continuous segment of key values, such as all key values from key A to key D arranged lexicographically. The preset compression condition can be a pre-defined threshold range.
[0052] In practical applications, based on the key value range corresponding to the current traversal process, the first statistical number of valid key values and the second statistical number of deleted key values within the traversal key value range can be counted. Then, based on the first and second statistical numbers, the proportion of the second statistical number to (the first statistical number + the second statistical number) can be calculated. Finally, this proportion can be compared with a preset threshold to determine whether the traversal key value range meets the compression conditions.
[0053] In some embodiments of the present invention, determining whether the traversal key-value range corresponding to the current traversal process meets the preset compression conditions based on the first statistical quantity and the second statistical quantity includes: Determine the ratio of the second statistical quantity to the sum of the first statistical quantity and the second statistical quantity; determine whether the ratio is greater than a threshold; if the ratio is greater than the threshold, determine that the traversal key value interval corresponding to the current traversal process meets the preset compression conditions.
[0054] As examples, a pre-defined linked list and a threshold can be set in advance; the threshold can be a range of values, for example, the threshold can be set to a value between 0 and 1.
[0055] In practical applications, the ratio of the second statistical quantity (del_num) to the sum of the first statistical quantity (key_num) and the second statistical quantity can be calculated. Then, this ratio is compared to a preset threshold. If the ratio is greater than the preset threshold, it indicates that the proportion of deleted keys within the currently traversed key-value range is relatively high, satisfying the preset compression conditions.
[0056] In some examples, a pre-defined linked list (i.e., a compression task queue) can be set, and the threshold t can be set to the range (0, 1). When the ratio of (del_num / (key_num+del_num)) is greater than the threshold t, it means that the number of keys deleted within the key-value interval from start_key (the start position of the traversal) to end_key (the end position of the traversal) has reached the set threshold t, that is, the key-value interval corresponding to this traversal process meets the preset compression conditions.
[0057] Then, the tuple (start_key, end_key) can be appended to the end of the pre-set linked list, and another thread can periodically take the tuple from the head of the linked list for compression.
[0058] Step 103: When the traversal key-value interval corresponding to the current traversal process meets the preset compression conditions, compress the traversal key-value interval corresponding to the current traversal process.
[0059] In practical applications, when the key-value range corresponding to the current traversal process meets the preset compression conditions, a compression task can be created to compress the key-value range corresponding to the current traversal process.
[0060] For example, when the proportion of the second statistical quantity in a certain traversed key value range (such as from start_key to end_key) exceeds a preset threshold t, a compression task can be generated. This compression task can carry the traversed key value range (i.e., from start_key to end_key) that needs to be compressed.
[0061] In the above embodiments, by counting the second statistical quantity during the traversal operation, the compression action of a specific traversal key value range can be automatically triggered.
[0062] In some embodiments of the present invention, when the traversal key value range corresponding to the current traversal process meets a preset compression condition, the traversal key value range corresponding to the current traversal process is compressed, including: When the traversal key-value interval corresponding to the current traversal process meets the preset compression conditions, an identifier of the traversal key-value interval corresponding to the current traversal process is added to the preset linked list; when performing the compression task, the traversal key-value interval recorded in the preset linked list is compressed.
[0063] In some embodiments of the present invention, the identifier of the traversed key value range includes the traversal start key value and the traversal end key value of the traversed key value range.
[0064] As examples, the identifier for traversing a key-value range can be a set of data that can uniquely identify the key-value range, such as the starting key being start_key and the ending key being end_key.
[0065] Once a key-value range that meets the compression conditions is detected, its identifier (start_key, end_key) is added to a pre-defined linked list, indicating that the key-value range needs to be compressed. The pre-defined linked list can be considered a compression task queue, recording all key-value ranges that require compression.
[0066] In some embodiments of the present invention, the compression task is executed periodically by creating a thread.
[0067] In some examples, a thread can be created to periodically execute compression tasks. When executing a compression task, the key-value range to be compressed, such as (start_key', end_key'), can be retrieved from a pre-defined linked list, and then the key-value pairs within the range (start_key', end_key') can be compressed.
[0068] In some embodiments of the present invention, when performing a compression task, the compression of the traversed key-value range recorded in the preset linked list includes: When performing a compression task, the target compression key value range is determined from the traversed key value range recorded in the preset linked list, and the target compression key value range is compressed.
[0069] In practical applications, when performing compression tasks, a key-value range can be selected as the target compression key-value range from the key-value ranges recorded in the pre-set linked list, either sequentially or according to a priority strategy.
[0070] For example, the key-value range can be selected for compression sequentially according to the record order of the key-value range in the preset linked list; or the target key-value range can be determined according to priority rules, such as prioritizing the compression of key-value ranges containing more deleted key-values.
[0071] Once the target compression key value range is determined, compression operations can be performed on the key value pairs within that range.
[0072] In some embodiments of the present invention, when performing a compression task, determining the target compression key value range from the traversed key value range recorded in the preset linked list includes: Sub-step 11: When performing a compression task, determine the current compression key value range.
[0073] In some embodiments of the present invention, the method further includes: when the current compressed key value range is an initial value, obtaining the head traversal key value range located at the head of the preset linked list, and updating the current compressed key value range to the head traversal key value range; wherein the initial value of the current compressed key value range is empty.
[0074] In some examples, since a large number of small-range traversals will generate many traversal key-value ranges, the traversal key-value ranges can be merged to improve compression efficiency.
[0075] In practical applications, the current compressed key-value range can be used as the starting point for merging compressed ranges. The current compressed key-value range can be a range with an initial value of zero (i.e., the starting and ending key values during traversal are both 0) as the base range for merging. For example, the current compressed key-value range is key 0 - key 0 (i.e., 0 - 0). When the current compressed key-value range is the initial value (i.e., 0-0), the head traversal key-value range can be obtained from the head of the preset linked list, and the current compressed key-value range can be updated to the head traversal key-value range.
[0076] For example, if the key value range of the head traversal of the preset linked list is key A-key D, then the current compressed key value range will be updated from the initially empty key 0-key 0 to key A-key D.
[0077] In the above embodiments, by merging key value ranges to reduce the number of compression tasks, the additional overhead of write amplification and compression can be reduced.
[0078] Sub-step 12: Determine whether the current compressed key value interval and the head traversal key value interval located at the head of the preset linked list have an intersecting key value interval. If the current compressed key value interval and the head traversal key value interval have an intersecting key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval.
[0079] As examples, the head of the pre-set list refers to the position of the first traversed key-value range recorded in the pre-set list. The head traversed key-value range refers to the first traversed key-value range recorded at the head position of the pre-set list.
[0080] After determining the current compressed key value range, the head traversal key value range can be extracted from the head of the preset linked list. It can then be determined whether the current compressed key value range and the head traversal key value range intersect. If they do intersect, it means that the two key value ranges have an overlap.
[0081] Then, the union of the current compressed key-value range and the header traversal key-value range can be calculated; and the range of the current compressed key-value range can be updated to the range of this union key-value range to achieve the merging of key-value ranges. The union key-value range can include all key-values in both the current compressed key-value range and the header traversal key-value range.
[0082] For example, the current compressed key-value interval is from key A to key D (i.e., AD), and the head traversal key-value interval of the preset linked list head is from key C to key F (i.e., CF). First, it is determined whether there is an intersecting key-value interval. For example, the overlapping part of AD and CF is CD, so there is an intersecting key-value interval. Then, the union key-value interval is calculated, that is, the range covered by the merged union key-value interval is from A to F (AF). Finally, the current compressed key-value interval is updated from key A to key D (i.e., AD) to key A to key F (i.e., AF), completing the merging of the two intervals.
[0083] Sub-step 13: Delete the head traversal key value interval and use a new head traversal key value interval. Perform a judgment to determine whether the current compressed key value interval and the head traversal key value interval located at the head of the preset linked list have an intersecting key value interval. If the current compressed key value interval and the head traversal key value interval have an intersecting key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval.
[0084] In some examples, after determining the union of the current compressed key value range and the head traversal key value range, and updating the current compressed key value range to the union key value range, the merged head traversal key value range can be deleted.
[0085] In practical applications, after the head traversal key-value range of the pre-defined linked list head is deleted, the traversal key-value ranges below it will move up in sequence to become the new head traversal key-value ranges.
[0086] After deleting the header traversal key value range, it can be determined whether the updated current compressed key value range and the new header traversal key value range intersect. If there is an intersecting key value range, it means that the two key value ranges have an overlap.
[0087] Then, the union of the updated current compressed key value range and the new head traversal key value range can be calculated; and the range of the updated current compressed key value range can be updated again to the range of this new union key value range.
[0088] For example, after the previous merge is completed, the current compressed key value range becomes key A to key F (AF). At this time, the new head traversal key value range becomes key E to key H (EH), which were originally located below the original head traversal key value range in the preset linked list.
[0089] By determining that AF and EH have an intersecting key value interval, namely EF, we can calculate their union key value interval to obtain AH, and update the current compressed key value interval from AF to AH.
[0090] In some examples, the above operation can be repeated until the current compressed key-value range and the head traversed key-value range do not intersect or the preset linked list is empty, thus completing the merging process of the traversed key-value ranges in the preset linked list. This improves compression efficiency, reduces unnecessary compression overhead, and reduces the impact of write amplification on database performance.
[0091] Sub-step 14: When there is no record in the preset linked list, or when there is no intersecting key value interval between the current compressed key value interval and the head traversal key value interval, the current compressed key value interval is determined as the target compressed key value interval.
[0092] As examples, "no record in the preset linked list" means that there are no other compressed key-value ranges in the preset linked list besides the current compressed key-value range. In this case, the current compressed key-value range can be used as the target compressed key-value range, and the target compressed key-value range can be compressed.
[0093] In practical applications, when it is determined that there is no intersecting key value interval between the current compressed key value interval and the head traversal key value interval, it means that there is no overlap between the current compressed key value interval and the head traversal key value interval in the preset linked list, and no further merging operation is required.
[0094] At this point, the current compressed key value range can be directly determined as the target compressed key value range, and then compressed.
[0095] For example, if the current compressed key value range is from key I to key K (IK), and the head traversal key value range of the preset linked list head is from key M to key P (MP), and there is no intersecting key value range between the two, then IK can be used as the target compressed key value range, and the compression process can be prepared.
[0096] In some examples, while compressing the target compressed key value range (IK), the head traversal key value range (MP) can be used as the current compressed key value range and deleted from the preset linked list. Then, the new head traversal key value range is used again. It is determined whether the current compressed key value range intersects with the head traversal key value range located at the head of the preset linked list. If the current compressed key value range intersects with the head traversal key value range, the union of the current compressed key value range and the head traversal key value range is determined, and the current compressed key value range is updated to the union key value range.
[0097] As examples, since numerous small-range traversals generate many key-value intervals, to improve efficiency, the key-value intervals in the pre-defined linked list can be merged before compression. This improves the compression efficiency of a single operation, avoids the additional overhead caused by compressing multiple small ranges, and reduces write amplification. The specific steps are as follows: S1. Assume the current compressed key value range is (start_key', end_key'), where the initial values of start_key' (the starting key value of the current traversal) and end_key' (the ending key value of the current traversal) are empty.
[0098] S2. Obtain the head of the compaction queue (pre-defined linked list) and traverse the key-value range 'iter_tuple'. If the queue is empty, jump to S5.
[0099] S3. Obtain the two tuples (i.e., the starting key and the ending key of the head traversal) in iter_tuple, and denote them as start_key and end_key.
[0100] S4. If start_key' and end_key' are empty, or (start_key', end_key') intersects with (start_key'', end_key''), then assign (start_key', end_key') the union of (start_key', end_key') and (start_key'', end_key''), delete the record of iter_tuple'' at the head of the compaction_queue, and jump to S2 to continue merging. If (start_key', end_key') does not intersect with (start_key'', end_key''), then jump to S5.
[0101] S5. End. At this point, (start_key', end_key') will be used as the target compressed key range for compression processing.
[0102] In the above embodiments, by merging intersecting key value ranges, multiple smaller compressions are replaced with a single, slightly larger compression task, thereby significantly reducing the number of compressions and overall system overhead, and reducing write amplification.
[0103] In this embodiment of the invention, during the traversal of the database, a first statistical count of valid key values and a second statistical count of deleted key values are recorded; wherein deleted key values are marked with tombstones; based on the first and second statistical counts, it is determined whether the traversed key value interval corresponding to the current traversal process meets a preset compression condition; when the traversed key value interval corresponding to the current traversal process meets the preset compression condition, the traversed key value interval corresponding to the current traversal process is compressed. This achieves compression of the traversed key value interval based on the statistical count of valid key values and the statistical count of deleted key values marked with tombstones, thereby reducing the number of deleted key values in the database, improving the database read performance, reducing the underlying traversal overhead, avoiding read amplification problems, thereby reducing the system load, improving the overall performance of the database, and enhancing the availability of the database.
[0104] Reference Figure 2 The diagram illustrates a flowchart of another database-based compression method provided by some embodiments of the present invention, which may specifically include the following steps: Step 201: In response to the deletion operation of the key value in the database, set a tombstone mark for the deleted key value.
[0105] Step 202: During the traversal of the database, record the first statistical count of valid key values and the second statistical count of deleted key values; wherein the deleted key values are marked with tombstones.
[0106] Step 203: Based on the first statistical quantity and the second statistical quantity, determine whether the traversal key value range corresponding to the current traversal process meets the preset compression conditions.
[0107] Step 204: When the traversal key-value interval corresponding to the current traversal process meets the preset compression conditions, add the identifier of the traversal key-value interval corresponding to the current traversal process to the preset linked list.
[0108] Step 205: When performing the compression task, the traversal key value range recorded in the preset linked list is compressed.
[0109] In this embodiment of the invention, in response to a deletion operation on a key value in the database, a tombstone marker is set for the deleted key value. During the traversal of the database, a first statistical count of valid key values and a second statistical count of deleted key values are recorded. Deleted key values are marked with a tombstone. Based on the first and second statistical counts, it is determined whether the traversed key value interval corresponding to the current traversal process meets a preset compression condition. When the traversed key value interval corresponding to the current traversal process meets the preset compression condition, an identifier for the traversed key value interval corresponding to the current traversal process is added to a preset linked list. During the compression task, the traversed key value interval recorded in the preset linked list is compressed. This achieves compression based on the statistical count of valid key values and the statistical count of deleted key values marked with tombstones. By determining that the traversed key value interval corresponding to the current traversal process meets the preset compression condition, the traversed key value interval is compressed, thereby reducing the number of deleted key values in the database, improving database read performance, reducing underlying traversal overhead, avoiding read amplification problems, thus reducing system load, improving overall database performance, and enhancing database availability.
[0110] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0111] Reference Figure 3 The diagram illustrates a database-based compression processing apparatus according to some embodiments of the present invention, which may specifically include the following modules: The quantity statistics module 301 is used to record the first statistical count of valid key values and the second statistical count of deleted key values during the traversal of the database; wherein the deleted key values are marked with tombstones. Compression condition judgment module 302 is used to determine whether the traversal key value range corresponding to the current traversal process meets the preset compression condition based on the first statistical quantity and the second statistical quantity. Compression module 303 is used to compress the traversal key value range corresponding to the current traversal process when the preset compression conditions are met.
[0112] In some embodiments of the present invention, the compression module 303 includes: The identifier addition submodule is used to add an identifier for the traversal key-value range corresponding to the current traversal process to the preset linked list when the traversal key-value range corresponding to the current traversal process meets the preset compression conditions. The key-value range compression submodule is used to compress the key-value ranges recorded in the preset linked list when performing compression tasks.
[0113] In some embodiments of the present invention, the key-value range compression submodule includes: The target compression key value interval compression unit is used to determine the target compression key value interval from the traversed key value intervals recorded in the preset linked list when performing a compression task, and to compress the target compression key value interval.
[0114] In some embodiments of the present invention, the target compression key value range compression unit is used for: When performing a compression task, determine the current range of compression key values; Determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. Delete the head traversal key value interval and adopt a new head traversal key value interval. Perform a judgment to determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. If there is no record in the preset linked list, or if the current compressed key value interval does not intersect with the head traversal key value interval, the current compressed key value interval is determined as the target compressed key value interval.
[0115] In some embodiments of the present invention, the apparatus further includes: The key-value range update module is used to obtain the head traversal key-value range located at the head of the preset linked list when the current compressed key-value range is the initial value, and update the current compressed key-value range to the head traversal key-value range; wherein, the initial value of the current compressed key-value range is empty.
[0116] In some embodiments of the present invention, the identifier of the traversed key value range includes the traversal start key value and the traversal end key value of the traversed key value range.
[0117] In some embodiments of the present invention, the compression task is executed periodically by creating a thread.
[0118] In some embodiments of the present invention, the compression condition determination module 302 includes: The ratio determination submodule is used to determine the ratio of the second statistical quantity to the sum of the first statistical quantity and the second statistical quantity; The ratio determination submodule is used to determine whether the ratio is greater than a threshold. The compression condition determination submodule is used to determine that if the ratio is greater than a threshold, the traversal key value interval corresponding to the current traversal process meets the preset compression condition.
[0119] In some embodiments of the present invention, the apparatus further includes: The tombstone marker setting module is used to set a tombstone marker for deleted key values in response to a deletion operation on the key values in the database.
[0120] In some embodiments of the present invention, the quantity statistics module 301 includes: The statistics quantity determination submodule is used to determine the first statistical quantity of valid key values and the second statistical quantity of deleted key values during the traversal of the database, based on the tombstone marker settings of the key values.
[0121] In some embodiments of the present invention, the database uses a hierarchical structure to store data, and the database has multiple hierarchical files.
[0122] Some embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.
[0123] Some embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the method described above.
[0124] Some embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0125] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0128] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0129] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0132] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0133] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.
[0134] The database-based compression processing method, apparatus, device, medium, and product provided above have been described in detail. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A database-based compression processing method, characterized in that, The method includes: During the traversal of the database, the first count of valid key values and the second count of deleted key values are recorded; wherein the deleted key values are marked with a tombstone. Based on the first statistical count and the second statistical count, determine whether the traversal key value range corresponding to the current traversal process meets the preset compression conditions; When the key-value range corresponding to the current traversal process meets the preset compression conditions, the key-value range corresponding to the current traversal process is compressed.
2. The method according to claim 1, characterized in that, When the key-value range corresponding to the current traversal process meets the preset compression conditions, the key-value range corresponding to the current traversal process is compressed, including: When the key-value interval corresponding to the current traversal process meets the preset compression conditions, add the identifier of the key-value interval corresponding to the current traversal process to the preset linked list. When performing the compression task, the traversal key value range recorded in the preset linked list is compressed.
3. The method according to claim 2, characterized in that, When performing a compression task, the key-value ranges recorded in the preset linked list are compressed, including: When performing a compression task, the target compression key value range is determined from the traversed key value range recorded in the preset linked list, and the target compression key value range is compressed.
4. The method according to claim 3, characterized in that, When performing a compression task, the target compression key value range is determined from the traversed key value ranges recorded in the preset linked list, including: When performing a compression task, determine the current range of compression key values; Determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. Delete the head traversal key value interval and adopt a new head traversal key value interval. Perform a judgment to determine whether the current compressed key value interval intersects with the head traversal key value interval located at the head of the preset linked list. If the current compressed key value interval intersects with the head traversal key value interval, determine the union key value interval of the current compressed key value interval and the head traversal key value interval, and update the current compressed key value interval to the union key value interval. If there is no record in the preset linked list, or if the current compressed key value interval does not intersect with the head traversal key value interval, the current compressed key value interval is determined as the target compressed key value interval.
5. The method according to claim 4, characterized in that, Also includes: When the current compressed key value range is the initial value, obtain the head traversal key value range located at the head of the preset linked list, and update the current compressed key value range to the head traversal key value range; wherein, the initial value of the current compressed key value range is empty.
6. The method according to claim 2, characterized in that, The identifier of the traversal key value range includes the traversal start key value and the traversal end key value of the traversal key value range.
7. The method according to claim 2, characterized in that, The compression task is executed periodically by creating a thread.
8. The method according to any one of claims 1-7, characterized in that, Based on the first statistical count and the second statistical count, determine whether the traversal key-value range corresponding to the current traversal process meets the preset compression conditions, including: Determine the ratio of the second statistical quantity to the sum of the first and second statistical quantities; Determine whether the ratio is greater than a threshold; If the ratio is greater than the threshold, it is determined that the traversal key value range corresponding to the current traversal process meets the preset compression conditions.
9. The method according to any one of claims 1-7, characterized in that, Before recording the first count of valid key values and the second count of deleted key values during the traversal of the database, the process also includes: In response to a deletion operation on a key value in the database, a tombstone marker is set for the deleted key value.
10. The method according to claim 9, characterized in that, During the traversal of the database, the first statistical count of valid key values encountered and the second statistical count of deleted key values are recorded, including: During the traversal of the database, the first statistical count of valid key values and the second statistical count of deleted key values are determined based on the tombstone marker settings of the key values.
11. The method according to any one of claims 1-7, characterized in that, The database uses a hierarchical structure to store data, and the database has multiple hierarchical files.
12. A database-based compression processing device, characterized in that, The device includes: The quantity statistics module is used to record the first statistical count of valid key values and the second statistical count of deleted key values during the traversal of the database; wherein the deleted key values are marked with a tombstone. The compression condition judgment module is used to determine whether the traversal key value range corresponding to the current traversal process meets the preset compression condition based on the first statistical quantity and the second statistical quantity. The compression module is used to compress the traversal key-value range corresponding to the current traversal process when the preset compression conditions are met.
13. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1 to 11.
15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.