Efficient log recovery method for key value storage based on key value separation
By introducing status bitmaps and address translation tables into the key-value storage system, the log recycling process is optimized, and the problem of low log recycling efficiency in the existing technology is solved, and more efficient data migration and storage system performance is achieved.
Patent Information
- Application Number
- CN202411821357.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-05-02
AI Technical Summary
The existing key-value separation technology is less efficient in the log recycling process, resulting in log recycling becoming a system bottleneck and affecting the performance of the storage system.
By introducing a status bitmap to mark the validity of values in the log, avoid frequent LSM-tree queries, and record logical addresses and physical addresses in the address translation table, update physical addresses only when necessary, and reduce unnecessary I/O operations.
It effectively reduces data migration and log recycling time, avoids log recycling becoming a new bottleneck in the storage system, and improves the performance of the storage system.
Smart Images
Figure CN119917465A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer networks and computer storage, and in particular to an efficient log recovery method for key-value storage based on key-value separation. Background Art
[0002] In recent years, with the development and widespread use of AI technology, the amount of data generated worldwide each year has exploded. The processing of massive data is highly dependent on the performance of the storage system, which places increasingly higher demands on the storage system. Among them, the key-value storage system uses the log structure merge tree (LSM-tree for short) as its storage engine to achieve efficient data writing. In addition, it also merges files by periodically triggering compression to provide efficient reading performance. Therefore, the key-value storage system based on LSM-tree has been widely used in the processing and storage of massive data.
[0003] Compression is a resource-intensive operation that consumes a lot of system resources such as I / O and CPU. A common method to reduce the overhead of compression operations and improve compression efficiency is key-value separation. The core idea is to store the keys and values in the key-value storage system in different data structures. Only the keys are inserted into the LSM-tree for compression, while the values are stored separately in the append-only log. The main advantage is that the value part of the key-value pair usually occupies a large storage space, while the value does not need to be frequently updated or involved in the lookup operation in most cases. In addition, it greatly reduces the overhead of compression because the values are not read and written repeatedly during the compression process.
[0004] Although key-value separation improves the efficiency of compression, it causes the bottleneck of the system to shift from compression to the recycling operation of the log. Since the values are stored in the log, compression cannot automatically recycle invalid values. Therefore, a recycling mechanism is needed to recycle the storage space occupied by invalid values in the log. However, the existing recycling mechanism is not cost-effective. During the recycling process, on the one hand, the log needs to be scanned sequentially and the LSM-tree needs to be randomly queried to check the validity of the value; on the other hand, the valid key-value pairs in the recycled log need to be rewritten; this mixed I / O mode is inefficient in the storage system, especially when the log is very large, the recycling overhead will increase rapidly. Eventually, log recycling will become a bottleneck, greatly reducing the efficiency of key-value separation and the performance of the storage system. Summary of the invention
[0005] In order to solve the problems existing in the prior art, the present invention provides an efficient log recovery method based on key-value storage with key-value separation, which solves the problems mentioned in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: an efficient log recovery method based on key-value storage with key-value separation, characterized in that it comprises the following steps:
[0007] S1, initialize the storage system;
[0008] S2. Record the logical address and physical address of the newly written key-value pair in the address translation table;
[0009] S3, determining whether an invalid key-value pair is generated during the compression process. If an invalid key-value pair is generated during the compression process, then executing step S4, otherwise no operation is required;
[0010] S4. Extract the invalid key-value pair index, obtain the "file number" and "offset" information of the invalid key-value pair, mark it as invalid at the corresponding position k of the corresponding bitmap, and increase the count C of the invalid value of the log;
[0011] S5. Determine whether the number C of invalid values in the log exceeds a threshold. If the number C of invalid values in the log exceeds a threshold N, the log recycling process will be triggered. When the data writing is completed, the entire log recycling process is completed.
[0012] Preferably, in step S1, it specifically includes the following: first, initialize the log, assign a unique file number to each log, and create a corresponding status bitmap for it; each bitmap item corresponds to a value in the log, where 0 represents a valid value and 1 represents an invalid value; second, set a logical address for each key, and record its physical address and logical address information in the address conversion table.
[0013] Preferably, in step S1, all mark bits of the newly created bitmap are 0.
[0014] Preferably, in step S4, according to The corresponding position k in the corresponding bitmap is marked as 1, that is, marked as invalid, F represents the offset of the invalid value in the log, and V represents the size of the invalid value.
[0015] Preferably, in step S5, the log recovery process includes the following:
[0016] S51, scanning bitmap: scanning the bitmap of the log and selecting all valid key-value pairs therein;
[0017] S52, migrate valid values: migrate the valid values to the new log, and update the physical addresses corresponding to these valid values in the address translation table;
[0018] S53. Recycling invalid values: The remaining invalid values are recycled together with the logs to release storage space.
[0019] The beneficial effects of the present invention are:
[0020] 1) Reduced data migration: The existing recycling mechanism is not cost-effective. During the recycling process, the log needs to be scanned sequentially. During the scanning process, if a valid key-value pair is encountered, it needs to be migrated to a new log. This sequential recycling scheme ultimately leads to the migration of a large amount of data. An efficient log recycling method is to recycle when the number of invalid key-value pairs in a log reaches a large value (i.e., the threshold), and recycle all the space of the entire log at one time. Therefore, this method can effectively reduce the amount of data migration.
[0021] 2) Reduced log recovery time: Existing recovery mechanisms are inefficient because they require random searches of the LSM-tree to check the validity of the key-value pairs. The LSM-tree has poor random query performance, which causes log recovery to become a system bottleneck and greatly increases the log recovery time. The proposed efficient log recovery method avoids random searches of the LSM-tree during log recovery. It only needs to sequentially scan the bitmaps corresponding to the logs to be recovered to obtain information about valid key-value pairs, and then migrate them, that is, only update the physical addresses in the address conversion table. The remaining invalid key-value pairs will be recovered together with the logs. Therefore, the log recovery method of the present invention effectively reduces the log recovery time. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is an architecture diagram of an efficient log recovery method based on key-value storage with key-value separation according to the present invention;
[0023] Figure 2 The present invention is a flow chart of an efficient log recovery method for key-value storage based on key-value separation. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] See also Figure 1-Figure 2 , the present invention provides a technical solution: an efficient log recovery method based on key-value storage with key-value separation, such as Figure 1As shown, first, by adopting multiple logs and limiting their size, the amount of data that needs to be processed during the recovery process can be effectively reduced, thereby reducing the I / O burden of the system. Then, a state bitmap is introduced to mark the validity of the values in the log, avoiding frequent LSM-tree queries, thereby reducing expensive random I / O operations during the recovery process. Secondly, an address translation table from logical addresses to physical addresses is designed. In the LSM-tree, only the logical addresses of the values in the log are stored. When valid data needs to be rewritten, only the changes in the physical address need to be modified in the address translation table without rewriting in the LSM-tree. Finally, this method effectively avoids the recovery of logs from becoming a new bottleneck of the storage system. The method includes an initialization process, an invalid value marking process, and a log recovery process.
[0026] Initialization process: First, a log needs to be initialized, and its maximum size is G MB. In addition, for each log, a corresponding status bitmap needs to be created for it. Each bitmap entry corresponds to a value in the log, where 0 represents a valid value and 1 represents an invalid value. Logs and bits Figure 1 One-to-one correspondence, common creation and destruction. It is worth noting that all the mark bits of the newly created bitmap are 0. Secondly, set a logical address for each key and record its physical address and logical address information in the address translation table.
[0027] Invalid value marking process: In LSM-tree, invalid key-value pairs are generated by the execution of compression operations. Through compression operations, invalid key-value pair indexes are extracted to obtain their "file number" and "offset" information. After obtaining this information, according to formula (1), the corresponding position k of the corresponding bitmap is set. Then, the corresponding mark bit is set to 1 at this position in the bitmap, indicating that the value is invalid. In addition, the system will also increase the invalid value count C of the log to monitor the number of invalid values so that the recycling operation can be triggered when the recycling threshold is reached.
[0028]
[0029] In the formula, F represents the offset of the invalid value in the log, and V represents the size of the invalid value.
[0030] Log recycling process: When the number of invalid values C exceeds the set threshold N, the system will start the log recycling operation. The specific steps of recycling are as follows:
[0031] 1) Scan the bitmap: Scan the status bitmap corresponding to the log to identify which values are valid (the bitmap is marked as 0);
[0032] 2) Migrate valid values: All valid values are migrated to a new log through a new write operation to ensure that the data will not be lost. At this time, only the physical address corresponding to the valid value (i.e., the new log file number and offset) needs to be updated in the address translation table;
[0033] 3) Recycling invalid values: After all valid values are migrated, the remaining invalid values (bitmap marked as 1) will be recycled along with the logs to release storage space.
[0034] The present invention provides an efficient log recovery method based on key-value storage with key-value separation, which specifically includes the following implementation steps: Figure 2 As shown:
[0035] 1) Initialize the storage system and set the maximum size of the log to G MB. Assign a unique file number to each log and create a corresponding status bitmap for it. All flag bits of the newly created bitmap are 0;
[0036] (2) Record the logical address and physical address of the newly written key-value pair in the address translation table;
[0037] (3) If an invalid key-value pair is generated during the compression process, then execute step (4). Otherwise, no operation is required;
[0038] (4) Obtain the "file number" and "offset" information of the invalid key-value pair;
[0039] (5) According to formula (1), the corresponding position k of the corresponding bitmap is marked as invalid (i.e., the corresponding position of the bitmap is marked as 1). At the same time, the count C of the invalid value of the log is also increased;
[0040] (6) If the number of invalid values C in the log exceeds the threshold N, the log recycling process will be triggered;
[0041] (7) Scan the bitmap of the log and select all valid key-value pairs in it;
[0042] (8) Migrate the valid value to the new log;
[0043] (9) Update the new physical addresses of these valid values in the address translation table;
[0044] (10) The remaining invalid values are recycled together with the log;
[0045] (11) If the data writing is completed, then the entire log recovery mechanism is completed. Otherwise, return to step (2) to continue loading data and determine whether there is a capacity limit;
[0046] At this point, all the implementation steps of the efficient log recovery method for key-value storage based on key-value separation described in the present invention are completed.
[0047] In the embodiments provided in the embodiments of the present invention, it should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0048] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0049] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0050] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0051] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.
[0052] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An efficient log recovery method based on key-value storage with key-value separation, characterized in that: The steps include: S1, initialize the storage system; S2. Record the logical address and physical address of the newly written key-value pair in the address translation table; S3, determining whether an invalid key-value pair is generated during the compression process. If an invalid key-value pair is generated during the compression process, then executing step S4, otherwise no operation is required; S4. Extract the invalid key-value pair index, obtain the "file number" and "offset" information of the invalid key-value pair, mark it as invalid at the corresponding position k of the corresponding bitmap, and increase the count C of the invalid value of the log; S5. Determine whether the number C of invalid values in the log exceeds a threshold. If the number C of invalid values in the log exceeds a threshold N, the log recycling process will be triggered. When the data writing is completed, the entire log recycling process is completed.
2. The efficient log recovery method based on key-value storage with key-value separation according to claim 1 is characterized in that: In step S1, it specifically includes the following: first, initialize the log, assign a unique file number to each log, and create a corresponding status bitmap for it; each bitmap item corresponds to a value in the log, where 0 represents a valid value and 1 represents an invalid value; second, set a logical address for each key, and record its physical address and logical address information in the address translation table.
3. The efficient log recovery method based on key-value storage with key-value separation according to claim 1 is characterized in that: In step S1, all flag bits of the newly created bitmap are 0.
4. The efficient log recovery method based on key-value storage with key-value separation according to claim 1 is characterized in that: In step S4, according to The corresponding position k in the corresponding bitmap is marked as 1, that is, marked as invalid, where F represents the offset of the invalid value in the log, and V represents the size of the invalid value.
5. The efficient log recovery method based on key-value storage with key-value separation according to claim 1 is characterized in that: In step S5, the log recovery process includes the following steps: S51, scanning bitmap: scanning the bitmap of the log and selecting all valid key-value pairs therein; S52, migrate valid values: migrate the valid values to the new log, and update the physical addresses corresponding to these valid values in the address translation table; S53. Recycling invalid values: The remaining invalid values are recycled together with the logs to release storage space.