Metadata management system and method for storage system

By using a two-level log engine and a fused metadata management module, the read/write amplification problem in log-based metadata management systems is solved, improving system performance and operational efficiency, and enabling collaborative management of metadata and log data.

CN117873965BActive Publication Date: 2024-11-22CHONGQING DAODAO YUNXING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410141971.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-01
Publication Date
2024-11-22
Estimated Expiration
2044-02-01

AI Technical Summary

Technical Problem

Existing log-based metadata management systems suffer from read/write amplification issues, leading to system performance bottlenecks and increased operational complexity, making them unable to adapt to various workloads and requirements.

Method used

It adopts a two-level log engine and a merged metadata and log data management module. The log engine module handles the persistent lifecycle of log data, the metadata management module handles the persistent lifecycle of metadata, and the storage system adaptation module provides a unified metadata read and write interface, supporting asynchronous applications and atomicity and sequentiality. The dirty count mechanism is used to implement the aging mechanism of metadata dirty cache.

Benefits of technology

It improved system performance, reduced read/write amplification issues, simplified system architecture, improved IO performance and operational efficiency, and enabled collaborative management of metadata and log data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117873965B_ABST
    Figure CN117873965B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of storage technology, and specifically discloses a metadata management system and method for a storage system, and other systems include: a log engine module, which is used to process the persistent life cycle of log data; it is also used to support the submission, application and playback process of log data; a metadata management module, which is used to process the persistent life cycle of metadata; it is also used to provide commit_done, apply_done and replay_done interfaces of log data, which are respectively used to identify the processing flow after submission, application and playback of log data; a storage system adapter module, which is used to provide a metadata read-write interface in a storage system; it is also used to aggregate and group log data according to corresponding metadata address spaces, and apply the grouped primary log data to a secondary log engine to form secondary log data through a primary log engine. The technical solution of the present invention can improve system performance and operation and maintenance efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a metadata management system and method for a storage system. Background Art

[0002] Metadata refers to data used to describe and manage data in a storage system. This information includes file attributes, location, permissions, etc., and is critical to the reliability and performance of the storage system.

[0003] Based on SSD hardware, the mainstream is to use log-based software storage systems, which use logs to record key operations to ensure data consistency in the event of system failure or restart.

[0004] Among them, log-based metadata storage systems usually adopt a first-level log structure. This means that there is a main log layer for recording key metadata operations, and each module independently implements a first-level log, which leads to serious read-write amplification problems in log applications. Read-write amplification means that the actual read and write operations on the disk are far more than the requests at the application layer, which will lead to system performance bottlenecks and problems in versatility, and may not be able to adapt to various workloads and requirements. These problems have increased the difficulty of system operation and maintenance.

[0005] Therefore, there is a need for a metadata management system and method for a storage system that can improve performance and operation and maintenance efficiency. Summary of the invention

[0006] One of the objectives of the present invention is to provide a metadata management system for a storage system, which can improve system performance and operation and maintenance efficiency.

[0007] In order to solve the above technical problems, this application provides the following technical solutions:

[0008] Metadata management system for storage systems, including:

[0009] The log engine module is used to handle the persistence lifecycle of log data; it is also used to support the submission, application, and playback processes of log data;

[0010] The metadata management module is used to handle the persistence lifecycle of metadata. It is also used to provide the commit_done, apply_done, and replay_done interfaces of log data, which are used to identify the processing flow after committing, applying, and replaying log data respectively.

[0011] The storage system adapter module is used to provide a metadata read and write interface in the storage system; it is also used to aggregate and group log data according to the corresponding metadata address space, and apply the grouped first-level log data to the second-level log engine through the first-level log engine to form second-level log data.

[0012] Furthermore, the storage system adapter module is also used to support asynchronous application of primary log data and secondary log data, as well as atomicity and sequentiality during playback.

[0013] Furthermore, the metadata management module is also used to implement an aging mechanism of the metadata dirtycache using a dirty count mechanism.

[0014] Furthermore, the log engine module is also used in the submission process to submit log data, convert the metadata format into log data, persist it to the disk space corresponding to the log data, and sequentially update the corresponding metadata dirtycache;

[0015] It is also used to apply log data in the application process, convert the log data format into metadata, and persist it to the disk space corresponding to the metadata;

[0016] It is also used to replay log data in the playback process, load log data from the corresponding disk space into memory, convert the log data format into metadata, and update the corresponding metadata dirty cache in reverse order.

[0017] Furthermore, the metadata management module is also used to sequentially update the corresponding metadata dirty cache and increase the dirty count after calling the Commit_done interface;

[0018] It is also used to reduce the dirty count after calling the Apply_done interface;

[0019] It is also used to update the corresponding metadata dirty cache in reverse order and increase the dirty count after calling the Replay_done interface.

[0020] It is also used to determine whether the dirty count of the metadata dirty cache is 0. If it is 0, the metadata dirty cache aging operation is performed.

[0021] A second object of the present invention is to provide a metadata management method for a storage system, comprising the following contents:

[0022] Metadata writing steps:

[0023] S101, submitting primary log data in IO time order;

[0024] S102, updating the dirty cache of metadata;

[0025] S103, IO returns;

[0026] S104, applying the primary log, submitting the primary log data to the corresponding secondary log engine according to the corresponding metadata address space through the primary log engine to form secondary log data;

[0027] S105 , apply the secondary log data, convert the secondary log data format into metadata, and persist it in the metadata address space.

[0028] Furthermore, the metadata reading operation steps are also included:

[0029] S201, read the dirty cache of the corresponding metadata, if hit, IO returns, otherwise jump to S202;

[0030] S202, replay the first-level log data and read the corresponding metadata dirty cache. If a hit is found, the IO returns, otherwise jump to S203;

[0031] S203, replay the secondary log data and read the corresponding metadata dirty cache. If a hit is found, the IO returns, otherwise jump to S204;

[0032] S204: Load metadata from the corresponding metadata space, and IO returns.

[0033] Furthermore, in the steps S104 and S105, the first-level log data and the second-level log data of the asynchronous application are respectively generated.

[0034] Beneficial effects: 1. Compared with non-log-based metadata management systems, the log-based metadata management system of this solution can convert the metadata update operation from a "read-modify-write" operation to a "write" operation, thereby improving IO performance;

[0035] 2. Compared with the metadata management system that uses a one-level log, this solution uses a two-level log, which aggregates and groups the chronological log data according to the address space order, which can reduce the read and write amplification when using logs.

[0036] 3. Compared with implementing metadata and log data management systems separately, integrating metadata and log data management into one processing module provides a unified metadata read and write interface, simplifies the system architecture and management, and can be more conveniently integrated into various metadata processing flows of the storage system, reducing the complexity of the system and making the management of metadata and log data more coordinated and consistent.

[0037] In summary, this solution achieves the above advantages by adopting log-based metadata management, a two-level log engine, and integrating metadata and log data management modules, thereby improving system performance, reducing read-write amplification problems, and simplifying the system architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of metadata writing operation steps in a metadata management method for a storage system according to an embodiment;

[0039] Figure 2 The schematic diagram is a metadata reading operation step in a metadata management method for a storage system according to an embodiment. DETAILED DESCRIPTION

[0040] The following is further described in detail through specific implementation methods:

[0041] Example

[0042] The metadata management system for a storage system of this embodiment is applied to a log storage system and includes a log engine module, a metadata management module and a storage system adapter module.

[0043] The log engine module is used to handle the persistent lifecycle of log data; the persistent lifecycle refers to retaining data on long-term storage media to ensure that the data can still be recovered and accessed even in the event of a system restart or power outage.

[0044] It is also used to support the commit, apply, and replay processes of log data.

[0045] The Commit process is used to submit log data, convert the metadata format into the log data format, persist it to the disk space corresponding to the log data, and sequentially update the corresponding metadata dirty cache.

[0046] In the Apply process, it is used to apply log data, convert the log data format into metadata format, and persist it to the disk space corresponding to the metadata.

[0047] The Replay process is used to replay log data, load log data from the corresponding disk space into memory, convert the log data format to metadata format, and update the corresponding metadata dirty cache in reverse order.

[0048] The metadata management module is used to handle the persistence lifecycle of metadata. It is also used to provide the commit_done, apply_done, and replay_done interfaces of log data, which are used to identify the processing flow after committing, applying, and replaying log data.

[0049] It is also used to implement the aging mechanism of metadata dirty cache using the dirty count mechanism. In this embodiment, Dirty Cache refers to the state where the cache contains data that has been modified but has not yet been written back to permanent storage;

[0050] Dirty Count is a counter that records the number of dirty data in the system. When data is modified, Dirty Count increases. When these modifications are successfully submitted or applied to permanent storage, Dirty Count decreases.

[0051] Specifically, it is used to sequentially update the corresponding metadata dirty cache and increase the dirty count after calling the Commit_done interface, that is, after committing the log data.

[0052] It is also used to reduce the dirty count after calling the Apply_done interface, that is, after applying the log data.

[0053] It is also used to update the corresponding metadata dirty cache in reverse order and increase the dirty count after calling the Replay_done interface, that is, after replaying the log data.

[0054] It is also used to determine whether the dirty count of the metadata dirty cache is 0. If it is 0, perform metadata dirty cache aging operations, such as cleaning the dirty cache.

[0055] The storage system adapter module is used to provide a metadata read and write interface in the storage system;

[0056] It is also used to aggregate and group log data according to the corresponding metadata address space, and apply the grouped first-level log data to the second-level log engine through the first-level log engine to form second-level log data, ensuring that these log data are correctly applied to the storage system in the order of the address space.

[0057] It is also used to support asynchronous application of primary and secondary log data, as well as atomicity and sequentiality during playback.

[0058] Specifically, the secondary log groups logs according to the metadata space address, and each group guarantees the atomicity and order of the log. The benefits are: 1. When each secondary log engine applies the log, the disk data for load and flush becomes less, which can reduce read and write amplification. 2. Each secondary log engine applies the log in groups concurrently, which can improve the overall application speed of the secondary log.

[0059] In this embodiment, atomicity means that the application and playback operations of each log data are indivisible units, ensuring that the system will not partially apply or playback when processing logs, thereby maintaining data consistency. Sequentiality means that the application and playback of log data must be performed in the order in which they occur, ensuring the correct evolution of the system state. Even if the submission and processing of log data are asynchronous, during the application and playback process, the system will maintain the consistency of the log data in chronological order to prevent confusion in data operations.

[0060] This embodiment also provides a metadata management method for a storage system, including the following contents:

[0061] like Figure 1 As shown, the metadata writing operation steps are:

[0062] S101. Commit the first-level log data in IO (input / output) time order.

[0063] S102: Update the dirty cache of metadata.

[0064] S103, IO returns, that is, the input / output operation is completed;

[0065] S104, asynchronously apply the first-level log, and commit the first-level log data to the corresponding second-level log engine according to the corresponding metadata address space through the first-level log engine to form the second-level log data;

[0066] S105 , asynchronously apply the secondary log data, convert the secondary log data format into metadata, and persist it in the metadata address space.

[0067] like Figure 2 As shown, the metadata read operation steps are:

[0068] S201, read the corresponding metadata dirty cache, if hit, IO returns, otherwise jump to S202;

[0069] S202, replay the first-level log and read the corresponding metadata dirty cache. If a match is found, the IO returns. Otherwise, jump to S203.

[0070] S203, replay the secondary log and read the corresponding metadata dirty cache. If a match is found, the IO returns. Otherwise, jump to S204.

[0071] S204: Load metadata from the corresponding metadata space, and IO returns.

[0072] Taking SSD hardware as an example, the specific process of this method is introduced. Metadata writing steps:

[0073] S101, when metadata (such as file information) needs to be modified, first submit these modifications to the primary log data in the order of IO time;

[0074] S102, updating the metadata dirty cache to record which metadata has been modified, that is, which data has not been written back to permanent storage;

[0075] S103, the SSD performs the IO operation, that is, completes the modification of the metadata, and returns the result of the operation;

[0076] S104: The first-level log data records the changes to the metadata. Asynchronously, these changes will be applied to the second-level log engine to form the second-level log data, ensuring that these changes are correctly saved.

[0077] S105. Asynchronously, the secondary log data will be processed and converted into the actual metadata format, and then persisted to the metadata address space, that is, the location where the metadata is stored in the SSD.

[0078] Metadata read operation steps:

[0079] S201. When a certain metadata needs to be read, the metadata dirty cache is checked first. If it is found that the metadata has been modified but has not been written back to the permanent storage, the data in the dirty cache is directly returned.

[0080] S202: If the required metadata is not found in the dirty cache, replay the first-level log data to check whether there is any modification record about the metadata. If there is, update the dirty cache according to the modified content and return the corresponding data;

[0081] S203, if it is not found in the primary log data, replay the secondary log data to check whether there is a modification record about this metadata. If so, update the dirty cache according to the modified content and return the corresponding data.

[0082] S204: If the required data is not found in the primary log data and the secondary log data, load the metadata from the persistent metadata address space and return.

[0083] The advantages of this solution are as follows:

[0084] 1. Compared with non-log-based metadata management systems, log-based metadata management systems can convert metadata update operations from a "read, modify and write" operation to a "write" operation, thereby improving IO performance. In other words, the system avoids the overhead of directly reading and modifying metadata by recording modified logs, thereby improving IO performance.

[0085] 2. Compared with the metadata management system that uses a first-level log, this solution uses a two-level log engine. The second-level log engine aggregates and groups the time-sequential log data in the order of the address space. This can reduce the read-write amplification when applying the log. That is, when applying the log data, it can be more effectively aggregated and processed in the order of the address space, thereby improving system performance.

[0086] 3. Compared with implementing metadata and log data management systems separately, integrating metadata and log data management into one processing module provides a unified metadata read and write interface, simplifies the system architecture and management, and can be more conveniently integrated into various metadata processing flows of the storage system, reducing the complexity of the system and making the management of metadata and log data more coordinated and consistent.

[0087] The above are only embodiments of the present invention. The invention is not limited to the fields involved in this implementation case. The common sense such as the known specific structures and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to explain the content of the claims.

Claims

1. Metadata management system for storage systems, It is characterized in that include: The log engine module is used to handle the persistent life cycle of log data; It is also used to support the submission, application, and playback processes of log data; The metadata management module is used to handle the persistence lifecycle of metadata. It is also used to provide the commit_done, apply_done, and replay_done interfaces of log data, which are used to identify the processing flow after committing, applying, and replaying log data respectively. The storage system adapter module is used to provide a metadata read and write interface in the storage system; It is also used to aggregate and group the log data according to the corresponding metadata address space, and apply the grouped first-level log data to the second-level log engine through the first-level log engine to form second-level log data; The storage system adapter module is also used to support asynchronous application of primary log data and secondary log data, as well as atomicity and sequentiality during playback; The metadata management module is also used to implement an aging mechanism of metadata dirty cache using a dirty count mechanism; The log engine module is also used in the submission process to submit log data, convert the metadata format into log data, persist it to the disk space corresponding to the log data, and sequentially update the corresponding metadata dirty cache; It is also used to apply log data in the application process, convert the log data format into metadata, and persist it to the disk space corresponding to the metadata; It is also used to replay log data in the playback process, load log data from the corresponding disk space into memory, convert the log data format into metadata, and update the corresponding metadata dirty cache in reverse order.

2. The metadata management system for a storage system according to claim 1, Features: The metadata management module is also used to sequentially update the corresponding metadata dirty cache and increase the dirty count after calling the Commit_done interface; It is also used to reduce the dirty count after calling the Apply_done interface; It is also used to update the corresponding metadata dirty cache in reverse order and increase the dirtycount after calling the Replay_done interface. It is also used to determine whether the dirty count of the metadata dirty cache is 0. If it is 0, the metadata dirty cache aging operation is performed.

3. A metadata management method for a storage system, using the system according to any one of claims 1 to 2, It is characterized in that It includes the following: Metadata writing steps: S101, submitting primary log data in IO time order; S102, updating the dirty cache of metadata; S103, IO returns; S104, applying the primary log, submitting the primary log data to the corresponding secondary log engine according to the corresponding metadata address space through the primary log engine to form secondary log data; S105 , apply the secondary log data, convert the secondary log data format into metadata, and persist it in the metadata address space.

4. The metadata management method for a storage system according to claim 3, Features: It also includes metadata read operation steps: S201, read the dirty cache of the corresponding metadata, if it hits, the IO returns, otherwise jump to S202; S202, replay the first-level log data and read the corresponding metadata dirty cache. If a hit occurs, the IO returns, otherwise jump to S203; S203, replay the secondary log data and read the corresponding metadata dirty cache. If a match is found, the IO returns. Otherwise, jump to S204. S204. Load metadata from the corresponding metadata space, and IO returns.

5. The metadata management method for a storage system according to claim 4, Features: In the steps S104 and S105, the first-level log and the second-level log data are applied asynchronously respectively.