Metadata indexing storage method and system based on solid-state drive local block storage engine

Abstract

This invention provides a metadata indexing and storage method and system based on a solid-state drive (SSD) local block storage engine. The method includes: setting up a write-ahead log in the SSD to persistently record recent metadata changes, with associated metadata entries residing in memory; constructing a baseline index in the SSD to store full-quantum data block metadata, with each small data block corresponding to an index root node, and metadata entries linearly mapped to fixed physical locations; when performing metadata queries: first searching the record cache, and if no match is found, concurrently searching the log structure index and the baseline index, retrieving the entry with the largest sequence number; and periodically performing maintenance tasks through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup. Through this innovative four-layer hybrid index architecture and background maintenance mechanism, high-performance metadata access and efficient storage space utilization and I / O performance optimization are achieved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a metadata indexing storage method and system based on a local block storage engine for solid-state drives. Background Technology

[0002] Distributed block storage systems distribute data across multiple independent servers, with each server acting as a storage node and running a local block storage engine. Older distributed block storage systems were often built on top of local file systems like Ext4, managing all metadata and data through the kernel file system, leading to performance bottlenecks and limited scalability. Modern distributed block storage systems generally implement dedicated user-space local block storage engines, ensuring high I / O performance as well as system scalability and flexibility.

[0003] Generally, distributed block storage provides virtual machine volumes for users to use. Users can mount virtual machine volumes to virtual machines or compute nodes via storage protocols (such as iSCSI). Distributed block storage systems typically use a two-layer mapping relationship to manage metadata: virtual volumes are divided into several small data blocks for management, each such as 64 MB / 256 MB. The distributed storage system is responsible for storing the status information of these small data blocks (such as the number of replicas, location, etc.); the local block storage engine usually further divides the small data blocks into several sub-blocks (64KB / 256KB) for more granular management of data distribution and storage efficiency.

[0004] Local block storage engines typically handle tens or even hundreds of gigabytes of metadata, necessitating persistent storage and efficient indexing algorithms for fast data access. Mainstream local block storage engines, such as Ceph BlueStore and Nutanix AES, use RocksDB (based on a log-structured merge tree) to manage all metadata. Some engines, like Ceph SeaStore, use B-trees (B+ trees and their variants) to index metadata, while others use persistent hash tables to store it.

[0005] Solid-state drives (SSDs) offer faster read and write speeds and lower latency compared to hard disk drives (HDDs), with particularly outstanding random read and write performance. They are especially suitable for handling high-frequency, small-block read and write operations. Therefore, modern distributed block storage systems commonly use SSDs to store all metadata. As the storage medium changes from HDDs to SSDs, the metadata index structure should also be optimized accordingly to fully utilize the high-performance characteristics of SSDs.

[0006] Further research revealed that most local storage engines opt for metadata indexing methods based on log-structured merge trees or B-trees. Different metadata indexing methods have their own advantages and disadvantages, and choosing the appropriate indexing method is crucial for improving system performance and scalability. (See the appendix for existing technologies.) Figure 1 as well as Figure 2 The metadata index structure shown in the image; typically, the index structure can choose one of two strategies to handle updates, namely in-place updates and out-of-place updates.

[0007] In-situ update structures, such as B-trees, directly overwrite old records to store the new update. For example, in... Figure 1 In this example, to update the value of keyk1 from v1 to v4, the index entry (k1, v1) is directly modified to apply the update. These structures are typically optimized for reads because they only store the latest version of each record. However, this design sacrifices write performance because updates result in random I / O and high write amplification. For example, updating a 128B record in an 8KB node requires reading and writing the entire node, resulting in a 64x write amplification.

[0008] In contrast, out-of-place update structures, such as log-structured merge trees, always store updated records in a new location rather than overwriting old entries. For example, in... Figure 2 In this approach, the updated record (k1, v4) is stored in a new location instead of directly updating the old entry (k1, v1). This design improves write performance because it can utilize sequential I / O to handle writes. It also simplifies the recovery process by not overwriting old data. However, the main problem with this design is the sacrifice of read performance, as records may be stored in any of the multiple locations. Furthermore, these structures typically require a separate data reorganization process to merge records in the background to maintain index efficiency and consistency, increasing system complexity and maintenance costs. If the background merging process cannot keep up with the foreground update speed, it can lead to index fragmentation, which in turn affects query efficiency.

[0009] Not only can metadata index updates be divided into in-place updates and append updates, but local block storage engines also face similar choices in handling data updates. In-place updates directly overwrite old data, simplifying storage management, but frequent random write operations can impact performance. Append updates, on the other hand, write new data to free areas, avoiding overwriting and transforming small random writes into large sequential writes, significantly increasing write throughput. However, append updates come at the cost of extremely high metadata update frequencies, placing enormous pressure on index management. For example, high-performance all-flash storage can typically achieve millions of I / Os per second for 4K random write performance, but in append update mode, every data update triggers a synchronous metadata update, meaning the metadata update frequency is also millions of times per second.

[0010] In this scenario, naive log structure merge trees (such as LevelDB) and B-tree implementations struggle to efficiently handle high-concurrency data read and write demands. Optimizing storage structure and balancing read and write performance become key challenges. This embodiment proposes a novel hybrid indexing mechanism that combines the advantages of in-place updates and append updates, specifically designed and optimized for local block storage engines, providing ultra-high metadata query and update performance. Summary of the Invention

[0011] This invention provides a metadata index storage method based on a solid-state drive-based local block storage engine, comprising the following steps:

[0012] Set up a record cache in memory to cache hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number;

[0013] Write-ahead logs are set up on the solid-state drive (SSD) to persistently record the latest metadata changes, and the associated metadata entries reside in memory. A log structure index is built on the SSD to store incrementally updated metadata, containing internally ordered log index blocks.

[0014] A baseline index is built in the solid-state drive to store metadata of the full quantum data blocks. Each small data block corresponds to the index root node, and the metadata entries are linearly mapped to fixed physical locations.

[0015] When performing a metadata query: first search the record cache; if no match is found, concurrently search the log structure index and baseline index, and take the entry with the largest sequence number.

[0016] When performing metadata updates: write the metadata changes to the write-ahead log, persist them, and then update the record cache;

[0017] Maintenance tasks are performed periodically through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

[0018] Preferably, as one possible implementation, when performing "setting up a record cache in memory, caching hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number", the method also includes record cache management:

[0019] The cache unit for record cache management is a single metadata entry;

[0020] Assign a globally monotonically increasing sequence number to identify the version for each individual metadata entry;

[0021] When multiple versions of an entry exist, only the metadata entry with the largest serial number is retained as the latest metadata entry.

[0022] Preferably, as one possible implementation, the write-ahead log recycling process includes:

[0023] A background thread periodically scans the record cache to identify cold data metadata entries;

[0024] Determine the latest version of the item to be downloaded based on the global serial number;

[0025] Calculate the physical storage location of the target metadata entry in the baseline index, where:

[0026] The physical storage location is determined by a linear mapping formula: Physical offset = base address + (sub-block ID × fixed size of metadata entry);

[0027] Ensure the target location is within the index range of the corresponding small data block;

[0028] Write the latest version of the metadata entries directly to the corresponding physical location of the baseline index and perform an in-place update;

[0029] After the update is complete, mark the entry in the record cache as "flushed" to reduce memory usage;

[0030] If an entry with a higher sequence number is detected in the baseline index during the write process, the write operation is abandoned to ensure data consistency.

[0031] Preferably, as one possible implementation method, the record cache flushing includes:

[0032] The background maintenance thread periodically scans and records metadata entries in the cache, identifying cold data entries based on access frequency;

[0033] Perform version verification on each cold data entry to be flushed:

[0034] Read the current version of the entry in the baseline index;

[0035] Compare the sequence numbers of the record cache entries with those of the baseline index entries;

[0036] The flush operation is performed only if the sequence number of the cached entry is larger:

[0037] Locate the storage area in the baseline index based on the sub-data block ID;

[0038] Calculate the target physical offset using the sub-data block ID;

[0039] The operation directly writes the latest version of the metadata entry in the record cache to the baseline index to achieve an in-place update;

[0040] Update memory management status after flashing:

[0041] Release memory for persistent cold data entries in the record cache;

[0042] Update the access count for the cache replacement policy;

[0043] Preferably, as one possible implementation, the log structure space index reclamation and cleanup includes:

[0044] A background thread periodically reads the oldest log index block from the log structure index of the solid-state drive;

[0045] Iterate through all metadata entries in the oldest log index block sequentially, performing the following operations for each metadata entry:

[0046] Query the record cache to check if a version with a larger sequence number than the current entry exists;

[0047] Query the baseline index to check if there is a version with a larger sequence number than the current entry;

[0048] If no updated version exists in the record cache or the baseline index, then the metadata entry is flushed to the baseline index.

[0049] After the flush is complete, the storage space of the log index block is reclaimed for use by subsequent log structure indexing;

[0050] Repeat the above steps until all reclaimable old log index blocks in the log structure index have been processed.

[0051] Preferably, as one possible implementation, a baseline index is constructed in the solid-state drive to store metadata of the full quantum data blocks. Each small data block corresponds to an index root node, and metadata entries are linearly mapped to fixed physical locations. Specifically, this includes the following steps:

[0052] A baseline index storage structure is built in the solid-state drive to store the metadata mapping relationship from all small data blocks to sub-data blocks;

[0053] Create an index root node for each small data block, where:

[0054] The root node of the index records the storage base address of the metadata of all child data blocks corresponding to the small data block;

[0055] The index root node contains information about the number of child data blocks;

[0056] Sub-data block metadata is organized using a linear mapping method:

[0057] Each sub-data block corresponds to a fixed-length metadata entry;

[0058] Metadata entries are arranged sequentially in sub-block ID order on physical storage;

[0059] The physical location of metadata is calculated using the formula: Physical Offset = Base Address + (Sub-block ID × Fixed Size of Metadata Entry);

[0060] When performing a metadata query: locate the index root node by the small data block ID; obtain the metadata storage base address from the root node; calculate the physical location of the target sub-data block metadata;

[0061] When performing metadata updates: locate the physical location of the target metadata; perform an in-place update operation.

[0062] Preferably, as one possible implementation, the sub-data block ID is a unique identifier ID assigned to the sub-data blocks within each small data block.

[0063] Preferably, as one possible implementation, it also includes establishing a direct mapping relationship between sub-data block IDs and physical storage locations; and keeping the mapping relationship constant during system operation.

[0064] Secondly, the present invention provides a metadata indexing storage system based on a local block storage engine of a solid-state drive, comprising a first setting module, a second setting module, a mapping processing module, a retrieval processing module, an update processing module, and a background processing module:

[0065] The first setting module is used to set up a record cache in memory, cache hot metadata at the metadata entry granularity, and mark each metadata entry with a globally incrementing sequence number;

[0066] The second setting module is used to set up a write-ahead log in the solid-state drive to persistently record the latest metadata changes, and its associated metadata entries reside in memory; and to build a log structure index in the solid-state drive to store incrementally updated metadata, including internally ordered log index blocks.

[0067] The mapping processing module is used to build a baseline index in the solid-state drive, store full quantum data block metadata, with each small data block corresponding to an index root node, and metadata entries linearly mapped to fixed physical locations;

[0068] The retrieval processing module is used to perform metadata queries by first searching the record cache, and if no match is found, concurrently searching the log structure index and the baseline index, and taking the entry with the largest sequence number.

[0069] The update processing module is used to: write the metadata changes to the write-ahead log, persist them, and then update the record cache when performing metadata updates;

[0070] The background processing module is used to periodically perform maintenance tasks through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

[0071] Thirdly, a computer storage medium stores a computer program, which, when executed by a computer, implements the metadata index storage method of a local block storage engine based on a solid-state drive as described above.

[0072] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0073] In summary, the metadata index storage scheme based on a solid-state drive (SSD) local block storage engine provided by the above-mentioned technical solution of this invention includes the following methods: Setting up a record cache in memory to cache hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number; setting up a write-ahead log in the SSD to persistently record the most recent metadata changes, with its associated metadata entries residing in memory; constructing a log structure index in the SSD to store incrementally updated metadata, including internally ordered log index blocks; constructing a baseline index in the SSD to store full-quantum data block metadata, with each small data block corresponding to an index root node, and metadata entries linearly mapped to fixed physical locations; when performing a metadata query: first searching the record cache, and if no match is found, concurrently searching the log structure index and the baseline index, taking the entry with the largest sequence number; when performing a metadata update: writing the metadata changes to the write-ahead log, persisting it, and then updating the record cache; and periodically performing maintenance tasks through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

[0074] Analysis of the above technical solutions reveals that the metadata indexing and storage method based on a solid-state drive (SSD) local block storage engine provided by this invention achieves high-performance metadata access through an innovative four-layer hybrid index architecture and background maintenance mechanism. It significantly improves the response speed for frequently accessed metadata by caching hot data at a fine-grained level (metadata entry level) and marking it with a global sequence number. During queries, the memory cache is checked first; if not found, the log structure index and baseline index are searched concurrently. The sequence number ensures data up-to-dateness and reduces read latency. Simultaneously, an incremental-baseline separation design is implemented, allowing the log structure index to store incremental updates (write optimization) and the baseline index to store full data (read optimization), balancing read and write performance and adapting to SSD characteristics.

[0075] Meanwhile, the above scheme achieves efficient storage space utilization and I / O performance optimization by recording metadata changes through write-ahead logs and ensuring crash consistency, storing incremental data in ordered log blocks to reduce random write amplification, and using fixed physical location mapping. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of a one-data index structure in the existing technology;

[0078] Figure 2 This is a schematic diagram of another metadata index structure in the existing technology;

[0079] Figure 3 This is a flowchart illustrating a metadata index storage method based on a local block storage engine using a solid-state drive, as provided in an embodiment of the present invention.

[0080] Figure 4 This is a schematic diagram of the read / write relationship of the metadata hybrid indexing mechanism in a metadata indexing storage method based on a local block storage engine of solid-state drive provided in an embodiment of the present invention;

[0081] Figure 5 This describes the relationship between small data blocks and sub-data blocks in the baseline index mapping structure of this invention embodiment;

[0082] Figure 6 This is a schematic diagram illustrating the principle of the metadata index storage system based on the local block storage engine of a solid-state drive according to an embodiment of the present invention.

[0083] Labels: First setting module 10, Second setting module 20, Mapping processing module 30, Retrieval processing module 40, Update processing module 50, Background processing module 60.

[0084] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0085] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0086] To address the problems in the prior art, the following embodiments of the present invention provide a metadata index storage scheme based on a local block storage engine using solid-state drives, aiming to solve the problem of quickly detecting faults and rapidly switching to disaster recovery in the above-mentioned load balancing scenario, thereby improving the high availability of the system.

[0087] Example 1

[0088] To achieve the above objectives, see [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating a metadata index storage method based on a local block storage engine using a solid-state drive, as provided in an embodiment of the present invention.

[0089] like Figure 3 As shown in Embodiment 1 of the present invention, a metadata index storage method based on a local block storage engine using a solid-state drive includes the following steps:

[0090] This invention provides a metadata index storage method based on a solid-state drive-based local block storage engine, comprising the following steps:

[0091] S10. Set up a record cache in memory to cache hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number;

[0092] S20. Set up a write-ahead log in the solid-state drive to persistently record the latest metadata changes, and keep the associated metadata entries in memory; build a log structure index in the solid-state drive to store incrementally updated metadata, including internally ordered log index blocks; record metadata changes through the write-ahead log and ensure crash consistency, with related entries keeping in memory;

[0093] S30. Build a baseline index in the solid-state drive to store full quantum data block metadata. Each small data block corresponds to the index root node, and the metadata entries are linearly mapped to fixed physical locations.

[0094] S40. When performing a metadata query: first search the record cache; if no match is found, concurrently search the log structure index and baseline index, and take the entry with the largest sequence number.

[0095] S50. When performing metadata updates: write the metadata changes to the write-ahead log, persist them, and then update the record cache;

[0096] S60. Periodically execute maintenance tasks via background threads, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup: specifically, the following steps are included: (a) convert the metadata of the write-ahead log into log index blocks and write them to the log structure index; (b) update the cold data in the record cache to the corresponding physical location of the baseline index in place; (c) clean up the log structure index, discard entries that have been flushed or expired, and flush the rest to the baseline index.

[0097] Appendix Figure 4 This is a schematic diagram of the read / write relationship of the metadata hybrid indexing mechanism in this embodiment; as shown below. Figure 4 As shown, the components of the metadata hybrid indexing mechanism in this embodiment include: a record cache, a write-ahead log, a log structure index, and a baseline index. The record cache resides in memory, while the data of the other components is persisted on a solid-state drive (SSD). The record cache caches hot metadata in memory at the metadata entry granularity (rather than leaf nodes). In most cases, access to metadata will hit the record cache, avoiding SSD I / O overhead. Each metadata entry has a globally incrementing sequence number to mark its version; when multiple versions of metadata entries exist, the one with the largest sequence number represents the latest metadata entry. The write-ahead log records recent metadata changes, ensuring data consistency in the event of a system crash. Relevant metadata entries stored in the write-ahead log must reside in memory for fast querying. The log structure index stores recent incremental updates to metadata and contains several internally ordered log index blocks for easy and fast retrieval; the metadata ranges of multiple log index blocks can overlap. The write-ahead log periodically reclaims space, quickly converting its metadata into log index blocks and writing them into the log structure index. The relevant metadata entries stored in the log structure index do not reside permanently in memory, but are dynamically loaded based on access frequency, further optimizing memory usage. This is the most fundamental difference between it and write-ahead logs.

[0098] like Figure 5 As shown, Figure 5This illustrates the relationship between small data blocks and sub-data blocks in the baseline index mapping structure. The baseline index stores all metadata related to the small data blocks and their sub-data blocks. Each small data block corresponds to an index root node, which points to the metadata of all sub-data blocks contained within that small data block. Each sub-data block corresponds to a fixed-length metadata entry. Unlike a typical B-tree, because the number of sub-data blocks contained in each small data block is fixed, the number of corresponding metadata entries is also fixed. Therefore, here, metadata entries are linearly mapped to their corresponding physical storage locations. This design simplifies the index structure, eliminates the need for complex tree structure maintenance (splitting and merging), and improves query and update efficiency.

[0099] Preferably, as one possible implementation, when performing "setting up a record cache in memory, caching hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number", the method also includes record cache management:

[0100] The cache unit for record cache management is a single metadata entry;

[0101] Assign a globally monotonically increasing sequence number to identify the version for each individual metadata entry;

[0102] When multiple versions of an entry exist, only the metadata entry with the largest serial number is retained as the latest metadata entry.

[0103] Preferably, as one possible implementation, the write-ahead log recycling process includes:

[0104] S611. A background thread periodically scans the record cache to identify cold data metadata entries;

[0105] S612. Determine the latest version of the entry to be downloaded based on the global serial number;

[0106] S613. Calculate the physical storage location of the target metadata entry in the baseline index, where:

[0107] S6131. ​​The physical storage location is determined by a linear mapping formula: Physical offset = base address + (sub-block ID × fixed size of metadata entry);

[0108] S6132. Ensure that the target location is within the index range of the corresponding small data block;

[0109] S614. Write the latest version of the metadata entry directly to the corresponding physical location of the baseline index and perform an in-place update;

[0110] S615. After the update is complete, mark the entry in the record cache as "washed down" to reduce memory usage;

[0111] S616. If an entry with a higher sequence number is detected in the baseline index during the write process, the write operation is abandoned to ensure data consistency.

[0112] It should be noted that the above write-ahead log reclamation process includes: a background thread periodically updating the corresponding physical location in the baseline index directly in-place with cold data in the record cache. Because each metadata entry has a global sequence number, the system can directly write the latest version of that metadata entry in the record cache to the baseline index. This in-place update strategy avoids redundant data copying, reduces I / O overhead, and ensures data consistency and integrity.

[0113] Preferably, as one possible implementation method, the record cache flushing includes:

[0114] S621. The background maintenance thread periodically scans and records metadata entries in the cache, and identifies cold data entries based on access frequency;

[0115] S622. Perform version verification for each cold data entry to be refreshed:

[0116] S6221. Read the current version of this entry in the baseline index;

[0117] S6222. Compare the sequence numbers of the record cache entries with those of the baseline index entries;

[0118] S623. Perform the flush operation if and only if the record cache entry sequence number is larger:

[0119] S6231. Locate the storage area in the baseline index based on the sub-data block ID;

[0120] S6232. Calculate the target physical offset using the sub-data block ID;

[0121] S6233. Execute an in-place update operation by directly writing the latest version of the metadata entry in the record cache to the baseline index;

[0122] S624. Update memory management status after flashing:

[0123] S6241. Release memory for persistent cold data entries in the record cache;

[0124] S6242. Update the access count for the cache replacement policy;

[0125] It should be noted that the aforementioned background thread periodically updates the corresponding physical location in the baseline index directly in-place with cold data from the record cache. Because each metadata entry has a global sequence number, the system can directly write the latest version of that metadata entry from the record cache to the baseline index. This in-place update strategy avoids redundant data copying, reduces I / O overhead, and ensures data consistency and integrity.

[0126] Preferably, as one possible implementation, the log structure space index reclamation and cleanup includes:

[0127] S631. A background thread periodically reads the oldest log index block from the log structure index on the solid-state drive;

[0128] S632. Sequentially traverse all metadata entries in the oldest log index block, and perform the following operations for each metadata entry:

[0129] S6321. Query the record cache and check if a version with a larger sequence number than the current entry exists;

[0130] S6322. Query the baseline index to check if there is a version with a larger sequence number than the current entry;

[0131] S6323. If no updated version exists in the record cache and the baseline index, then flush the metadata entry to the baseline index;

[0132] S633. After the flush is completed, the storage space of the log index block is reclaimed for use by subsequent log structure indexing;

[0133] S634. Repeat the above steps until all reclaimable old log index blocks in the log structure index have been processed.

[0134] A further optional implementation scheme includes, in addition to the above steps, the following: S635. Using a batch flushing mechanism, multiple eligible metadata entries are merged and written into the baseline index to reduce the random I / O overhead of the solid-state drive; S636. During the flushing process, if an entry with the same sub-block ID is detected in the baseline index, only the version with the largest sequence number is retained to ensure data consistency; S637. After space reclamation, the metadata range information of the log structure index is updated to optimize subsequent query efficiency.

[0135] By periodically cleaning up the oldest log index blocks, storage space for the log structure index is effectively reclaimed, preventing storage bloat. A version comparison mechanism ensures that only overwritten metadata entries are flushed down, maintaining data consistency. Batch flushing optimizes I / O performance and reduces the write amplification effect of solid-state drives.

[0136] It should be noted that the above process involves scanning the metadata entries marked as cold data in the record cache; locating the physical address corresponding to the entry based on the linear mapping relationship of the baseline index; directly overwriting the latest version of the entry data to the physical location of the baseline index; and retaining the entry sequence number and version identifier.

[0137] Preferably, as one possible implementation, a baseline index is constructed in the solid-state drive to store metadata of the full quantum data blocks. Each small data block corresponds to an index root node, and metadata entries are linearly mapped to fixed physical locations. Specifically, this includes the following steps:

[0138] S301. Construct a baseline index storage structure in the solid-state drive to store the metadata mapping relationship from all small data blocks to sub-data blocks;

[0139] S302. Create an index root node for each small data block, where:

[0140] S3021. The index root node records the storage base address of the metadata of all sub-data blocks corresponding to the small data block;

[0141] S3022. The index root node contains information about the number of child data blocks;

[0142] S303. Organize sub-data block metadata using a linear mapping method:

[0143] S3031. Each sub-data block corresponds to a fixed-length metadata entry;

[0144] S3032. Metadata entries are arranged consecutively in sub-block ID order on physical storage;

[0145] S3033. Calculate the physical location of metadata using the formula: Physical Offset = Base Address + (Sub-block ID × Fixed Size of Metadata Entry);

[0146] S304. When performing a metadata query: S3041. Locate the index root node using the small data block ID; S3042. Obtain the metadata storage base address from the root node; S3043. Calculate the physical location of the target sub-data block metadata;

[0147] S305. When performing metadata updates: S3051. Locate the physical location of the target metadata; S3052. Perform an in-place update operation;

[0148] It's important to note that a baseline index is built on the solid-state drive (SSD) to store metadata for all quantum data blocks. Each small data block corresponds to an index root node, and metadata entries are linearly mapped to fixed physical locations. Specifically, the baseline index stores all metadata related to the small data blocks and their child data blocks. Each small data block corresponds to an index root node, which points to the metadata of all child data blocks contained within that small data block. Each child data block corresponds to a fixed-length metadata entry. The number of metadata entries associated with each small data block is fixed; a direct mapping is established from the small data block index root node to the physical storage location; and child data block metadata entries are stored contiguously at fixed lengths. Unlike a typical B-tree, because the number of child data blocks contained in each small data block is fixed, the number of corresponding metadata entries is also fixed. Therefore, here, metadata entries are linearly mapped to their corresponding physical storage locations. This design simplifies the index structure, eliminates the need for complex tree structure maintenance (splitting and merging), and improves query and update efficiency.

[0149] Preferably, as one possible implementation, the sub-data block ID is a unique identifier ID assigned to the sub-data blocks within each small data block.

[0150] Preferably, as one possible implementation, it also includes establishing a direct mapping relationship between sub-data block IDs and physical storage locations; and keeping the mapping relationship constant during system operation.

[0151] The existing storage solutions based on log structure merge trees or B-trees are typically designed for general key-value (KV) storage and are not specifically optimized for local block storage engine metadata. Therefore, while these solutions offer excellent versatility and are theoretically applicable to various scenarios, they still require further customization and improvement to meet the specific needs of local block storage engine metadata.

[0152] In this embodiment, the write-ahead log and log structure index store incremental metadata updates, and all metadata entries contained therein can be considered as the parts of the system with high write activity. The record cache, as a memory cache, also carries these frequently accessed metadata entries. Depending on memory capacity and dynamic load, there are typically two scenarios: sufficient memory and limited memory. We define sufficient memory as the case where the record cache contains all metadata entries from the write-ahead log and log structure index, and limited memory as the case where the record cache does not contain all metadata entries. This embodiment improves the overall system performance and efficiency by efficiently combining the local block storage engine's metadata memory cache with a persistent storage mechanism.

[0153] When querying metadata, there are three scenarios: hitting the record cache, retrieving from the log structure index, or reading directly from the baseline index. When memory is sufficient, only two scenarios occur: hitting the record cache and reading directly from the baseline index. The former does not require accessing the SSD, while the latter only requires one SSD access. When memory is limited, the system also needs to retrieve metadata from the log structure index, increasing the number of SSD accesses. However, by optimizing the retrieval strategy by referencing the implementation of the log structure merging tree, the overall access time can still be significantly reduced, ensuring the system maintains high efficiency under high load.

[0154] When inserting or updating metadata, the system only needs to append the changed metadata entries to the write-ahead log, exhibiting the same characteristics and performance advantages as other append-only write schemes. In the background mechanism, the system directly updates the baseline index with the latest version of the metadata entries from the record cache, avoiding frequent data migration and minimizing write amplification. Provided there is sufficient memory, regardless of how frequently metadata is updated, the theoretical write amplification factor is close to 1-2, a significant advantage compared to the typically tens of write amplification factors in log structure merge trees. The design of the log structure index allows the system to efficiently handle metadata updates even with limited memory. Compared to a single B-tree or log structure merge tree, it balances memory utilization and SSD access efficiency, achieving high performance under complex load environments.

[0155] The baseline index, as the core storage structure for metadata, employs a linear mapping approach to ensure rapid location of the physical storage location of metadata entries. Compared to the hierarchical storage structure of the log structure's merge tree, the baseline index requires only a single SSD access to read or update metadata entries, greatly simplifying the storage path, reducing SSD I / O overhead, and improving data access efficiency. Compared to B-trees, the baseline index eliminates the need for frequent node splitting and merging operations, resulting in a more stable structure, reduced maintenance costs, and further improved system response speed and stability under high concurrency scenarios.

[0156] Example 2

[0157] In addition, based on the same concept of the above method embodiments, the present invention also provides a metadata index storage system based on a solid-state drive local block storage engine to implement the above method of the present invention. Since the principle and method of solving the problem in this system embodiment are similar, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0158] See Figure 6This invention provides a metadata indexing and storage system based on a solid-state drive local block storage engine, comprising a first setting module 10, a second setting module 20, a mapping processing module 30, a retrieval processing module 40, an update processing module 50, and a background processing module 60.

[0159] The first setting module 10 is used to set up a record cache in memory, cache hot metadata at the metadata entry granularity, and mark each metadata entry with a globally incrementing sequence number;

[0160] The second setting module 20 is used to set a write-ahead log in the solid-state drive to persistently record the latest metadata changes, and its associated metadata entries are resident in memory; and to build a log structure index in the solid-state drive to store incrementally updated metadata, including internally ordered log index blocks.

[0161] The mapping processing module 30 is used to build a baseline index in the solid-state drive, store full quantum data block metadata, with each small data block corresponding to the index root node, and the metadata entries linearly mapped to a fixed physical location;

[0162] The retrieval processing module 40 is used to perform metadata queries by first searching the record cache, and if no match is found, concurrently searching the log structure index and the baseline index, and taking the entry with the largest sequence number.

[0163] The update processing module 50 is used to write the metadata changes to the write-ahead log and then update the record cache after persistence when performing metadata updates.

[0164] The background processing module 60 is used to periodically perform maintenance tasks through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

[0165] Example 3

[0166] On the other hand, this third embodiment, based on the metadata indexing and storage method of a local block storage engine based on a solid-state drive provided in the first embodiment, also provides a computer storage medium (hereinafter referred to as the storage medium). When the computer program of the computer storage medium provided in this third embodiment is executed by a computer, it implements the metadata indexing and storage method of a local block storage engine based on a solid-state drive as described above.

[0167] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, and optical storage) containing computer-usable program code.

[0168] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0169] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc., does not indicate any order. These words can be interpreted as names.

[0172] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0173] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A metadata indexing and storage method based on a solid-state drive-based local block storage engine, characterized in that, Includes the following steps: Set up a record cache in memory to cache hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number; Write-ahead logs are set up on the solid-state drive (SSD) to persistently record the latest metadata changes, and the associated metadata entries reside in memory. A log structure index is built on the SSD to store incrementally updated metadata, containing internally ordered log index blocks. A baseline index is built in the solid-state drive to store metadata of the full quantum data blocks. Each small data block corresponds to the index root node, and the metadata entries are linearly mapped to fixed physical locations. When performing a metadata query: first search the record cache; if no match is found, concurrently search the log structure index and baseline index, and take the entry with the largest sequence number. When performing metadata updates: write the metadata changes to the write-ahead log, persist them, and then update the record cache; Maintenance tasks are performed periodically through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

2. The method according to claim 1, characterized in that, The execution of "setting up a record cache in memory, caching hot metadata at the metadata entry granularity, with each metadata entry marked with a globally incrementing sequence number" also includes record cache management: The cache unit for record cache management is a single metadata entry; Assign a globally monotonically increasing sequence number to identify the version for each individual metadata entry; When multiple versions of an entry exist, only the metadata entry with the largest serial number is retained as the latest metadata entry.

3. The method according to claim 1, characterized in that, The write-ahead log reclamation process includes: A background thread periodically scans the record cache to identify cold data metadata entries; Determine the latest version of the item to be downloaded based on the global serial number; Calculate the physical storage location of the target metadata entry in the baseline index, where: The physical storage location is determined by a linear mapping formula: Physical offset = base address + (sub-block ID × fixed size of metadata entry); Ensure the target location is within the index range of the corresponding small data block; Write the latest version of the metadata entries directly to the corresponding physical location of the baseline index and perform an in-place update; After the update is complete, mark the entry in the record cache as "flushed" to reduce memory usage; If an entry with a higher sequence number is detected in the baseline index during the write process, the write operation is abandoned to ensure data consistency.

4. The method according to claim 1, characterized in that, The record cache flushing includes: The background maintenance thread periodically scans and records metadata entries in the cache, identifying cold data entries based on access frequency; Perform version verification on each cold data entry to be flushed: Read the current version of the entry in the baseline index; Compare the sequence numbers of the record cache entries with those of the baseline index entries; The flush operation is performed only if the sequence number of the cached entry is larger: Locate the storage area in the baseline index based on the sub-data block ID; Calculate the target physical offset using the sub-data block ID; The operation directly writes the latest version of the metadata entry in the record cache to the baseline index to achieve an in-place update; Update memory management status after flashing: Release memory for persistent cold data entries in the record cache; Update the access count for the cache replacement policy.

5. The method according to claim 1, characterized in that, The log structure space index reclamation and cleanup includes: A background thread periodically reads the oldest log index block from the log structure index of the solid-state drive; Iterate through all metadata entries in the oldest log index block sequentially, performing the following operations for each metadata entry: Query the record cache to check if a version with a larger sequence number than the current entry exists; Query the baseline index to check if there is a version with a larger sequence number than the current entry; If no updated version exists in the record cache or the baseline index, then the metadata entry is flushed to the baseline index. After the flush is complete, the storage space of the log index block is reclaimed for use by subsequent log structure indexing; Repeat the above steps until all reclaimable old log index blocks in the log structure index have been processed.

6. The method according to claim 1, characterized in that, A baseline index is built in the solid-state drive to store metadata of the full quantum data blocks. Each small data block corresponds to the index root node, and the metadata entries are linearly mapped to fixed physical locations. The specific steps include the following: A baseline index storage structure is built in the solid-state drive to store the metadata mapping relationship from all small data blocks to sub-data blocks; Create an index root node for each small data block, where: The root node of the index records the storage base address of the metadata of all child data blocks corresponding to the small data block; The index root node contains information about the number of child data blocks; Sub-data block metadata is organized using a linear mapping method: Each sub-data block corresponds to a fixed-length metadata entry; Metadata entries are arranged sequentially in sub-block ID order on physical storage; The physical location of metadata is calculated using the formula: Physical Offset = Base Address + (Sub-block ID × Fixed Size of Metadata Entry); When performing a metadata query: locate the index root node by the small data block ID; obtain the metadata storage base address from the root node; calculate the physical location of the target sub-data block metadata; When performing metadata updates: locate the physical location of the target metadata; perform an in-place update operation.

7. The method according to claim 1, characterized in that, The sub-data block ID is a unique identifier assigned to the sub-data blocks within each small data block.

8. The method according to claim 1, characterized in that, It also includes establishing a direct mapping relationship between sub-data block IDs and physical storage locations; and keeping the mapping relationship constant during system operation.

9. A metadata indexing storage system based on a solid-state drive local block storage engine, characterized in that, It includes a first settings module, a second settings module, a mapping processing module, a retrieval processing module, an update processing module, and a background processing module: The first setting module is used to set up a record cache in memory, cache hot metadata at the metadata entry granularity, and mark each metadata entry with a globally incrementing sequence number; The second setting module is used to set up a write-ahead log in the solid-state drive to persistently record the latest metadata changes, and its associated metadata entries reside in memory; and to build a log structure index in the solid-state drive to store incrementally updated metadata, including internally ordered log index blocks. The mapping processing module is used to build a baseline index in the solid-state drive, store full quantum data block metadata, with each small data block corresponding to an index root node, and metadata entries linearly mapped to fixed physical locations; The retrieval processing module is used to perform metadata queries by first searching the record cache, and if no match is found, concurrently searching the log structure index and the baseline index, and taking the entry with the largest sequence number. The update processing module is used to: write the metadata changes to the write-ahead log, persist them, and then update the record cache when performing metadata updates; The background processing module is used to periodically perform maintenance tasks through a background thread, including write-ahead log recycling, record cache flushing, and log structure space index recycling and cleanup.

10. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements a metadata indexing storage method for a local block storage engine based on a solid-state drive as described in any one of claims 1 to 8.

Images