Append-oriented persistent buckets and metadata indexing methods and systems with reconstructable volatile mappings

By introducing a first-level ordered routing layer and a second-level append-only persistent bucket into the metadata index, combined with the volatile mapping structure of volatile storage, the scalability and query efficiency issues of the metadata index when volatile storage space is limited are solved, the index state is made persistent and reconstructable, and the frequency of in-situ rewrites is reduced.

CN123115136APending Publication Date: 2026-11-17EAST CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611272568.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing metadata indexes struggle to maintain the persistence and rebuildability of the index state when volatile storage space is limited. This results in low efficiency for point queries, frequent in-situ rewriting during updates, and limited index scalability and query capabilities as bucket capacity increases.

Method used

The method combines a first-level ordered routing layer with a second-level append-only persistent bucket. By establishing append-only persistent buckets in non-volatile storage media and establishing volatile mapping structures on demand in volatile memory, the splitting and refreshing module splits and switches local buckets that have reached the capacity threshold, reducing in-situ rewriting and supporting local expansion and controlled memory residency.

Benefits of technology

It enables control over the number of mappings when volatile storage space is limited, maintains the persistence and rebuildability of the index state, improves point query efficiency, reduces in-situ rewriting during the update process, and maintains the scalability and query capability of the index.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115136A_ABST
    Figure CN123115136A_ABST
Patent Text Reader

Abstract

This invention discloses a metadata indexing method and system using append-only persistent buckets and reconstructable volatile mappings. The indexing method includes: establishing multiple append-only persistent buckets in a non-volatile storage medium; appending inserted, updated, or deleted records to the target bucket and updating them synchronously while the mapping resides; establishing volatile mappings and performing point queries; scanning and replaying records within the bucket to reconstruct the mapping when it is missing; determining the split key, persisting the new bucket first, and then switching routes and connecting adjacent buckets. The system includes: a first-level ordered routing layer, a non-volatile storage, a volatile storage, a mapping management and reconstruction module, and a splitting and refreshing module. Compared with existing technologies, this invention maintains the persistence and reconstructability of the index state in the metadata index, improves point query efficiency, reduces in-situ rewriting, maintains the scalability and query capability of the index when volatile storage space is limited, and supports local expansion and controlled memory residency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of file system metadata management and key-value indexing technology, and in particular to a metadata indexing method and system for append-only persistent buckets and reconstructable volatile mappings. Background Technology

[0002] Metadata indexes, used to locate metadata values ​​based on index keys, are a fundamental component of file systems, distributed storage systems, and data management systems. Existing metadata indexes typically employ tree, log, or hash structures. Tree structures provide ordered retrieval capabilities, but updates may involve node lookups, key movements, and structural adjustments; fully persistent hash structures offer faster point queries, but bucket resizing, in-situ updates, and random persistent writes can increase storage write overhead. Another implementation stores hash maps in volatile memory to speed up queries. However, as the index size increases, the long-term residence of all maps consumes significant amounts of volatile storage space; after process exits, the runtime environment restarts, or the index state in volatile memory is lost, the index state needs to be retrieved. If only volatile maps are stored without a reconstructable persistent representation, it becomes difficult to continue providing correct queries after the maps are released or lost.

[0003] In summary, existing metadata workloads typically involve numerous point queries, as well as insert, update, and delete operations. If all runtime mappings reside in volatile memory for extended periods, it results in significant memory consumption. Conversely, directly scanning the complete persistent data when mappings are missing increases point query latency. Therefore, how to improve point query efficiency, reduce in-situ rewrites during updates, and maintain index scalability and query capabilities while ensuring index persistence and rebuildability, while also mitigating bucket size growth and limited volatile storage space, becomes a critical technical challenge in metadata index design. Consequently, a metadata indexing mechanism that combines append-only persistent records, fast point queries, local structure expansion, and on-demand rebuilding of volatile mappings is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by designing a metadata indexing method and system with append-only persistent buckets and reconstructable volatile mappings. It employs a method of establishing corresponding volatile mapping structures on demand in volatile memory, enabling the index to control the number of resident mappings when volatile storage space is limited, while retaining the ability to recover runtime state from persistent buckets. This effectively solves the problem of maintaining the persistence and reconstructability of the index state in the metadata index while improving point query efficiency, reducing in-situ rewrites during the update process, and maintaining the scalability and query capability of the index when bucket capacity grows and volatile storage space is limited. The method's first-level ordered routing layer arranges routing entries according to the smallest valid key of the second-level append-only persistent buckets, locating the corresponding key field through the bucket identifier. The second-level append-only persistent buckets are set in non-volatile storage media, storing the corresponding change records for insertion, update, and deletion in an append-only manner. The index also includes a splitting and refreshing module to split local buckets that have reached the capacity threshold and switch the corresponding routes. This invention can reduce in-situ rewrites, provide fast point queries, and support local expansion and controlled memory residency, showing promising application prospects.

[0005] The specific technical solution to achieve the objective of this invention is: a metadata indexing method for append-only persistent buckets and reconstructable volatile mappings, characterized in that the metadata indexing includes the following steps: Step 1: Establish multiple append-only persistent buckets linked in key field order in non-volatile storage medium, maintain a first-level ordered routing entry based on the least valid key of the multiple append-only persistent buckets, and establish volatile mappings corresponding to the accessed buckets in volatile memory as needed. Step 2: Locate the target append-only persistent bucket based on the first-level ordered routing entry, append the change record corresponding to the write request to the target append-only persistent bucket, and update the volatile mapping synchronously when the corresponding volatile mapping has been established; Step 3: Locate the target append-only persistent bucket based on the first-level ordered routing item, and obtain the latest valid metadata value of the query key when the corresponding volatile mapping has been established; Step 4: If the volatile mapping does not exist, scan and replay the change records in the target append-only persistent bucket in the order of writing, and complete the point query after establishing the volatile mapping; Step 5: When the target append-only persistent bucket reaches the preset capacity threshold, determine the split key based on the current valid key set of the bucket, construct and persist the first new bucket and the second new bucket, switch the corresponding first-level ordered routing entries and adjacent bucket links, and mark the original target append-only persistent bucket as invalid.

[0006] A metadata indexing system with append-only persistent buckets and reconstructable volatile mappings is characterized by comprising: a first-level ordered routing layer, a non-volatile memory, a volatile memory, a mapping management and reconstruction module, and a splitting and refreshing module. The first-level ordered routing layer maintains ordered routing entries based on the least valid key of multiple second-level append-only persistent buckets, and each routing entry records at least the least valid key and bucket identifier of the corresponding second-level append-only persistent bucket. The non-volatile memory contains multiple second-level append-only persistent buckets linked sequentially by key fields, and each second-level append-only persistent bucket... The persistent bucket stores metadata change records for insertion, update, or deletion in an append-only manner; the volatile memory contains multiple rebuildable volatile mapping structures, each corresponding one-to-one with the accessed secondary append-only persistent bucket, used to record the latest valid metadata value corresponding to the index key; the mapping management and rebuilding module manages the residence, release, and rebuilding of rebuildable volatile mapping structures; the splitting and refreshing module performs partial splitting on secondary append-only persistent buckets that reach a preset capacity threshold and refreshes the first-level ordered routing layer; the metadata indexing system, when a write request is made, undergoes a... The first-level ordered routing layer locates the target secondary append-only persistent bucket and appends the corresponding change records to it. When the corresponding reconstructable volatile mapping structure is already established, it is updated synchronously. When the reconstructable volatile mapping structure is not yet established, the change records are included in the current valid state obtained from subsequent reconstruction, and a query is performed on the target secondary append-only persistent bucket located by the first-level ordered routing layer. When the corresponding reconstructable volatile mapping structure is already established, the latest valid metadata value corresponding to the query key is obtained from it. When the reconstructable volatile mapping structure does not exist, the change records in the target secondary append-only persistent bucket are scanned according to the record writing order, and the corresponding reconstructable volatile mapping structure is rebuilt. When the target secondary append-only persistent bucket reaches a preset capacity threshold, the splitting and refreshing module determines the split key based on the current valid key set of the bucket, writes valid key-value items smaller than the split key to the first new bucket, and writes valid key-value items greater than or equal to the split key to the second new bucket. After the persistence of the first and second new buckets is completed, the corresponding first-level routing item and the adjacent bucket link are switched, and the original target secondary append-only persistent bucket is marked as invalid.

[0007] The secondary append-only persistent bucket adopts an append-only record layout. Each change record includes at least: an index key, metadata value, operation type, and a monotonically increasing record number within the bucket used to determine the order of writing. When rebuilding the reconstructable volatile mapping structure, the rule of overwriting earlier written records with later written records is applied to the same index key, and the corresponding index key is removed according to the deletion marker record. The data stored in the secondary append-only persistent bucket constitutes the basis for the persistence of the index state.

[0008] The first-level ordered routing layer selects the second-level append-only persistent bucket corresponding to the last routing entry whose minimum valid key is not greater than the index key as the target bucket. When the index key is less than the minimum valid key of the first valid bucket, the query returns no result, the insertion request is located in the first valid bucket and its minimum valid key is updated. When there is no valid second-level append-only persistent bucket, an initial empty bucket is created, and when the first insertion request is received, the index key of the insertion request is determined as the minimum valid key of the initial empty bucket and the corresponding first-level routing entry is created.

[0009] When the secondary append-type persistent bucket reaches a preset capacity threshold, multiple sampling keys are selected from the current valid key set of the bucket, the sampling keys are sorted and the split key is determined, and valid key-value items smaller than the split key are written into the first new bucket, and valid key-value items greater than or equal to the split key are written into the second new bucket; when the valid key set of any new bucket is empty, a new split key is selected, or the median key of the current valid key set is used as the split key.

[0010] The append-only persistent bucket and the bucket description area of ​​the new bucket include: bucket status and bucket version information. When splitting, the first new bucket and the second new bucket are marked as being in the construction state. The first new bucket, the second new bucket and the corresponding description information are constructed and persisted. A commit record containing the new bucket identifier and the target route version is generated. The first-level route item and the adjacent bucket link are updated through versioned commit or atomic metadata switching. The original bucket to be split is marked as invalid. When rebuilding, only the second-level append-only persistent buckets that are in the valid state and referenced by the committed route version are processed.

[0011] The reconstructable volatile mapping structure is a runtime query structure that can be recovered based on the change records in the corresponding bucket. A threshold for the number of reconstructable volatile mapping structures residing or a threshold for memory usage are set. When the number of reconstructable volatile mapping structures reached the upper limit for the number of residing or the upper limit for memory usage, at least one reconstructable volatile mapping structure is released according to the least recently used strategy, and the released reconstructable volatile mapping structure is reconstructed when the corresponding secondary append-only persistent bucket is accessed again.

[0012] When the first-level ordered routing layer is missing, a route entry is detected that is inconsistent with a valid second-level append-only persistent bucket, or the amount of change in the route entry reaches a preset refresh threshold, it selects a valid second-level append-only persistent bucket that is referenced by a submitted route version and reconstructs the first-level ordered routing layer based on the bucket status, bucket version, bucket identifier, and minimum valid key of each second-level append-only persistent bucket.

[0013] The non-volatile memory includes at least one of non-volatile memory, solid-state storage device, or disk storage device; the volatile memory includes dynamic random access memory.

[0014] When the index key of the insertion request is less than the current minimum valid key of the target second-level append-only persistent bucket, the minimum valid key of the bucket and the corresponding routing entry are updated; when the deletion request deletes the minimum valid key, a new minimum valid key is determined from the current set of valid keys represented by the corresponding reconstructable volatile mapping structure, or a new minimum valid key is determined by replaying the change records in the bucket if the reconstructable volatile mapping structure does not exist; when the target bucket has no valid key and other valid buckets still exist, the routing entry corresponding to the bucket is deleted or invalidated and the links of adjacent buckets are adjusted, or the bucket is marked as an empty bucket for subsequent reuse; when the target bucket is the last valid bucket, an initial empty bucket is retained or re-established to receive the first subsequent insertion request.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress: 1) The two-level append-only persistent bucket uses append records to save changes and a volatile mapping structure to save the latest valid state, which separates the persistent write path from the point query path, so that ordinary point queries do not need to scan the historical records in the bucket.

[0016] 2) The volatile mapping structure can be created, released and rebuilt as needed, enabling the index to control the number of resident mappings when volatile storage space is limited, while retaining the ability to recover runtime state from the persistent bucket.

[0017] 3) Missing mappings do not block append writes, and subsequent reconstruction restores the current valid state in the order of records, which helps reduce the runtime memory required for cold bucket writing.

[0018] 4) Bucket splitting only processes local buckets that have reached the capacity threshold, and determines the splitting position based on the current set of valid keys, reducing the impact of historical duplicate records or deleted records on the splitting results.

[0019] 5) The first-level ordered routing layer completes bucket-level positioning based on the smallest valid key of each bucket, and can be reconstructed from the valid bucket description information, which is beneficial to maintaining the key domain boundary and routing accuracy of point queries. Attached Figure Description

[0020] Figure 1 A schematic diagram illustrating the overall hierarchical structure and point query path of the metadata index for Example 1; Figure 2 This is a schematic diagram showing the correspondence between the two-level append-type persistent bucket and the reconstructable volatile mapping in Example 1; Figure 3 This is a schematic diagram of the two-level append-type persistent bucket splitting and routing switching in Example 2; Figure 4 This is a schematic diagram illustrating the state transitions of volatile mapping release, missing data, and reconstruction in Examples 1 and 3. Detailed Implementation

[0021] This invention provides a metadata index with append-only persistent buckets and reconstructable volatile mappings. The index establishes a first-level ordered routing layer and a second-level append-only persistent bucket layer. The first-level ordered routing layer arranges routing entries according to the least valid key of the second-level append-only persistent buckets and locates the corresponding key fields through bucket identifiers. The second-level append-only persistent buckets are stored in non-volatile storage media and save the corresponding change records for insertion, update, and deletion in an append-only manner. The index also includes a splitting and refreshing module to split local buckets that have reached a capacity threshold and switch the corresponding routes.

[0022] This invention addresses the accessed two-level append-only persistent bucket by establishing a corresponding volatile mapping structure in volatile memory as needed. The volatile mapping structure only stores the latest valid metadata value corresponding to each index key, excluding historical states overwritten by subsequent records. Point queries first determine the target bucket through a first-level ordered routing layer; if the volatile mapping corresponding to the target bucket has already been established, the search within the bucket is performed in the volatile mapping structure; if the volatile mapping does not exist, scanning and replaying are performed according to the writing order of records in the target bucket to establish the volatile mapping before completing the query.

[0023] The insert, update, and delete operations append new change records to the target secondary append-only persistent bucket. When the corresponding volatile map is already resident, the insert or update operation writes the latest metadata value into the map, and the delete operation removes the corresponding key from the map. When the corresponding volatile map is not resident, the write operation does not need to rebuild the map immediately to complete the append write, and the appended change record is included in the current valid state during subsequent scans and reconstructions.

[0024] For a given index key, the first-level ordered routing layer selects the bucket corresponding to the last route entry whose minimum valid key is not greater than that index key as the target bucket. If the query key is less than the minimum valid key of the first valid bucket, it returns "not found"; if the insertion key is less than the minimum valid key of the first valid bucket, it is written to the first valid bucket and the minimum valid key and corresponding route entry of that bucket are updated. When there is no valid bucket for the index, an initial empty bucket is created first; after the first insertion request arrives, the insertion key is used as the minimum valid key of the initial empty bucket and the first route entry is created.

[0025] For update requests, they can be pre-configured to update only existing index keys or use update-insert merge semantics. In the implementation that only updates existing index keys, it returns "not found" if the target key does not exist. For delete requests, it returns "not found" if the target key does not exist and does not append the deleted record. The operation semantics used are consistent across the same index instance.

[0026] When a deletion operation invalidates the minimum valid key of the target bucket, a new minimum valid key is determined from the current set of valid keys. When there are no valid keys left in a bucket but other valid buckets still exist, the bucket can be removed from the valid bucket chain and its corresponding routing entry deleted or invalidated, or it can be marked as an empty bucket for later reuse. When the last valid bucket is cleared, the bucket is retained or a new initial empty bucket is created so that the index can still receive the first subsequent insert request. Buckets marked as empty but not yet assigned key fields do not participate in first-level ordered routing until they are reactivated.

[0027] When a secondary append-only persistent bucket reaches its capacity threshold, a sampling key is selected from the bucket's current set of valid keys. After sorting, a split key is determined, and valid key-value items are divided into two new buckets based on this split key. To avoid creating empty new buckets, a new split key is selected when either side is empty, or the median key from the current set of valid keys is used. After the new buckets and their descriptions are persisted, the primary routing entries and adjacent bucket links are updated via versioned commit records or atomic metadata switching, and the original bucket is removed from the valid bucket chain.

[0028] In implementations that allow concurrent execution of write requests with map rebuilding or bucket splitting, a write lock, bucket version, or record sequence number high-water mark can be set for each bucket. Map rebuilding first records the upper bound of the scan, and after the initial scan is completed, it replays the tail records appended after the high-water mark, and publishes the new map if the bucket version has not changed; during bucket splitting, writes to the target bucket can be temporarily serialized, or new records after the split snapshot can be replayed to the corresponding new bucket to avoid missing concurrent updates.

[0029] When the first-level ordered routing layer is missing, a route entry is detected to be inconsistent with a valid bucket, or the amount of change in a route entry reaches a preset threshold, the bucket status, bucket version, minimum valid key, and bucket identifier of each second-level append-only persistent bucket can be read. Only buckets that are in a valid state and referenced by a submitted route version are selected to rebuild the first-level ordered routing layer. When no valid bucket exists, an initial empty bucket is created.

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments. The same reference numerals in the drawings denote the same or corresponding components. The following embodiments are used to explain the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Example 1

[0031] See Figure 1The metadata index 100 constructed in this embodiment includes: a first-level ordered routing layer 110, a second-level append-only persistent bucket layer 120, and a rebuildable volatile mapping layer 130, illustrating the first-level ordered routing layer, rebuildable volatile mapping relationships, and point query, write, and rebuild paths. R1 to Rn represent first-level routing entries 111, B1 to Bn represent second-level append-only persistent buckets 121, and M1 to Mn represent volatile mapping structures 131 corresponding to the respective persistent buckets.

[0032] The first-level ordered routing layer 110 maintains first-level ordered routing entries 111. Each first-level routing entry 111 includes at least one minimum valid key and a bucket identifier. Multiple first-level routing entries 111 are arranged in ascending order according to the minimum valid key. For a given index key, the first-level ordered routing layer 110 selects the second-level append-only persistent bucket 121 indicated by the last routing entry whose minimum valid key is not greater than the index key as the target bucket. When the query key is less than the minimum valid key of the first valid bucket, the key is returned as not existing. When the insertion key is less than the minimum valid key of the first valid bucket, the first valid bucket is located and its minimum valid key is updated. When no valid bucket exists, the first-level ordered routing layer 110, in conjunction with the second-level append-only persistent bucket layer 120, establishes an initial empty bucket and creates the corresponding first-level routing entry 111 after the first insertion request arrives.

[0033] The secondary append-only persistent bucket layer 120 establishes multiple secondary append-only persistent buckets 121 in a non-volatile storage medium. Each secondary append-only persistent bucket 121 is connected via forward and backward links according to key field order. Each secondary append-only persistent bucket 121 includes a bucket description area and a record area. The bucket description area records the bucket identifier, minimum valid key, forward bucket identifier, backward bucket identifier, bucket status, and bucket version. The record area is used to save change records in an append-only manner. Each change record may include record length, monotonically increasing record number within the bucket, operation type, index key, metadata value, checksum, and commit flag. The metadata value field for deleted records can be empty. During recovery, complete records are identified based on record length, checksum, and commit flag, while incompletely committed tail records are ignored.

[0034] In an optional implementation, the bucket capacity is set based on the persistent media access granularity, average record length, and target reconstruction overhead. The bucket identifier, minimum valid key, bucket state, bucket version, and committed route version information are stored in the persistent media. The first-level ordered routing layer 110 can cache its working copy during runtime. The memory address of the runtime volatile mapping structure is only valid in the current process and is not used as the persistent address after restart.

[0035] The reconstructable volatile mapping layer 130 establishes volatile mapping structures 131 on demand for hotspot secondary append-only persistent buckets 121 in dynamic random access memory. The volatile mapping structure 131 can employ unordered mappings, hash tables, or other data structures that support key-based lookup and replacement. For the same index key, the volatile mapping structure 131 stores at most one currently valid item.

[0036] See Figure 2 The persistent bucket B2 can retain multiple historical change records with the same index key, while the corresponding volatile mapping structure only retains the current state determined by the last valid operation. The persistent bucket B2 sequentially stores the records of inserting value v1 for k1, inserting value v2 for k2, updating k1 to v3, deleting k2, and inserting value v4 for k3. After replaying according to the record number, the mapping M2 can be reconstructed to retain only the two currently valid items k1 to v3 and k3 to v4.

[0037] When an insert request is received, the first-level ordered routing layer 110 determines the target second-level append-only persistent bucket 121 based on the index key to be inserted. The second-level append-only persistent bucket layer 120 appends a record, including the index key, metadata value, and insert operation type, to the target bucket's record area. If the corresponding volatile mapping structure 131 has already been established, the reconstructable volatile mapping layer 130 writes the corresponding key-value item into the volatile mapping structure 131. If the mapping has not yet been established, it is not required to immediately reconstruct the mapping to complete the append write. If the index key to be inserted is less than the current minimum valid key of the target bucket, the minimum valid key of the bucket and the corresponding first-level routing entry 111 are updated simultaneously.

[0038] When an update request is received, the first-level ordered routing layer 110 determines the target second-level append-only persistent bucket 121. In the implementation that only updates existing index keys, it first confirms that the target key is currently valid; if the target key does not exist, it returns "not found." In the implementation that uses update-as-insert semantics, if the target key does not exist, it is processed as an insertion request. The second-level append-only persistent bucket layer 120 appends the new key-value record to the target bucket record area. If the corresponding volatile mapping structure 131 has already been established, the volatile mapping layer 130 can be rebuilt to replace the old metadata value of the same index key in the mapping with the new metadata value. The old record in the persistent bucket is still retained as historical record.

[0039] When a deletion request is received, the first-level ordered routing layer 110 determines the target second-level append-only persistent bucket 121; if the target key does not exist, it returns "not found" and no deletion record is appended. If the target key exists, the second-level append-only persistent bucket layer 120 appends a deletion marker record containing the index key and deletion operation type to the target bucket; if the corresponding volatile mapping structure 131 has been established, the volatile mapping layer 130 can be rebuilt to delete the corresponding key from the mapping. If the deleted key is the minimum valid key of the target bucket, a new minimum valid key is determined from the current set of valid keys; if the mapping does not exist, a new minimum valid key can be determined by replaying the records in the bucket. If the target bucket no longer has a valid key, the process of exiting the valid bucket chain, reusing an empty bucket, or retaining the initial empty bucket is performed depending on whether other valid buckets still exist.

[0040] See Figure 4 When a point query request is received, the first-level ordered routing layer 110 first determines the target bucket. If the volatile mapping structure 131 of the target bucket has been established, the volatile mapping layer 130 can be rebuilt to directly query the target key; if the mapping does not exist, the mapping management and reconstruction module 150 scans the target bucket according to the record sequence number or physical write order, establishes the mapping, and then queries the target key. Thus, a point query during mapping residency only requires completing bucket-level routing and intra-bucket mapping lookup. In implementations where mapping reconstruction and write requests may be concurrent, the mapping management and reconstruction module 150 records the bucket version and the upper bound of the record sequence number at the start of reconstruction, scans records not greater than this upper bound to form a temporary mapping, and then replays subsequent records within a controlled synchronization interval; the temporary mapping is only published when the bucket version is still consistent with that at the start of reconstruction; otherwise, reconstruction is re-executed or new tail records are replayed. Example 2

[0041] See Figure 3 This embodiment describes a two-level append-only persistent bucket split and a first-level route refresh. In the diagram, B represents the bucket to be split, Bprev and Bnext represent its adjacent buckets, and s represents the split key. When the number of records, the number of bytes occupied, or the proportion of invalid records in the two-level append-only persistent bucket 121 reaches a preset capacity threshold, the split and refresh module 140 triggers a local split. The split and refresh module 140 first obtains the current set of valid keys for the bucket to be split. This set can be obtained directly from the resident volatile mapping structure 131, or it can be obtained by scanning and replaying the change records within the bucket.

[0042] To avoid a full sort of all valid keys, multiple sample keys can be selected from the current set of valid keys using equal-interval, pseudo-random, or segmented sampling methods. These keys are then sorted in volatile memory, and the median sample key or the sample key adjacent to the median position is used as the split key s. The number and position of samples can be adjusted based on the bucket capacity and key distribution. If selecting a split key results in either side not containing a valid key, a new split key is selected; alternatively, the median key of the current set of valid keys can be used directly. Subsequently, valid key-value pairs with keys less than the split key s are written to the first new bucket B1, and valid key-value pairs with keys greater than or equal to the split key s are written to the second new bucket B2. Corresponding volatile mapping structures M1 and M2 are then established or registered as needed. The smallest valid key in the original bucket becomes the smallest valid key in the first new bucket B1, and the split key s becomes the smallest valid key in the second new bucket B2. When the query key equals the split key s, the query is directed to the second new bucket B2.

[0043] In one optional embodiment, the first new bucket B1 and the second new bucket B2 are marked as being in a build state during construction. At the start of the split, the bucket version and record sequence number high-water mark of the bucket B to be split are recorded and temporarily serialized using a bucket-level write lock, or the tail records after the high-water mark are replayed to the corresponding new bucket after the new bucket is built. After the two new buckets and their description information are persisted, a commit record containing the first new bucket B1, the second new bucket B2, and the target route version is written. The first-level routing entries and adjacent bucket links are updated through versioned commits or atomic metadata switching, so that the forward bucket previously pointing to the original bucket B points to the first new bucket B1, the first new bucket B1 links to the second new bucket B2, and the second new bucket B2 links to the backward bucket of the original bucket. Subsequently, the original bucket B is marked as invalid and released when the reclamation conditions are met. During restart recovery, only buckets that are in a valid state and referenced by committed route versions are added to the valid bucket chain.

[0044] During index initialization, a batch loading can be performed based on existing index keys, distributing the sorted consecutive key fields to multiple initial buckets; alternatively, a single initial empty bucket can be created that does not participate in ordered routing. When using a single initial empty bucket, the first insertion request sets its index key to the minimum valid key of that bucket and creates the first level-one routing entry, subsequently expanding gradually through insertions and local splits. This invention does not limit the number of initial buckets, the initial fill rate, or the specific loading algorithm. When the level-one ordered routing layer 110 is missing, the level-one routing entry is inconsistent with the valid bucket state, or the amount of change in routing entries due to additions, deletions, and splits reaches the preset refresh conditions, the splitting and refresh module 140 reads the bucket state, bucket version, minimum valid key, and bucket identifier of each bucket, selects the second-level append-only persistent bucket 121 that is in a valid state and referenced by the submitted routing version, and reconstructs the ordered arrangement of the level-one routing entries 111. This refresh operation does not require traversing all valid key-value items; if no valid bucket exists, an initial empty bucket is created. Example 3

[0045] See Figure 4 This embodiment addresses volatile mapping quantity control and reconstruction. Given limited volatile storage space, the mapping management and reconstruction module 150 sets an upper limit on the number of resident mappings or a memory usage limit. When the established volatile mapping structure reaches this upper limit, at least one low-hot mapping is released according to a least recently used strategy, access frequency strategy, or other cache replacement strategy. Only the volatile mapping structure 131 is released; appended records in the secondary append-only persistent bucket 121 remain in non-volatile storage. When a subsequent query accesses the corresponding bucket again, the mapping management and reconstruction module 150 records the current bucket version and the upper bound of the record sequence number, and scans for changes not exceeding the upper bound and fully committed according to the record sequence number within the bucket. For inserted and updated records, the metadata value in the record is written to a temporary mapping, with later-written records overwriting earlier-written records for the same key; for deleted records, the corresponding key is removed from the temporary mapping. After the scan is complete, the appended tail records after the upper bound are replayed, and after the bucket version verification passes, the temporary mapping is registered as a new volatile mapping structure 131.

[0046] When a write request accesses a bucket with a missing mapping, a change record can be directly appended to the persistent bucket. When the bucket is subsequently rebuilt, the rebuild process scans all valid change records, including the newly appended record, to obtain the latest state. In another optional implementation, if the write operation needs to immediately read the current valid state within the bucket, the mapping can be rebuilt first before performing the write operation.

[0047] In the event of abnormal process exit, runtime environment restart, or loss of runtime index state in volatile storage, the above reconstruction process can be performed on the valid secondary append-only persistent bucket 121. During recovery, tail records with failed verification, invalid commit tags, or incomplete length fields are ignored, and only committed route versions are used. Reconstruction of different buckets is independent, and therefore can be processed in parallel by different execution threads. This embodiment assumes that the persistent medium can be read normally and does not involve data redundancy recovery in the event of physical damage to the storage medium. Example 4

[0048] This embodiment provides an optional implementation of the index structure. The first-level ordered routing layer 110 can be implemented using an ordered array, skip list, B-tree, or other ordered routing directories. The second-level append-only persistent bucket layer 120 can be deployed on non-volatile memory, solid-state storage, or disk storage. The rebuildable volatile mapping layer 130 can employ an unordered mapping, hash table, or other data structure that supports point lookup and replacement. This metadata index can receive requests from file system metadata services, distributed storage metadata services, or general key-value metadata management programs. The index key can be a numeric key, a composite key, a name digest, or other metadata key that can be compared and partitioned, and uses a defined total order comparison rule. When using a name digest as the index key, the original name or collision check information can be saved simultaneously, and a secondary comparison can be performed to distinguish different metadata items if the digests are the same. The metadata value can be a metadata record identifier, object identifier, location identifier, or metadata content.

[0049] This invention does not limit the first-level ordered routing layer 110 to reside entirely in volatile memory. In one embodiment, a working copy of the first-level routing entry resides in volatile memory, while the minimum valid key, bucket identifier, bucket state, and version information of the valid bucket are stored in persistent media, enabling the first-level ordered routing layer to be reconstructed based on the valid bucket description information.

[0050] The scope of protection of this invention is not limited to the examples described above. All variations and advantages conceivable in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A metadata indexing method using append-only persistent buckets and reconstructable volatile mappings, characterized in that, Metadata indexing includes the following steps: Step 1: Establish multiple append-only persistent buckets with sequential key field links in a non-volatile storage medium, maintain a first-level ordered routing entry based on the least valid key of the multiple append-only persistent buckets, and establish volatile mappings corresponding to the accessed buckets in the volatile memory as needed. Step 2: Locate the target append-only persistent bucket based on the first-level ordered routing entry, append the change record corresponding to the write request to the target append-only persistent bucket, and synchronously update the volatile mapping when the corresponding volatile mapping has been established; Step 3: Locate the target append-only persistent bucket based on the first-level ordered routing item, and obtain the latest valid metadata value of the query key when the corresponding volatile mapping has been established; Step 4: If the volatile mapping does not exist, scan and replay the change records in the target append-only persistent bucket in the order of writing, establish the volatile mapping and complete the point query; Step 5: When the target append-only persistent bucket reaches the preset capacity threshold, determine the split key based on the current valid key set of the bucket, construct and persist the first new bucket and the second new bucket, switch the corresponding first-level ordered routing entries and adjacent bucket links, and mark the original target append-only persistent bucket as invalid.

2. A metadata indexing system constructed using the append-only persistent bucket and reconstructable volatile mapping metadata indexing method as described in claim 1, characterized in that, The metadata indexing system includes: a first-level ordered routing layer, a non-volatile memory, a volatile memory, a mapping management and reconstruction module, and a splitting and refreshing module. The first-level ordered routing layer maintains ordered routing entries based on the minimum valid key of multiple second-level append-only persistent buckets, and each routing entry records at least the minimum valid key and bucket identifier of the corresponding second-level append-only persistent bucket. The non-volatile memory contains multiple second-level append-only persistent buckets linked sequentially by key fields, and each second-level append-only persistent bucket stores the corresponding metadata change records for insertion, update, or deletion in an append-only manner. The volatile memory contains multiple rebuildable volatile mapping structures, each corresponding one-to-one with the accessed second-level append-only persistent bucket, used to record the latest valid metadata value corresponding to the index key. The mapping management and reconstruction module manages the residence, release, and reconstruction of rebuildable volatile mapping structures. The splitting and refreshing module performs partial splitting on second-level append-only persistent buckets that reach a preset capacity threshold and refreshes the first-level ordered routing layer. When a write request is made, the metadata indexing system locates the target second-level append-only persistent bucket through the first-level ordered routing layer. The system persists the data in a persistent bucket and appends the corresponding change records to the target secondary append-only persistent bucket. When the corresponding reconstructable volatile mapping structure is already established, it is updated synchronously. When the reconstructable volatile mapping structure is not yet established, the change records are included in the current valid state obtained from subsequent reconstruction. A query is used to locate the target secondary append-only persistent bucket via the first-level ordered routing layer. When the corresponding reconstructable volatile mapping structure is already established, the latest valid metadata value corresponding to the query key is obtained from it. When the reconstructable volatile mapping structure does not exist, the change records in the target secondary append-only persistent bucket are scanned according to the record writing order, and the corresponding reconstructable volatile mapping structure is rebuilt. When the target secondary append-only persistent bucket reaches a preset capacity threshold, the splitting and refreshing module determines the split key based on the current valid key set of the bucket, writes valid key-value items smaller than the split key to the first new bucket, and writes valid key-value items greater than or equal to the split key to the second new bucket. After the persistence of the first and second new buckets is completed, the corresponding first-level routing item and adjacent bucket links are switched, and the original target secondary append-only persistent bucket is marked as invalid.

3. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 2, characterized in that, The secondary append-only persistent bucket adopts an append-only record layout. Each change record includes at least: an index key, metadata value, operation type, and a monotonically increasing record number within the bucket used to determine the order of writing. When rebuilding the reconstructable volatile mapping structure, the rule of overwriting earlier written records with later written records is applied to the same index key, and the corresponding index key is removed according to the deletion marker record. The data stored in the secondary append-only persistent bucket constitutes the basis for the persistence of the index state.

4. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 2, characterized in that, The first-level ordered routing layer selects the second-level append-only persistent bucket corresponding to the last routing entry whose minimum valid key is not greater than the index key as the target bucket. When the index key is less than the minimum valid key of the first valid bucket, the query returns no result, the insertion request is located in the first valid bucket and its minimum valid key is updated. When there is no valid second-level append-only persistent bucket, an initial empty bucket is created, and when the first insertion request is received, the index key of the insertion request is determined as the minimum valid key of the initial empty bucket and the corresponding first-level routing entry is created.

5. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 2, characterized in that, The reconstructable volatile mapping structure is a runtime query structure that can be recovered based on the change records in the corresponding bucket. Set a threshold for the number of residents or a threshold for memory usage of the reconstructable volatile mapping structure. When the number of re-established re-reconstructable volatile mapping structures reaches the maximum number of resident structures or the maximum memory usage limit, at least one re-reconstructable volatile mapping structure is released according to the least recently used strategy, and the released re-reconstructable volatile mapping structure is rebuilt when the corresponding secondary append-only persistent bucket is accessed again.

6. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 2, characterized in that, The non-volatile memory includes at least one of non-volatile memory, solid-state storage device, or disk storage device; the volatile memory includes dynamic random access memory.

7. The append-only persistent bucket and reconstructable volatile mapping metadata indexing system according to claim 2 or claim 3, characterized in that, When the secondary append-type persistent bucket reaches a preset capacity threshold, multiple sampling keys are selected from the current valid key set of the bucket, the sampling keys are sorted and the split key is determined, and valid key-value items smaller than the split key are written into the first new bucket, and valid key-value items greater than or equal to the split key are written into the second new bucket; when the valid key set of any new bucket is empty, a new split key is selected, or the median key of the current valid key set is used as the split key.

8. The append-only persistent bucket and reconstructable volatile mapping metadata indexing system according to claim 2 or claim 4, characterized in that, When the first-level ordered routing layer is missing, a route entry is detected that is inconsistent with a valid second-level append-only persistent bucket, or the amount of change in the route entry reaches a preset refresh threshold, it selects a valid second-level append-only persistent bucket that is referenced by a submitted route version and reconstructs the first-level ordered routing layer based on the bucket status, bucket version, bucket identifier, and minimum valid key of each second-level append-only persistent bucket.

9. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 4, characterized in that, When the index key of the insertion request is less than the current minimum valid key of the target second-level append-only persistent bucket, the minimum valid key of the bucket and the corresponding routing entry are updated; when the deletion request deletes the minimum valid key, a new minimum valid key is determined from the current set of valid keys represented by the corresponding reconstructable volatile mapping structure, or a new minimum valid key is determined by replaying the change records in the bucket if the reconstructable volatile mapping structure does not exist; when the target bucket has no valid key and other valid buckets still exist, the routing entry corresponding to the bucket is deleted or invalidated and the links of adjacent buckets are adjusted, or the bucket is marked as an empty bucket for subsequent reuse; when the target bucket is the last valid bucket, an initial empty bucket is retained or re-established to receive the first subsequent insertion request.

10. The metadata indexing system with append-only persistent buckets and reconstructable volatile mappings according to claim 5, characterized in that, The secondary append-only persistent bucket and the bucket description area of ​​the new bucket include: bucket status and bucket version information. When splitting, the first new bucket and the second new bucket are marked as being in the construction state. The first new bucket, the second new bucket and the corresponding description information are constructed and persisted. A commit record containing the new bucket identifier and the target route version is generated. The primary route item and the adjacent bucket link are updated through versioned commit or atomic metadata switching. The original bucket to be split is marked as invalid. When rebuilding, only the secondary append-only persistent buckets with valid status and referenced by the committed route version are processed.