Metadata management method and electronic device

Abstract

The application discloses a metadata management method and electronic equipment, and relates to the technical field of computers, wherein a tree identification bit is arranged in a logical block address of metadata, in the process of additional writing of the metadata, with the change of the logical block address of the written metadata, the sequentially written metadata can be allocated to different tree index structures, the continuous data originally concentrated in one tree is dispersed to multiple trees, multiple-tree parallel writing is realized, and the writing performance of the metadata is effectively improved; meanwhile, when the metadata is queried, traversal is started from a root node instead of depending on horizontal pointers between leaf nodes, since the metadata originally concentrated in one tree is dispersed to multiple trees, the traversal efficiency from the root node is improved, and the influence of the need to update the horizontal pointers of adjacent leaf nodes on the writing performance in the additional writing scenario is avoided, so that the technical effect that the reading and writing performances of the metadata are combined is achieved, and the high-performance storage architecture suitable for a mass high-concurrency scenario is adapted.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a metadata management method and electronic device. Background Technology

[0002] With the surge in stored data in storage systems, metadata management faces severe challenges. Traditional tree-structured metadata management methods result in increasingly large tree indexes. If every query requires traversing from the root node, query efficiency is severely impacted. Therefore, traditional solutions employ B+ trees and their variants, using horizontal pointers between leaf nodes to improve query efficiency. However, while horizontal pointers between leaf nodes improve query efficiency, they are unsuitable for the widely used metadata append-only scenarios. When metadata is written to disk, updating the pointers of adjacent leaf nodes simultaneously requires updating the leaf nodes, significantly impacting metadata write performance. Summary of the Invention

[0003] This invention provides a metadata management method and electronic device to at least solve the problems in related technologies where traversing from the root node of a tree index structure results in low query efficiency and using horizontal pointers between leaf nodes affects metadata write performance.

[0004] This invention provides a metadata management method, comprising: In response to an append write request for metadata, determine the logical block address of the metadata; The tree index structure corresponding to the metadata is determined based on the value of the tree identifier bit in the logical block address of the metadata; Write the metadata into the corresponding tree index structure; In response to a query request for target metadata, the target tree index structure is determined based on the target logical block address of the target metadata; Traverse the target tree index structure starting from the root node to query the target metadata.

[0005] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described metadata management methods when executing the computer program.

[0006] By setting a tree identifier in the logical block address of metadata, during the append writing process, as the logical block address of newly written metadata changes, the metadata written sequentially can be allocated to different tree index structures. This disperses continuous data originally concentrated in one tree to multiple trees, enabling parallel writing of multiple trees and effectively improving metadata write performance. Simultaneously, when querying metadata, traversal starts from the root node instead of relying on horizontal pointers between leaf nodes. Since the metadata originally concentrated in one tree is dispersed across multiple trees, the efficiency of traversing from the root node is improved, and the impact of updating the horizontal pointers of adjacent leaf nodes on write performance in append writing scenarios is avoided. This achieves a technical effect that combines read and write performance of metadata, adapting to high-performance storage architectures for massive high-concurrency scenarios. Attached Figure Description

[0007] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 An architecture diagram of a metadata management system provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a metadata management method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a tree index structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a metadata key provided in an embodiment of the present invention; Figure 5 This is a storage architecture diagram of a root node address provided in an embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0010] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] Here, we will first explain some key terms used in the embodiments of the present invention.

[0013] Metadata, as a core component of a storage system, is crucial data describing information such as the attributes and location of data objects. Metadata is used to describe data; when data is written to storage, corresponding metadata needs to be written synchronously, and when data is read, its location on the disk needs to be found through metadata. Therefore, the access performance of metadata directly determines the input / output (IO) response time, throughput, and user experience of the entire storage system. With the development of big data, cloud computing, and distributed storage technologies, the scale of metadata that storage systems need to manage is growing exponentially. Simultaneously, upper-layer applications are placing stringent demands on metadata access—low latency, high concurrency, and high reliability. Efficient metadata management has become a core technology for overcoming storage system performance bottlenecks.

[0014] Currently, the mainstream metadata organization and management method in storage systems is the B-tree. B-trees include various variant structures, such as B-trees, B+ trees, and B-trees. Among them, the B+ tree, due to its advantages such as ordered key (supporting efficient queries), disk-access friendly operation (node ​​size is typically 4KB or 512 bytes, aligned with the disk's input / output block size), support for efficient range and point queries, and a mature node splitting and merging mechanism, is well-suited to the storage and retrieval needs of metadata and has become the preferred technology solution for metadata indexing.

[0015] In traditional B-tree structures, all query operations must start from the root node and traverse the tree structure layer by layer down to the leaf nodes. As the size of the tree index structure increases, the tree height also increases, and the path length from the root node to the leaf node increases accordingly, severely impacting query efficiency. To overcome this problem, related technologies propose establishing horizontal pointers (i.e., pointers to adjacent leaf nodes) between leaf nodes. This allows range queries to directly jump between leaf nodes using horizontal pointers, eliminating the need to traverse from the root node again, thus significantly improving query efficiency.

[0016] However, while maintaining horizontal pointers between leaf nodes improves metadata access performance, there are still many technical drawbacks in real-world scenarios involving massive storage and high concurrency access, especially in all-flash storage environments.

[0017] Specifically, metadata is typically written to disk in an append-only manner, meaning that when a metadata node is updated, it is written to a new physical address rather than overwritten in place. The leaf nodes of a traditional B+ tree contain horizontal pointers between leaf nodes, a structure unsuitable for append-only metadata writing. This is because when metadata nodes are written to disk in an append-only manner, updating a leaf node requires writing it to a new physical address. This necessitates updating not only the pointers to that leaf node in the root node and intermediate nodes, but also the horizontal pointers in the adjacent leaf nodes. This mechanism results in a large number of additional random write operations, severely impacting the write performance of the storage system and failing to meet the storage system's requirements for low latency and high Input / Output Operations Per Second (IOPS).

[0018] To address the issues of low query efficiency when traversing from the root node of a tree index structure and the impact of using horizontal pointers between leaf nodes on metadata write performance in related metadata management schemes, this invention provides a metadata management method and electronic device. By setting a tree identifier bit in the logical block address of metadata, during the append writing process, as the logical block address of newly written metadata changes, the sequentially written metadata can be allocated to different tree index structures. This disperses continuous data originally concentrated in one tree across multiple trees, enabling parallel writing across multiple trees and effectively improving metadata write performance. Simultaneously, when querying metadata, traversal starts from the root node instead of relying on horizontal pointers between leaf nodes. Since the metadata originally concentrated in one tree is dispersed across multiple trees, the efficiency of traversing from the root node is improved, and the impact of updating the horizontal pointers of adjacent leaf nodes in append writing scenarios on write performance is avoided. This achieves a technical effect that balances metadata read and write performance, adapting to high-performance storage architectures for massive high-concurrency scenarios.

[0019] Figure 1 This is an architecture diagram of a metadata management system provided in an embodiment of the present invention.

[0020] The specific application environment architecture or specific hardware architecture on which the execution of the metadata management method depends is described here.

[0021] like Figure 1As shown, the metadata management system upon which the metadata management method provided in this embodiment of the invention is based may include a storage controller, non-volatile memory, cache, disk, etc. The non-volatile memory is connected to the storage controller and is used to persistently store the root object. The root object includes the root node addresses of each tree-like index structure managed by the storage controller. The root node address includes at least the disk address of the root node on the disk, and may also include the cache address updated after the root node data is loaded into the cache.

[0022] The cache is connected to the storage controller. After node data in the tree-indexed structure is loaded, it can be temporarily stored in the cache. For cached node data, root node data is prohibited from being evicted; leaf node data is evicted first.

[0023] The disk connects to the storage controller and is used to persistently store the data of each node in the tree index structure, including the complete data of the root node, intermediate nodes, and leaf nodes. The tree index structure can adopt a B-tree family structure, and the metadata nodes are written to disk in an append-only manner, that is, when a node is updated, it is written to a new physical address rather than overwriting it in place.

[0024] To improve the reliability of the root node address, the metadata management system upon which the metadata management method provided in this embodiment of the invention is based can employ multiple storage nodes, for example... Figure 1 The dual-controller storage system shown includes a first storage node comprising a first storage controller, a first non-volatile memory, a first cache, and a first disk; and a second storage node comprising a second storage controller, a second non-volatile memory, a second cache, and a second disk. The first and second storage nodes achieve mirror synchronization via a dual-controller communication channel. When the root node address in either storage node is updated, the updated root node address is synchronized to the root object of the other storage node, thus achieving redundancy protection for the root node address.

[0025] The embodiments of the present invention provide a metadata management method. The method is described in detail below in conjunction with the execution flow of the metadata management method.

[0026] Figure 2 A flowchart illustrating a metadata management method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a tree index structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a metadata key provided in an embodiment of the present invention.

[0027] like Figure 2 As shown, the metadata management method provided in this embodiment of the invention may include: S201: in response to an append write request for metadata, determining the logical block address of the metadata.

[0028] S202: Determine the tree index structure corresponding to the metadata based on the value of the tree identifier bit in the logical block address of the metadata.

[0029] S203: Write the metadata into the corresponding tree index structure.

[0030] S204: In response to a query request for target metadata, determine the target tree index structure based on the target logical block address of the target metadata.

[0031] S205: Traverse from the root node of the target tree index structure to query the target metadata.

[0032] This invention provides a metadata management strategy based on spatial segmentation, which divides the metadata space to be managed into multiple tree-like index structures. By dividing it into multiple tree-like index structures, segmented management of the metadata region can be achieved. For example, the height of each segment's tree-like index structure can be controlled within 6 levels to improve the access efficiency of the tree-like index structure.

[0033] In this embodiment of the invention, the tree index structure is one that requires traversal starting from the root node during querying. For example... Figure 3 As shown, the tree index structure can be a B-family tree, such as a B-tree, B+ tree, B-tree, etc., and no horizontal pointers are set between the leaf nodes of the tree index structure.

[0034] To achieve spatial segmentation management, embodiments of this invention provide a key structure for metadata. For example... Figure 4 As shown, the metadata key can consist of a volume identifier (Logical Unit Number ID, LUN ID) and a volume logical block address (LBA). The volume identifier is located in the high-order bits of the binary representation (e.g., Figure 4 The 38th to 54th bits shown are used to distinguish metadata from different volumes.

[0035] In this embodiment of the invention, the logical block address of the volume is split, including at least a tree identifier bit. Therefore, as metadata is appended and written, the logical block address of the written metadata changes continuously (e.g., increments sequentially), causing a change in the tree identifier bit, thereby allocating the metadata to different attribute index structures. Since the tree identifier bit is located at a specific position in the logical block address, during the incrementing process of consecutive logical block addresses, a carry occurs in the lower bits. When the carry is passed to the tree identifier bit, the value of the tree identifier bit changes, thereby allocating subsequent metadata to different tree index structures.

[0036] For example, if the tree identifier can be located in the low-order region of the logical block address, then two adjacent metadata on the logical block address will be assigned to different tree index structures until the logical block address carries over to the high-order region. Metadata that are not adjacent on the logical block address may be reassigned to the same tree index structure.

[0037] In this embodiment of the invention, the tree identifier can be located in the high-order region of the logical block address, so that when the logical block address increments, the tree identifier changes when a carry-over occurs from the preceding low-order bit. By setting the tree identifier in the high-order region of the logical block address, the metadata of a continuous logical block address segment corresponding to the low-order region before the tree identifier will be allocated to the same tree index structure. The tree identifier only changes when a carry-over occurs from the low-order region. This allows a large range of continuous logical block address space to be distributed across multiple tree index structures, while maintaining a small range of continuous logical block addresses within the same tree index structure, thus achieving both I / O concurrency and local contiguity.

[0038] In some optional embodiments of the present invention, the logical block address may further include a page identifier bit and a page offset bit, wherein the page offset bit, page identifier bit, and tree identifier bit are arranged sequentially from the least significant bit to the most significant bit of the logical block address. Specifically, the page offset bit is used to identify the offset within a page of the tree index structure, the page identifier bit is used to identify the page number in the tree index structure, and the tree identifier bit is used to identify the tree index structure number. Through this division, when consecutive logical block addresses change within the range of the page offset bit, the metadata falls within the same page; when the page offset bit is full, it carries over to the page identifier bit, and the metadata changes within the same page of the same tree; when the page identifier bit is full, it carries over to the tree identifier bit, and the metadata is allocated to the next tree. This metadata key structure allows for flexible control of the granularity of metadata distribution, improving the configurability of the storage system.

[0039] In some alternative embodiments of the present invention, the logical block address may further include a high-order offset bit, with the page offset bit, page identifier bit, tree identifier bit, and high-order offset bit arranged sequentially from the low-order bit to the high-order bit of the logical block address. For example... Figure 4 As shown, the high-order offset bit is located in the highest bit region of the logical block address, and together with the low-order offset bit, they form a discontinuous disk key. The high-order offset bit is used for large-granular control across the tree, further expanding the capacity and distribution flexibility of the tree index structure.

[0040] Furthermore, such as Figure 4As shown, in this embodiment of the invention, the metadata key can be constructed as follows: First, the metadata key is combined, consisting of the volume identifier (bits 54-38) and the volume's logical block address (bits 37-0). Then, the volume's logical block address is split into four parts: a high-order offset bit, a tree identifier bit (Tree ID), a page identifier bit (Page ID), and a low-order offset bit. The high-order and low-order offset bits together constitute the disk key. The disk key is discontinuous; that is, the disk key is not continuously distributed in the binary representation of the logical block address, but is divided into two or more discontinuous segments by the tree identifier bit and the page identifier bit. This discontinuity of the disk key prevents a large contiguous logical block address space from belonging to the same tree, instead allowing multiple small contiguous spaces (not adjacent to each other) to be distributed within the same tree. The purpose of this is to distribute I / O requests as evenly as possible across multiple tree index structures. Since each tree index structure is bound to a central processing unit (CPU) core, the I / O load is evenly distributed across multiple CPU cores, achieving the technical effect of I / O concurrency.

[0041] In this embodiment of the invention, multiple tree index structures can be uniformly managed by a root page management structure. The root page can manage the root node information of multiple trees (e.g., 128 trees), primarily for centralized management of root nodes. The root page stores the disk address of each tree's root node on disk, and can also store the memory address (or cache address) of the root node in memory. Through the root page, the root node of the corresponding tree can be quickly accessed based on the tree identifier: first, the corresponding root node address is searched in the root page based on the tree identifier; if the root page stores a memory address, the root node data is directly read from the cache; if it stores a disk address, the root node data is read from the disk and loaded into the cache, while simultaneously updating the address in the root page to the cache address. The root page is stored in non-volatile memory, possessing the characteristic of being resistant to power loss, ensuring that the root node address information is not lost after a system restart. The root page, combined with the tree identifier (Tree ID), is unique throughout the entire system; these two numbers uniquely identify the disk address and memory address of the root node of the corresponding tree index structure.

[0042] Based on any of the metadata key structures provided in the embodiments of the present invention, for S201, in response to the metadata append write request, the volume where the metadata is located and the address of the allocated logical block are determined, and the complete key of the metadata is organized according to the structure of the values ​​of any of the above-mentioned metadata.

[0043] For S202, the tree index structure corresponding to the metadata is determined based on the value of the tree identifier bit in the logical block address of the metadata. Specifically, the tree identifier bit (e.g., bits 15-21) is extracted from the logical block address, and its value is used as the number of the tree index structure. Since the tree identifier bit is located in the high-order region of the logical block address, the value of the tree identifier bit changes periodically as consecutive logical block addresses increment, thereby evenly distributing the metadata across multiple different tree index structures and achieving parallel processing of the IO load.

[0044] For S203, the metadata is written to the corresponding tree index structure. First, the root node address information of the target tree is obtained from the root page; then, the root node data is read from the cache or disk according to the root node address; next, starting from the root node, the tree index structure is traversed downwards, and the target leaf node is located according to the page identifier bit and page offset bit in the logical block address; finally, the key pairs of the metadata are written to the leaf node.

[0045] For S204, in response to a query request for target metadata, the target tree index structure is determined based on the target logical block address of the target metadata. Specifically, the target logical block address is extracted from the query request, and the value of the tree identifier bit is extracted from the target logical block address according to the same key structure as when writing, and the number of the target tree index structure is determined based on this value.

[0046] For S205, traversal begins from the root node of the target tree index structure to query the target metadata. Since the tree index structure in this embodiment does not have horizontal pointers between leaf nodes, all queries must start from the root node. However, through the metadata management strategy based on spatial segmentation provided in this embodiment, each tree index structure is relatively small, allowing for rapid querying of target node data after locating the target tree index structure. This significantly improves query efficiency compared to traditional solutions that start traversing from the root node of a large-scale tree index structure.

[0047] The metadata management method provided in this invention sets a tree identifier bit in the logical block address of metadata. During the append writing process, as the logical block address of newly written metadata changes, the sequentially written metadata can be allocated to different tree index structures. This disperses continuous data originally concentrated in one tree across multiple trees, enabling parallel writing across multiple trees and effectively improving metadata write performance. Simultaneously, when querying metadata, traversal starts from the root node instead of relying on horizontal pointers between leaf nodes. Since the metadata originally concentrated in one tree is dispersed across multiple trees, the efficiency of traversing from the root node is improved, and the impact of updating the horizontal pointers of adjacent leaf nodes on write performance in append writing scenarios is avoided. This achieves a technical effect that combines read and write performance of metadata, adapting to high-performance storage architectures for massive high-concurrency scenarios.

[0048] Based on the above embodiments, the metadata management method provided by the present invention may further include: storing the queried metadata in a cache; and managing the eviction of metadata stored in the cache according to the hierarchical priority of the tree index structure, wherein the eviction of root node data of the tree index structure is prohibited, the eviction of leaf node data of the tree index structure is prioritized, and the eviction of intermediate node data of the tree index structure is followed by the eviction of intermediate node data of the tree index structure.

[0049] Traditional metadata caching designs employ the same eviction policy for all nodes, such as the Least Recently Used (LRU) policy. Since cache space is limited and insufficient to cache all metadata nodes, a reasonable eviction policy needs to be designed to maximize cache hit rate and access efficiency. However, this invention eliminates the horizontal pointers between leaf nodes, meaning all traversal operations must pass through the root node. This further amplifies the root node's importance in the entire metadata space, as every metadata query requires a query through the root node. Therefore, this invention employs a differentiated eviction policy: never evicting the root node, minimizing the evicting of intermediate nodes, and prioritizing the evicting of leaf nodes. Thus, the root node, as the path for all queries, is permanently retained in the cache to avoid frequent disk accesses and significantly reduce query latency; leaf nodes, being the most numerous and with the most dispersed access patterns, are prioritized for eviction to promptly release cache space and improve cache utilization.

[0050] The eviction strategies used in this invention include, but are not limited to, LRU, Least Frequently Used (LFU), and First In First Out (FIFO) algorithms. Those skilled in the art can select an appropriate eviction strategy as the base algorithm based on the actual application scenario and combine it with the hierarchical priority proposed in the embodiments of this invention.

[0051] The eviction management of metadata stored in the cache is then carried out according to the hierarchical priority of the tree index structure. This can include: eviction of metadata of the same priority based on the order of their most recent access time. In other words, within the same priority level, an LRU (Least Recently Used) strategy is used for eviction: the metadata node with the earliest access time (i.e., the one that has not been accessed for the longest time) is evicted first. For example, for multiple leaf nodes, the leaf node with the earliest access time is evicted first. This further utilizes the principle of temporal locality on top of hierarchical priority to improve the cache hit rate.

[0052] Alternatively, metadata stored in the cache can be evicted according to the hierarchical priority of the tree index structure. This can include: eliminating metadata of the same priority level based on its access frequency. That is, within the same priority level, an LFU (Least Frequently Used) strategy is employed: the metadata node with the fewest accesses (i.e., the lowest access frequency) is evicted first. For example, for multiple leaf nodes, the leaf node with the fewest accesses is evicted first. This further leverages the principle of frequency locality, based on hierarchical priority, to retain frequently accessed data in the cache for a longer period.

[0053] In some optional embodiments of this invention, intermediate node data at different levels can be set to the same priority. That is, all intermediate nodes (regardless of their level in the tree index structure) have the same eviction priority in the eviction policy, higher than the root node (which is prohibited from eviction) and lower than the leaf nodes (which are prioritized for eviction). For example, based on the level of the node, let the root node be at level 0, the intermediate nodes at levels 1 to 4, and the leaf nodes at level 5. The root node at level 0 is not evicted, the leaf nodes at level 5 are prioritized for eviction, and the intermediate nodes at levels 1 to 4 have the same eviction priority. Of course, for leaf nodes at the same priority level (e.g., level 5) between different tree index structures, an eviction policy such as LRU is also used to prioritize evicting leaf nodes with less frequently accessed data.

[0054] In some alternative embodiments of the present invention, the higher the level of the intermediate node data, the lower the elimination priority. That is, the intermediate nodes can be further subdivided, with intermediate nodes closer to the root node (higher level) being less likely to be eliminated, and intermediate nodes closer to the leaf nodes (lower level) being more likely to be eliminated. For example, if the root node is located at level 0 (lowest elimination priority, eliminated), the elimination priority of intermediate nodes at level 1 is lower than that of intermediate nodes at level 2, the elimination priority of intermediate nodes at level 2 is lower than that of intermediate nodes at level 3, and so on, with the leaf node level having the highest elimination priority.

[0055] Figure 5 This is a storage architecture diagram of a root node address provided in an embodiment of the present invention.

[0056] Based on the above embodiments, the metadata management method provided by the present invention may further include: persistently storing the address of the root node data of the tree index structure; the address of the root node data includes at least the disk address of the root node data.

[0057] In other words, a root management structure (i.e., the root page described in the above embodiment) is maintained in non-volatile memory to store the address information of the root nodes of each tree index structure, ensuring that the root node addresses are not lost after a power outage or restart. As the entry point for all query operations, the persistent storage of the root node's address information ensures that the system can quickly locate the root node after fault recovery, avoiding the unavailability of the entire metadata index due to the loss of the root node address.

[0058] The metadata management method provided in this embodiment of the invention may further include: after the root node data is queried for the first time after system startup, storing the root node data in a cache and updating the cache address of the root node data to the address of the root node data. That is, in the initial state, the root management structure stores the disk address of the root node on the disk; when the root node data is accessed for the first time and loaded into the cache, the address in the root management structure is updated to the memory address of the root node in the cache. Subsequent queries can directly read the root node data from the cache through the memory address, without needing to access the disk again. By dynamically updating the root node address from the disk address to the cache address, all subsequent root node accesses can directly hit the cache, significantly reducing root node access latency and improving overall query efficiency.

[0059] Storing root node data in a cache can include storing root node data in an array within the cache.

[0060] like Figure 5As shown, in the root management (rootPage) data structure of this embodiment, a cached pointer address of the root node (rootBtreeNode) is added to further accelerate root address access. Specifically, the root management structure stores 128 root node memory addresses and 128 root node disk addresses in a tree-indexed structure, which are used for fast access to root node data in the cache and root node data on the disk, respectively. Since cache lookup is less efficient than array lookup, and the root node is the most frequently accessed node, root node data can be stored in array form, thereby further improving the root node access speed.

[0061] In this embodiment of the invention, persistently storing the address of the root node data of the tree index structure may include: persistently storing the address of the root node data to at least two storage nodes respectively; when the address of the root node is updated, synchronously updating the data structure of the root node address in at least two storage nodes; when a storage node fails, after the storage node failure is recovered, restoring the address of the root node based on the data in the storage node that has not failed.

[0062] Specifically, the root node physical address adopts a highly reliable non-loss design (i.e., power-loss protection design) to ensure that the root node address is not lost under any fault scenario.

[0063] like Figure 1 As shown, the metadata management system used in this embodiment of the invention can be a dual-controller storage system. When the disk address of the root node is updated, the storage node performing the update operation will synchronize the address to the same data structure (i.e., the root object) in the other storage node. In this way, the root node address achieves mirror redundancy protection between the two storage nodes. When the root node is refreshed, the disk address of the corresponding root object is updated, and the address is synchronized to the root object of the other node, maintaining a mirror protection relationship between the two. When either storage node fails, the other storage node still retains complete root node address information, avoiding the risk of data loss due to a single point of failure.

[0064] When any storage node fails, it restarts and recovers the root node address from another storage node that did not fail.

[0065] If two storage nodes fail simultaneously, recovery will proceed based on the last node to fail. The node that fails first will wait for the second node to recover before synchronizing. Because the root node address is protected against power loss, data can be recovered from this protected area upon restarting. The system only recognizes data from the last node to fail, as data from the first node may be outdated. Therefore, in the extreme scenario of simultaneous dual-controller failure, this recovery strategy prioritizes the last node to fail, ensuring that the latest version of the root node address is restored and avoiding data inconsistencies caused by using outdated addresses.

[0066] Based on the above embodiments, in this embodiment of the invention, traversing from the root node of the target tree index structure to query target metadata in S205 may include: determining the target root node address from the root node address structure of the target tree index structure; if the target root node address includes a cache address, then querying the target root node data from the cache; if the target root node address only contains a disk address, then obtaining the target root node data according to the disk address and storing the target root node data in the cache; traversing downwards along the target tree index structure according to the target root node data to query the target leaf node data, wherein if the intermediate node data or leaf node data of the target tree index structure encountered during traversal is not in the cache, then storing the corresponding intermediate node data or leaf node data in the cache; and using the target leaf node data as target metadata.

[0067] Specifically, metadata lookup can include: finding the corresponding value based on a key. The lookup process can include: ① First, assembling a key based on the logical block address and volume identifier issued by the upper-layer business.

[0068] ② Based on the tree partitioning strategy, determine which page ID and tree ID the key value belongs to, thereby determining which tree index structure to search for. Specifically, extract the tree ID and page ID from the binary representation of the logical block address. The value of the tree ID determines the number of the target tree index structure, and the value of the page ID determines the number of the target page within the tree.

[0069] ③ First, find the memory address of the root node of the corresponding tree in the root object, such as... Figure 5 As shown by arrow ①, the root node data is retrieved via array lookup. The root object stores the memory addresses and disk addresses of the root nodes in a tree-indexed structure, organized in array form, and can be directly addressed using tree identifiers. If the system has just started (i.e., the read cache is empty), proceed directly to step ⑥.

[0070] ④ Find the value of the corresponding disk key based on the root node. Start from the root node and traverse downwards until you find the leaf node containing the disk key. If an intermediate or leaf node is not in the cache during the search, you need to find the corresponding node on the disk and then insert it into the read cache.

[0071] ⑤ If the disk key can be found in the leaf node, return the corresponding value, which is the query result. If it cannot be found, return the result of not finding the key.

[0072] ⑥ When a storage node first starts up, the read cache is empty. Therefore, it is necessary to find the corresponding root node using the root node disk address in the root object (at this time, the read cache is empty in step ③), and then execute steps ④-⑤ to obtain the corresponding value after traversal. Specifically, when the system starts up, the root object stores the disk address of the root node (there is no cache address yet). Therefore, the first query needs to read the root node data from the disk based on the disk address. After reading, the root node data is stored in the cache, and the address in the root object is updated and added to the cache address of the root node.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0074] Embodiments of the present invention also provide a metadata management device, which may include a first determining unit for determining the logical block address of metadata in response to an append write request for metadata; a second determining unit for determining the tree index structure corresponding to the metadata based on the value of the tree identifier bit in the logical block address of the metadata; a write controlling unit for writing the metadata into the corresponding tree index structure; a third determining unit for determining the target tree index structure based on the target logical block address of the target metadata in response to a query request for the target metadata; and a query unit for traversing from the root node of the target tree index structure to query the target metadata.

[0075] In this embodiment of the invention, the tree identifier can be located in the high-order region of the logical block address, so that when the logical block address is incremented, the tree identifier is changed when the carry-over from the previous low-order bit to the tree identifier.

[0076] In this embodiment of the invention, the logical block address may further include a page identifier bit and an intra-page offset bit, wherein the intra-page offset bit, the page identifier bit, and the tree identifier bit are arranged sequentially from the least significant bit to the most significant bit of the logical block address.

[0077] In this embodiment of the invention, the logical block address may further include a high-order offset bit, and the page offset bit, page identifier bit, tree identifier bit and high-order offset bit are arranged sequentially from the low-order bit to the high-order bit of the logical block address.

[0078] The metadata management device provided in this embodiment of the invention may further include: a cache control unit, used to store queried metadata into a cache; and an eviction management unit, used to manage the eviction of metadata stored in the cache according to the hierarchical priority of the tree index structure, wherein eviction of root node data of the tree index structure is prohibited, leaf node data of the tree index structure is prioritized for eviction, and intermediate node data of the tree index structure is then evicted.

[0079] In this embodiment of the invention, the eviction management unit manages the eviction of metadata stored in the cache according to the hierarchical priority of the tree index structure, which may include: eviction processing of metadata of the same priority according to the order of their most recent access time.

[0080] In this embodiment of the invention, the eviction management unit manages the eviction of metadata stored in the cache according to the hierarchical priority of the tree index structure, which may include: eviction processing of metadata of the same priority according to the access frequency according to the hierarchical priority of the tree index structure.

[0081] In this embodiment of the invention, data from intermediate nodes in different layers can have the same priority.

[0082] In this embodiment of the invention, the higher the level of the intermediate node data, the lower the elimination priority.

[0083] The metadata management device provided in this embodiment of the invention may further include: a root node address storage unit, used to persistently store the address of the root node data of the tree index structure; the address of the root node data includes at least the disk address of the root node data.

[0084] In this embodiment of the invention, the cache control unit can also be used to store the root node data in the cache after the first query of the root node data after the system starts up, and update the cache address of the root node data to the address of the root node data.

[0085] In this embodiment of the invention, the cache control unit stores the root node data into the cache, which may include storing the root node data in the cache in the form of an array.

[0086] In this embodiment of the invention, the root node address storage unit persistently stores the address of the root node data of the tree index structure, which may include: persistently storing the address of the root node data to at least two storage nodes respectively; when the address of the root node is updated, synchronously updating the data structure of the root node address in at least two storage nodes; when a storage node fails, after the storage node failure is recovered, restoring the address of the root node based on the data in the storage node that has not failed.

[0087] In this embodiment of the invention, the query unit traverses from the root node of the target tree index structure to query target metadata, which may include: determining the target root node address from the root node address structure of the target tree index structure; if the target root node address includes a cache address, then querying the target root node data from the cache; if the target root node address only contains a disk address, then obtaining the target root node data according to the disk address and storing the target root node data in the cache; and traversing downwards along the target tree index structure according to the target root node data to query the target leaf node data, wherein if the intermediate node data or leaf node data of the target tree index structure encountered during traversal is not in the cache, then the corresponding intermediate node data or leaf node data is stored in the cache.

[0088] Use the target leaf node data as the target metadata.

[0089] For a description of the features in the embodiment corresponding to the metadata management device, please refer to the relevant description in the embodiment corresponding to the metadata management method, which will not be repeated here.

[0090] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described metadata management method embodiments.

[0091] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described metadata management method embodiments at runtime.

[0092] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0093] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described metadata management method embodiments.

[0094] Embodiments of the present invention also provide another computer program product, including a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described metadata management method embodiments.

[0095] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0097] The foregoing has provided a detailed description of the metadata management method and electronic device provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A metadata management method, characterized in that, include: In response to an append write request for metadata, determine the logical block address of the metadata; The tree index structure corresponding to the metadata is determined based on the value of the tree identifier bit in the logical block address of the metadata; Write the metadata into the corresponding tree index structure; In response to a query request for target metadata, the target tree index structure is determined based on the target logical block address of the target metadata; Traverse the target tree index structure starting from the root node to query the target metadata.

2. The metadata management method according to claim 1, characterized in that, The tree identifier is located in the high-order region of the logical block address, so that when the logical block address is incremented, the tree identifier is changed when the carry-over occurs from the previous low-order bit to the tree identifier.

3. The metadata management method according to claim 1, characterized in that, The logical block address also includes a page identifier bit and a page offset bit, wherein the page offset bit, the page identifier bit, and the tree identifier bit are arranged sequentially from the least significant bit to the most significant bit of the logical block address.

4. The metadata management method according to claim 3, characterized in that, The logical block address also includes a high-order offset bit, and the page offset bit, the page identifier bit, the tree identifier bit, and the high-order offset bit are arranged sequentially from the low-order bit to the high-order bit of the logical block address.

5. The metadata management method according to claim 1, characterized in that, Also includes: Store the queried metadata in the cache; The metadata stored in the cache is evicted according to the hierarchical priority of the tree index structure. Specifically, the root node data of the tree index structure is prohibited from being evicted, the leaf node data of the tree index structure is evicted first, and the intermediate node data of the tree index structure is evicted second.

6. The metadata management method according to claim 5, characterized in that, The metadata stored in the cache is evicted according to the hierarchical priority of the tree index structure, including: According to the hierarchical priority of the tree index structure, metadata of the same priority is eliminated in the order of the most recent access time.

7. The metadata management method according to claim 5, characterized in that, The metadata stored in the cache is evicted according to the hierarchical priority of the tree index structure, including: Based on the hierarchical priority of the tree index structure, metadata of the same priority is eliminated according to the access frequency.

8. The metadata management method according to any one of claims 5 to 7, characterized in that, The intermediate node data in different layers have the same priority.

9. The metadata management method according to any one of claims 5 to 7, characterized in that, The higher the level of the intermediate node data, the lower the elimination priority.

10. The metadata management method according to claim 1, characterized in that, Also includes: The address of the root node data of the tree index structure is persistently stored; The address of the root node data includes at least the disk address of the root node data.

11. The metadata management method according to claim 10, characterized in that, Also includes: After the system starts up and queries the root node data for the first time, the root node data is stored in the cache, and the cache address of the root node data is updated to the address of the root node data.

12. The metadata management method according to claim 11, characterized in that, Storing the root node data into a cache includes: The root node data is stored in the cache in the form of an array.

13. The metadata management method according to claim 10, characterized in that, Persistently storing the address of the root node data of the tree index structure includes: The addresses of the root node data are persistently stored in at least two storage nodes respectively; When the address of the root node is updated, the data structure of the address of the root node in at least two storage nodes is updated synchronously. When the storage node fails, after the storage node recovers, the address of the root node is recovered based on the data in the storage node that did not fail.

14. The metadata management method according to claim 1, characterized in that, Traverse the target tree index structure starting from the root node to query the target metadata, including: The target root node address is determined from the root node address structure of the target tree index structure; If the target root node address includes a cache address, then the target root node data is retrieved from the cache. If the target root node address only contains a disk address, then the target root node data is obtained based on the disk address and stored in the cache; Based on the target root node data, the target tree index structure is traversed downwards to obtain the target leaf node data. If the intermediate node data or leaf node data of the target tree index structure is not in the cache, the corresponding intermediate node data or leaf node data is stored in the cache. The target leaf node data is used as the target metadata.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the steps of the metadata management method as described in any one of claims 1 to 14 when executing the computer program.

Images