Metadata management method and electronic device

Abstract

The application relates to the computer technical field, in particular to a metadata management method and an electronic device, the method comprises the following steps: dividing to-be-managed metadata into multiple metadata sets; generating an access association mapping table of a previous period according to the access association degrees of key-value pairs in at least part of the metadata sets in the previous period, and generating an access association mapping table of a current period according to the access association mapping table of the previous period; in response to a received metadata operation instruction, determining the association degrees between the key-value pairs in at least part of the metadata sets according to the access association mapping table of the current period, and performing a splitting operation and / or a merging operation on the key-value pairs in at least part of the metadata sets. Therefore, the problem that related key-value pairs are scattered, multiple disk I / Os are caused and access efficiency is reduced due to a fixed splitting and merging strategy in the prior art is solved, the number of disk I / Os required for a single metadata operation is effectively reduced, and the access efficiency of a storage system is significantly improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a metadata management method and electronic device. Background Technology

[0002] As data storage systems continue to expand, storage capacities have reached hundreds of terabytes (TB) or even petabytes (PB), resulting in massive amounts of data stored within these systems. To improve storage efficiency and data security, modern storage systems have widely adopted advanced features such as deduplication, compression, snapshots, backups, and active-active architectures. While these features enhance system performance, they also dramatically increase the amount of metadata required to manage stored data. Therefore, how to efficiently manage and access metadata has become a key issue affecting the overall data access speed of the storage system.

[0003] In current storage systems, B+ trees or their variants are commonly used to organize and manage metadata. However, traditional B+ trees typically employ a uniform distribution strategy when splitting and merging nodes. This means that when a node is fully loaded, it is divided into two nodes, and when the number of key-value pairs within a node falls below half, it attempts to merge with other nodes. This results in multiple related key-value pairs that need to be queried or modified in a single business I / O (Input / Output) operation being distributed across different nodes, requiring multiple node accesses and reducing the efficiency of metadata operations. This issue urgently needs to be addressed. Summary of the Invention

[0004] This application provides a metadata management method and electronic device to solve the problems of related technologies where fixed splitting and merging strategies lead to the dispersion of relevant key pairs, triggering multiple disk I / O operations and reducing access efficiency. It effectively reduces the number of disk I / O operations required for a single metadata operation and significantly improves the access efficiency of the storage system.

[0005] To achieve the above objectives, a first aspect of this application proposes a metadata management method, comprising the following steps:

[0006] Obtain the metadata to be managed, and divide the metadata to be managed into multiple metadata datasets according to a preset partitioning strategy;

[0007] Obtain the access correlation of key-value pairs in at least a portion of the metadata set in the previous period, generate an access correlation mapping table for the previous period based on the access correlation of key-value pairs in the at least a portion of the metadata set in the previous period, and generate an access correlation mapping table for the current period based on the access correlation mapping table for the previous period.

[0008] In response to a received metadata operation instruction, the association degree between key-value pairs in the at least partial metadata set is determined based on the access association mapping table of the current period, and a splitting operation and / or merging operation is performed on the key-value pairs in the at least partial metadata set based on the association degree between the key-value pairs in the at least partial metadata set.

[0009] According to the metadata management method proposed in this application, the metadata to be managed is divided into multiple metadata datasets; an access association mapping table for the previous period is generated based on the access association degree of key-value pairs in at least a portion of the metadata datasets in the previous period; an access association mapping table for the current period is generated based on the access association mapping table for the previous period; in response to a received metadata operation instruction, the association degree between key-value pairs in at least a portion of the metadata datasets is determined based on the access association mapping table for the current period, and splitting and / or merging operations are performed on the key-value pairs in at least a portion of the metadata datasets. This solves the problems of related technologies where fixed splitting and merging strategies lead to the dispersion of relevant key-value pairs, triggering multiple disk I / O operations and reducing access efficiency, effectively reducing the number of disk I / O operations required for a single metadata operation and significantly improving the access efficiency of the storage system.

[0010] To achieve the above objectives, a second aspect of this application provides a metadata management apparatus, comprising:

[0011] The partitioning module is used to acquire the metadata to be managed and partition the metadata to be managed into multiple metadata datasets according to a preset partitioning strategy;

[0012] The generation module is used to obtain the access correlation degree of key-value pairs in at least a portion of the metadata set in the previous period, generate an access correlation mapping table for the previous period based on the access correlation degree of key-value pairs in the at least a portion of the metadata set in the previous period, and generate an access correlation mapping table for the current period based on the access correlation mapping table for the previous period.

[0013] The management module is configured to, in response to a received metadata operation instruction, determine the degree of association between key-value pairs in the at least partial metadata set based on the access association mapping table of the current period, and perform splitting and / or merging operations on the key-value pairs in the at least partial metadata set based on the degree of association between the key-value pairs in the at least partial metadata set.

[0014] According to the metadata management apparatus proposed in this application, the metadata to be managed is divided into multiple metadata datasets; an access association mapping table for the previous period is generated based on the access association degree of key-value pairs in at least a portion of the metadata datasets in the previous period; an access association mapping table for the current period is generated based on the access association mapping table for the previous period; in response to a received metadata operation instruction, the association degree between key-value pairs in at least a portion of the metadata datasets is determined based on the access association mapping table for the current period, and splitting and / or merging operations are performed on the key-value pairs in at least a portion of the metadata datasets. This solves the problems of related technologies where fixed splitting and merging strategies lead to the dispersion of relevant key-value pairs, triggering multiple disk I / O operations and reducing access efficiency, effectively reducing the number of disk I / O operations required for a single metadata operation and significantly improving the access efficiency of the storage system.

[0015] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the metadata management method as described in the above embodiments.

[0016] To achieve the above objectives, a fourth aspect of this application provides a non-volatile computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the metadata management method as described in the above embodiments.

[0017] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the metadata management method as described in the above embodiments.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a typical B+ tree in related technologies;

[0021] Figure 2 This is a schematic diagram of a variant of the B+ tree in related technologies;

[0022] Figure 3This is a schematic diagram of a node splitting strategy in related technologies;

[0023] Figure 4 This is a flowchart of a metadata management method provided according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a node splitting strategy according to an embodiment of this application;

[0025] Figure 6 This is a block diagram of a metadata management device provided according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application.

[0027] Reference numerals: 10-Metadata management device, 100-Division module, 200-Generation module, 300-Management module; 701-Memory, 702-Processor, 703-Communication interface. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0029] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0030] As those skilled in the art will understand, in data storage systems, due to the large amount of metadata, the commonly used organization and management method is B+ trees and their variants. All nodes of B+ trees and their variants are stored on hard disks and are constantly updated.

[0031] in, Figure 1This is a typical B+ tree, consisting of intermediate nodes and leaf nodes. Intermediate nodes contain multiple key-value pairs (KV) and pointers P to their child nodes. Each key-value pair consists of a key K and a corresponding value V, used for indexing and navigation. The keys in the intermediate nodes are arranged sequentially and connected to the corresponding leaf nodes via pointers. Leaf nodes also contain key-value pairs, and all leaf nodes are linked sequentially via pointers P to form an ordered linked list, facilitating range queries. In the leaf nodes, each key corresponds to a value; the actual data storage is typically located in the leaf nodes. Figure 1 The example shows that keys K10, K20, K30, etc., point to different leaf nodes, and the leaf nodes store consecutive key-value pairs, such as K10-V10, K11-V11, etc., which demonstrates the characteristics of B+ trees in data organization and efficient search.

[0032] Figure 2 As a variant of the B+ tree, it is a variant in which there are no pointer associations between leaf nodes. That is, all key-value pairs are stored in the leaf nodes, and there are no unidirectional or mutual pointers between leaf nodes. This can reduce the number of nodes updated each time, so that the update of one node will not lead to the update of more nodes at the same level, thus avoiding the generation of larger disk read and write I / O.

[0033] In a storage system, metadata includes LP (Logic block address - Physical block address, the mapping relationship between logical block address and physical block address) and PL (Physical block address - Logic block address, the mapping relationship between physical block address and logical block address), which are the key-value pairs in the leaf nodes of B+ trees and their variants. Different types of metadata are stored in different B+ trees, and only one type of key-value metadata is stored in the same B+ tree.

[0034] In the B+ tree algorithm, the size of nodes may differ across levels, but the size of nodes within the same level is usually the same. Common splitting and merging strategies within nodes at each level are as follows: when a node is full of key-value pairs (KV), split the node in two, distributing the KV equally between the two nodes; when the number of KV pairs in a node is less than half, attempt to merge with a sibling node, or migrate some KV pairs from a sibling node to this node. However, such splitting and merging strategies are detrimental to actual business logic. They split the KV pairs that a single I / O operation requires querying / modifying across two nodes, resulting in two separate read / modification operations, reducing the efficiency of KV querying / modification and ultimately impacting business efficiency.

[0035] For example, such as Figure 3The diagram illustrates the splitting process of a leaf node in a B+ tree when a new key-value pair is inserted. The initial state is a single leaf node 0 containing multiple key-value pairs, such as 1-V1, 3-V3, 4-V4, 5-V5, 8-V8, 10-V10, 11-V11, 12-V12, and 13-V13, arranged in the order of the keys. When a new key-value pair (6, v6) is inserted into this node, a split operation is triggered because the node is full. After the split, the original leaf node 0 is divided into two new nodes: leaf node 1 and leaf node 2. Leaf node 1 retains key-value pairs 1-V1, 3-V3, 4-V4, 5-V5, 6-V6, 8-V8, 10-V10, and 11-V11, while leaf node 2 includes key-value pairs such as 12-V12 and 13-V13. That is, the above splitting process causes the originally consecutively stored 10~13 to be split into two nodes, and a sequential access requires two I / O operations.

[0036] Based on the technical problems existing in the aforementioned related technologies, this application proposes a metadata management optimization method to improve the efficiency of metadata access. First, the metadata is divided into multiple smaller metadata datasets based on the key, and different metadata datasets are stored separately in different B+ trees. Second, for each metadata dataset, the patterns of keys involved in metadata access (including the number of consecutive keys, etc.) are statistically analyzed to obtain the guidance template for KV operations in each metadata set and the method for calculating the correlation between different KV values. Third, when updating the B+ tree, the KV operation guidance template is used to perform operations to ensure that KV values ​​with high correlation are on the same node as much as possible. Finally, a background task can be created to perform background inspection and scanning of the B+ tree, and the B+ tree can be adjusted according to the KV operation guidance template to ensure that KV values ​​with high correlation are on the same node as much as possible.

[0037] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Specifically, Figure 4 This is a flowchart of a metadata management method according to an embodiment of this application.

[0039] like Figure 4 As shown, this metadata management method includes the following steps:

[0040] In step S401, the metadata to be managed is obtained and divided into multiple metadata datasets according to a preset partitioning strategy.

[0041] Furthermore, in some embodiments, when the metadata to be managed is a mapping relationship from logical block address to physical block address, the metadata to be managed is divided into multiple metadata datasets according to a preset partitioning strategy, including: partitioning the metadata to be managed based on the numerical range of logical block addresses according to the preset partitioning strategy to obtain multiple metadata datasets; or, after performing a modulo operation on the logical block addresses according to the preset partitioning strategy, grouping according to a preset hash grouping rule to obtain multiple metadata datasets.

[0042] Specifically, in this application embodiment, the metadata is pre-divided into multiple smaller metadata datasets based on the key. For example, it can be divided based on a numerical range, with the metadata segmented into multiple parts according to the numerical range of its key value (e.g., a logical block address). Assuming the metadata to be managed covers all logical block addresses from 1 to 1000, it can be divided into two metadata datasets: the first containing addresses from 1 to 500, and the second containing addresses from 501 to 1000. Alternatively, it can be divided based on hash grouping rules, i.e., using a hash function to operate on the key value (e.g., the logical block address), and determining which metadata dataset the key value belongs to based on the resulting hash value. For example, a modulo operation can be performed on the logical block address, and then the result can be assigned to different metadata datasets.

[0043] For example, for LP metadata, based on the volume size, LPs with LBAs within every 10TB size range are divided into a metadata dataset, or the LBAs are modulo 1GB, and the result is calculated as 100MB * n + k, where k < 100MB, thus dividing them into 10 metadata datasets.

[0044] It should be noted that the volume space is managed in granular form according to the GRAIN size (usually 4KB). Each GRAIN only requires one LP metadata, so the number of LPs in the range of 0~10TB is 2.5G.

[0045] Therefore, by adopting a partitioning strategy based on LBA numerical range or modulo hash, metadata can be reasonably distributed into multiple metadata datasets. This not only reduces the size of a single metadata dataset and alleviates the management overhead of B+ trees, but also provides a structural foundation for subsequent optimization operations such as intelligent splitting, merging, and key-value aggregation based on access relevance, thereby improving metadata access efficiency and overall system performance.

[0046] In step S402, the access correlation degree of key-value pairs in at least a portion of the metadata set in the previous period is obtained, and an access correlation mapping table for the previous period is generated based on the access correlation degree of key-value pairs in at least a portion of the metadata set in the previous period. Based on the access correlation mapping table for the previous period, an access correlation mapping table for the current period is generated.

[0047] Furthermore, in some embodiments, obtaining the access correlation of key-value pairs in at least a portion of the metadata set in the previous period includes: obtaining access requests to at least a portion of the metadata set and the range of key-value pairs corresponding to at least a portion of the access requests in the previous period; generating access logs of key-value pairs in at least a portion of the metadata set based on the access requests to at least a portion of the metadata set and the range of key-value pairs corresponding to at least a portion of the access requests; identifying key-value sequences that are continuously accessed within a preset time window based on the access logs, and obtaining the access correlation of key-value pairs in at least a portion of the metadata set in the previous period based on the frequency of occurrence of the key-value sequences and the length of continuous access.

[0048] Specifically, at least a portion of the metadata access requests originate from the I / O scheduling module of the upper-layer application or storage system, used to read or write data within a specific logical address range. Each access request corresponds to a key-value pair range, i.e., a set of consecutive or non-consecutive but logically related keys (such as logical block addresses LBA) and their mapped values ​​(such as physical block addresses PBA). For example, a read operation on a segment of a file triggers a query for all key-value pairs within the LBA range of 1000–1020.

[0049] Based on the above access requests and their corresponding key-value pair ranges, a structured access log is constructed. This log records, in chronological order, the specific keys involved in each access, the access type (read / write), the timestamp, and the metadata identifier to which it belongs. If multiple keys are accessed sequentially within this time window, it is considered to have potential access locality.

[0050] Furthermore, access logs are scanned using sliding window or streaming algorithms to identify frequently occurring key-value sequences within the given time window. For example, if the access pattern [LBA100, LBA101, LBA102] appears repeatedly in the logs, these keys constitute candidate association sequences. Based on the frequency of this sequence and the length of consecutive accesses throughout the entire period, the access association degree of key-value pairs in at least a portion of the metadata set in the previous period is obtained. For example, a sequence containing 8 consecutive keys and accessed thousands of times per day has a much higher association degree than a sequence containing only 2 keys and accessed only occasionally. Finally, the quantified results are organized into structured association degree data to generate an access association mapping table for the previous period, providing a decision-making basis for optimizing the metadata layout in the current period.

[0051] Therefore, by using the above technical means, the access patterns in real business loads can be effectively captured, enabling the B+ tree structure adjustment to fit the actual I / O behavior, thereby improving the efficiency of metadata access.

[0052] Further, in some embodiments, generating an access association mapping table for the current period based on the access association mapping table of the previous period includes: obtaining the access association degree of key-value pairs in the current period for at least a portion of the metadata set; obtaining the access association degree of key-value pairs in the previous period for at least a portion of the metadata set based on the access association mapping table of the previous period; if the difference between the access association degree of key-value pairs in the current period and the access association degree in the previous period is greater than a preset update degree threshold, then updating the access association mapping table of the previous period based on the access association degree of key-value pairs in the current period for at least a portion of the metadata set to generate the access association mapping table for the current period; otherwise, using the access association mapping table of the previous period as the access association mapping table for the current period.

[0053] Specifically, the access association mapping table in this application embodiment is used to record the access association degree between different key-value pairs in the metadata set. Access association degree refers to the probability or intensity of two or more key-value pairs being accessed consecutively or frequently by the same access request within a specific time window, and is quantified by statistically analyzing indicators such as co-occurrence frequency, access interval, and request context. For example, in a storage system, if an I / O request simultaneously reads the metadata corresponding to logical block addresses LBA 100 to LBA 105, then the key-value pairs corresponding to these LBAs have a high access association degree.

[0054] When generating the access association mapping table for the current period, access logs from at least a portion of the metadata dataset within the current period are first collected, and the access association degree of each key-value pair in the current period is calculated accordingly. Simultaneously, historical association degrees for the same key-value pairs are extracted from the access association mapping table retained from the previous period. Subsequently, the difference value is calculated for each pair of comparable association degree values. This difference value can be expressed as an absolute difference, relative rate of change, or other normalized distance metrics. If the difference value exceeds a preset update threshold (this threshold is used to control the sensitivity of the mapping table updates and avoid frequent reconstructions caused by minor fluctuations), it is determined that the current access pattern has changed significantly. In this case, the corresponding items in the original mapping table are overwritten or weighted and merged based on the association degree data of the current period to generate a new access association mapping table for the current period. Conversely, if the difference does not reach the threshold, the access pattern is considered stable, and the mapping table from the previous period is directly used as the mapping table for the current period.

[0055] In other words, this embodiment of the application statistically analyzes the specific patterns of key-value (KV) accesses within each metadata dataset according to a certain period (e.g., the I / O size of each access). After each period, the results are used as a "guidance template" to calculate the correlation between different KV values, guiding subsequent KV operations. Furthermore, since each period yields new statistical results, if the difference between the latest period's statistical results and the "guidance template" results from previous periods is less than a preset threshold (e.g., 20%), the "guidance template" does not need to be refreshed; otherwise, the template content is updated and used as a new "guidance template" to guide subsequent metadata KV operations.

[0056] For example, suppose that in the previous period, the access correlation of key-value pair K100 and K101 was 0.85, and in the current period, their correlation is found to have increased to 0.96. If the preset update threshold is 0.1, then the difference value of 0.11 is greater than the threshold, and the item in the mapping table will be updated to 0.96. However, if the correlation of another pair K200 and K201 changes from 0.70 to 0.73 (difference 0.03 < 0.1), then the original value of 0.70 will be retained.

[0057] Thus, the above mechanism effectively balances the timeliness of the mapping table with system overhead, ensuring that the metadata layout optimization always closely reflects the characteristics of real business load.

[0058] In step S403, in response to the received metadata operation instruction, the association degree between key-value pairs in at least a portion of the metadata set is determined according to the access association mapping table of the current period, and a split operation and / or merge operation is performed on the key-value pairs in at least a portion of the metadata set according to the association degree between the key-value pairs in at least a portion of the metadata set.

[0059] Furthermore, in some embodiments, in response to a received metadata operation instruction, determining the correlation between key-value pairs in at least a portion of the metadata set based on the access association mapping table of the current period includes: determining the operation target key-value pair involved in the metadata operation instruction; and querying the access association mapping table of the current period based on the operation target key-value pair to obtain the correlation between key-value pairs in at least a portion of the metadata set.

[0060] Specifically, the process first identifies the target key-value pair involved in the metadata operation instruction, i.e., the key and its corresponding value directly affected by the instruction, such as the logical block address (LBA) and its mapped physical block address (PBA) specified in the write request. Further, using this target key-value pair as an index, a query is performed in the access association mapping table for the current period. This mapping table is a pre-built data structure that records the association strength between different key-value pairs due to frequent shared access within the current statistical period, expressed as a correlation degree value (e.g., a floating-point number between 0 and 1). For example, if a snapshot operation consistently accesses multiple addresses within the LBA range of 100–110 simultaneously, the key-value pairs corresponding to these addresses will have a high correlation degree in the mapping table.

[0061] Therefore, by querying the table, we can learn the access characteristics of the target key-value pair and identify other key-value pairs that are highly related to it. Then, we can determine whether these related key-value pairs should be kept in the same physical storage unit (such as the same leaf node of a B+ tree) when splitting or merging nodes. This is to avoid subsequent I / O operations having to be read across multiple nodes due to structural splitting, thereby improving metadata access efficiency and overall system performance.

[0062] It should be noted that, in the business operation process of this application embodiment, for the metadata KV of the operation, the KV is initially inserted / deleted on the B+ tree according to the default algorithm (that is, the access association mapping table for the current period is not generated at this time). The main operations include: when KV is inserted into a node until the node is full, the KV on the node is evenly distributed to the two nodes when the node splits; when KV is deleted and the number of KV in the node is less than half of the node's capacity, it is merged with the sibling node or some KV is moved from the sibling node to the node.

[0063] Furthermore, during business operations, if a valid guidance template for key-value (KV) operations has been generated (i.e., the access association mapping table for the current period), meaning there is a method for calculating the association degree between different KVs, then when inserting / deleting KVs on the B+ tree, the "guidance template" should be referenced. Specifically, this includes: when inserting KVs until a node is full, if splitting the node would split high-association KVs onto two nodes, the operation needs to be changed to keep these high-association KVs on one node, or keep them all on the node before the split, or move them all to the newly split node; when deleting a KV, if the number of KVs on a node is less than half of the node's capacity, it needs to be merged with sibling nodes or some KVs need to be moved from sibling nodes. In this case, the sibling node will prioritize sibling nodes with high association degree with the KVs on its own node.

[0064] Further, in some embodiments, performing splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set includes: determining a first target node in the at least a portion of the metadata set whose node capacity is greater than a first preset capacity; identifying a first target key-value pair combination in the first target node whose correlation is greater than a first preset correlation threshold based on the correlation between key-value pairs in the at least a portion of the metadata set; determining whether splitting the first target node based on a preset default splitting point splits any first target key-value pair combination into different child nodes; and, if splitting the first target node based on the preset default splitting point splits any first target key-value pair combination into different child nodes, moving the preset default splitting point along the key-value order based on a preset node splitting strategy so that the first target key-value pair combination is in the same node.

[0065] Specifically, when the leaf nodes of a B+ tree corresponding to a certain metadata set reach or exceed the preset capacity limit (i.e., the first preset capacity) due to continuous insertion, the node is identified as the first target node and a split operation needs to be performed to maintain the balance and performance of the tree.

[0066] However, unlike traditional B+ trees that simply split key-value pairs into two parts at the middle position when a node is full, this embodiment first analyzes the access correlation between key-value pairs within the first target node based on the access correlation mapping table for the current period. This involves analyzing the frequency and density of accesses to the same I / O request or consecutive accesses within similar time windows in the historical access log. If the correlation of a combination of first target key-value pairs exceeds a first preset correlation threshold, it indicates that these key-value pairs are highly related in business logic and should be kept in the same physical node as much as possible to reduce cross-node I / O overhead during future accesses.

[0067] Based on this, a default split point is used for splitting, and it is determined whether this operation will split any highly correlated first target key-value pair combination into two different child nodes. If the determination result is yes, that is, the default split will destroy the high correlation, then the optimized splitting process is triggered: according to the preset node splitting strategy, the position of the split point is dynamically adjusted along the ordered sequence of key values. For example, if the default split point is located at key K50, but K48–K52 constitute a highly correlated combination, then the split point is moved left to after K47 or right to before K53, so that the entire K48–K52 combination is completely preserved in the left or right child nodes.

[0068] Therefore, through the above technical means, the structural evolution of the B+ tree is no longer driven solely by static capacity, but incorporates dynamic business access characteristics. This effectively aggregates related data, reduces the number of disk I / O operations for metadata, and improves the overall response efficiency of the storage system while ensuring the basic balance of the tree structure.

[0069] To facilitate a clearer and more intuitive understanding by those skilled in the art of the splitting process of leaf nodes in the B+ tree of this application's embodiments, the following is combined with... Figure 5 Please provide a detailed explanation.

[0070] like Figure 5 As shown, the initial state is a single leaf node 0 containing multiple key-value pairs arranged in the order of keys, such as 1-V1, 3-V3, 4-V4, 5-V5, 8-V8, 10-V10, 11-V11, 12-V12, and 13-V13. When this node reaches its capacity limit due to the insertion of new data 6-V6, a split operation is triggered. After the split, the original node is divided into two new leaf nodes: leaf node 1 and leaf node 2. Leaf node 1 contains key-value pairs 1-V1, 3-V3, 4-V4, 5-V5, 6-V6, and 8-V8; leaf node 2 contains key-value pairs 10-V10, 11-V11, 12-V12, and 13-V13. After the split, the previously consecutively stored 10~13 remain on the same node. A sequential access requires only one I / O operation. Through this splitting mechanism, the B+ tree maintains the orderliness of the nodes and the balance of the structure, ensuring efficient search and insertion performance.

[0071] Furthermore, in some embodiments, performing splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set further includes: determining a second target node whose node capacity in at least a portion of the metadata set is less than a second preset capacity, and determining the adjacent sibling nodes of the second target node; obtaining the access correlation between key-value pairs in the second target node and key-value pairs in adjacent sibling nodes based on the correlation between key-value pairs in at least a portion of the metadata set; determining a target sibling node whose access correlation with key-value pairs in the second target node is greater than a second preset correlation threshold; and performing a merging operation on the target sibling node and the second target node based on a preset node merging strategy, or migrating the associated key-value pairs in the target sibling node to the second target node.

[0072] Specifically, the first step is to identify a second target node whose node capacity is less than the second preset capacity. The node capacity refers to the proportion of the number of key-value pairs currently stored in a B+ leaf node to its maximum capacity. The second preset capacity can be set as a lower limit threshold of the node's maximum capacity (e.g., 50%) to determine whether the node is in a sparse state and requires space optimization.

[0073] Furthermore, the adjacent sibling nodes of the second target node in the B+ tree are determined, that is, the predecessor or successor leaf nodes that are logically adjacent to the target node. These two nodes are connected by horizontal pointers between the leaf nodes of the B+ tree, forming an ordered linked list structure. Based on the access association mapping table of the current period, the access association degree between each key-value pair in the second target node and the key-value pairs in its adjacent sibling nodes is calculated. Based on this, it is further determined whether there exists a target sibling node with which there is a key-value pair combination whose access association degree with the second target node is greater than a second preset association threshold.

[0074] If the conditions are met, one of two optimization strategies will be triggered: First, a node merging operation will be performed, which will merge all key-value pairs of the second target node and the target sibling node into a new node (but it must be ensured that the merged node does not exceed the maximum capacity of the node), thereby reducing the number of nodes and improving space utilization; Second, an associated key-value pair migration operation will be performed, which will only migrate the key-value pairs of the target sibling node that are highly associated with the second target node to the second target node, rather than merging the entire node, in order to avoid the node overflowing again or destroying other associated localities due to merging.

[0075] Therefore, this application embodiment, by referring to the statistical patterns of metadata access when updating the B+ tree, aims to ensure that the key-value pairs required for each metadata query / modification can be stored on a single node. This ensures that metadata queries / modifications can most likely access only one node, reducing the number and frequency of reading / modifying nodes.

[0076] Furthermore, in some embodiments, the metadata management method further includes: when a background optimization task is initiated, scanning the target B+ tree corresponding to the target metadata dataset according to the access association mapping table of the current period; identifying second target key-value pair combinations in the target B+ tree whose association degree is greater than a third preset association threshold and which are stored in different nodes, according to the access association mapping table of the current period; performing a cross-node key-value pair migration operation on the second target key-value pair combinations whose association degree is greater than the third preset association threshold and which are stored in different nodes, and re-aggregating the second target key-value pair combinations into the same node.

[0077] Understandably, once a valid template for key-value (KV) operations is obtained (i.e., the access association mapping table for the current period), and a method for calculating the correlation between different KV values ​​is available, background tasks can be created and selectively launched to traverse and scan the B+ tree corresponding to a portion of the metadata dataset. This checks whether the KV values ​​in the B+ tree match the patterns of metadata access. If they do not match, updates are performed, i.e., checking for KV values ​​that deviate significantly from the access association mapping table for the current period. If so, they are migrated between adjacent nodes to make them conform to the access association mapping table for the current period.

[0078] Specifically, to continuously optimize metadata layout to adapt to dynamically changing access patterns, background optimization tasks can be initiated during off-peak business periods or when resources permit. Based on the access association mapping table built in the current cycle, a full or incremental scan is performed on the B+ tree structure corresponding to the target metadata dataset. Second target key-value pair combinations with high access relevance but scattered across different leaf nodes due to historical splitting operations are identified. Then, cross-node key-value pair migration operations are performed to re-aggregate them into the same node, thereby reducing the number of disk I / O operations required for future access to these related data.

[0079] Furthermore, during the scanning process, the leaf nodes of the B+ tree are traversed, and the association mapping table is accessed to determine whether there are key-value pairs with high correlation that are physically separated in different nodes. Once such a situation is found, a cross-node key-value pair migration operation will be triggered: that is, some key-value pairs are removed from the source node and inserted into the target node, while ensuring that the target node does not exceed the maximum capacity limit; if the target node is full, a local rebalancing or temporary expansion mechanism can be triggered. For example, suppose that keys K10 and K12 were originally located in leaf nodes A and B respectively, but their correlation is 0.9 (higher than the threshold of 0.8), the background optimization task will migrate K12 from node B to node A (or vice versa), so that the two can coexist in the same node.

[0080] Therefore, by periodically sensing access patterns, identifying suboptimal layouts, and proactively reconstructing the physical distribution of metadata, an effective supplement to the traditional static splitting strategy of B+ trees is achieved. After optimization, the probability of highly related key-value pairs being clustered and stored in the same node is significantly increased. Therefore, when performing data read or overwrite operations, the probability of finding this metadata in the same node during the query or modification process is higher than before, resulting in faster lookup (or modification) speeds and higher efficiency, thereby reducing the number of cross-node lookups.

[0081] Furthermore, in some embodiments, when performing splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set, the method further includes: monitoring the system service operation status; and when the system service operation status is detected to be stopped, suspending the splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set.

[0082] The system's operational status refers to whether the storage system is currently processing user I / O requests, creating snapshots, deduplicating data, compressing, or engaging in other core business activities. This status can be determined through signals such as running flags, load metrics, task queue depth, or service process activity provided by the system kernel or management layer. The stopped status refers to situations where the system has entered maintenance mode, is undergoing shutdown procedures, is in fault recovery phases, or where administrators have manually suspended business services.

[0083] When the monitoring module detects that the system's business operation status is stopped, it will stop periodic statistics and updates to the access association mapping table, and pause the B+ tree adjustment (split or merge) operation triggered by the correlation between key-value pairs in the access association mapping table. When the business restarts, the periodic statistics, updates to the access association mapping table, and the B+ tree adjustment (split or merge) operation triggered by the correlation between key-value pairs in the access association mapping table will resume. This avoids resource contention (such as CPU, memory, or disk bandwidth) caused by background structure adjustments during critical system maintenance or shutdown, which could interfere with the shutdown process or prolong fault recovery time, thus balancing system reliability, maintainability, and operational security.

[0084] According to the metadata management optimization method proposed in this application, which improves the efficiency of metadata access, the method first obtains the relationship between key-value pairs and uses it to guide metadata operations, thereby increasing the likelihood that key-value pairs with high correlation are on the same node. As a result, when querying / modifying metadata, it is highly likely that only one node needs to be accessed, thus improving the efficiency of metadata key-value querying / modification and improving the overall IO performance of the system.

[0085] According to the metadata management method proposed in this application, the metadata to be managed is divided into multiple metadata datasets; an access association mapping table for the previous period is generated based on the access association degree of key-value pairs in at least a portion of the metadata datasets in the previous period; an access association mapping table for the current period is generated based on the access association mapping table for the previous period; in response to a received metadata operation instruction, the association degree between key-value pairs in at least a portion of the metadata datasets is determined based on the access association mapping table for the current period, and splitting and / or merging operations are performed on the key-value pairs in at least a portion of the metadata datasets. This solves the problems of related technologies where fixed splitting and merging strategies lead to the dispersion of relevant key-value pairs, triggering multiple disk I / O operations and reducing access efficiency, effectively reducing the number of disk I / O operations required for a single metadata operation and significantly improving the access efficiency of the storage system.

[0086] Next, the metadata management device proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0087] Figure 6 This is a block diagram of a metadata management device according to an embodiment of this application.

[0088] like Figure 6 As shown, the metadata management device 10 includes: a partitioning module 100, a generation module 200, and a management module 300.

[0089] The system includes a partitioning module 100, which acquires metadata to be managed and partitions it into multiple metadata datasets according to a preset partitioning strategy; a generation module 200, which acquires the access correlation of key-value pairs in at least a portion of the metadata datasets in the previous period, generates an access correlation mapping table for the previous period based on the access correlation of key-value pairs in the previous period, and generates an access correlation mapping table for the current period based on the access correlation mapping table for the previous period; and a management module 300, which, in response to received metadata operation instructions, determines the correlation between key-value pairs in at least a portion of the metadata datasets based on the access correlation mapping table for the current period, and performs splitting and / or merging operations on the key-value pairs in at least a portion of the metadata datasets based on the correlation between key-value pairs in the at least a portion of the metadata datasets.

[0090] Furthermore, in some embodiments, the management module 300 is configured to: determine a first target node whose node capacity in at least a portion of the metadata set is greater than a first preset capacity; identify a first target key-value pair combination in the first target node whose correlation degree is greater than a first preset correlation threshold based on the correlation degree between key-value pairs in at least a portion of the metadata set; determine whether splitting the first target node based on a preset default split point splits any first target key-value pair combination into different child nodes; and, if splitting the first target node based on the preset default split point splits any first target key-value pair combination into different child nodes, move the preset default split point along the key-value order based on a preset node splitting strategy so that the first target key-value pair combination is in the same node.

[0091] Furthermore, in some embodiments, the management module 300 is also configured to: determine a second target node whose node capacity in at least a portion of the metadata set is less than a second preset capacity, and determine the adjacent sibling nodes of the second target node; obtain the access correlation degree between key-value pairs in the second target node and key-value pairs between adjacent sibling nodes based on the correlation degree between key-value pairs in at least a portion of the metadata set; determine a target sibling node whose access correlation degree with key-value pairs in the second target node is greater than a second preset correlation threshold; and perform a merging operation on the target sibling node and the second target node based on a preset node merging strategy, or migrate the associated key-value pairs in the target sibling node to the second target node.

[0092] Furthermore, in some embodiments, the metadata management device 10 is also configured to: when a background optimization task is initiated, scan the target B+ tree corresponding to the target metadata dataset according to the access association mapping table of the current period; identify second target key-value pair combinations in the target B+ tree whose association degree is greater than a third preset association threshold and which are stored in different nodes according to the access association mapping table of the current period; perform cross-node key-value pair migration operation on the second target key-value pair combinations whose association degree is greater than the third preset association threshold and which are stored in different nodes, and re-aggregate the second target key-value pair combinations into the same node.

[0093] Further, in some embodiments, the generation module 200 is configured to: obtain access requests to at least a portion of the metadata dataset and the range of key-value pairs corresponding to at least a portion of the access requests in the previous period; generate access logs of key-value pairs in at least a portion of the metadata dataset based on the access requests to at least a portion of the metadata dataset and the range of key-value pairs corresponding to at least a portion of the access requests; identify key-value sequences that are continuously accessed within a preset time window based on the access logs; and obtain the access correlation of key-value pairs in at least a portion of the metadata dataset in the previous period based on the frequency of occurrence of the key-value sequences and the length of continuous access.

[0094] Further, in some embodiments, the generation module 200 is configured to: obtain the access relevance of key-value pairs in at least a portion of the metadata set in the current period; obtain the access relevance of key-value pairs in the previous period based on the access relevance mapping table of the previous period; if the difference between the access relevance of key-value pairs in the current period and the access relevance in the previous period is greater than a preset update threshold, then update the access relevance mapping table of the previous period based on the access relevance of key-value pairs in the current period to generate the access relevance mapping table of the current period; otherwise, use the access relevance mapping table of the previous period as the access relevance mapping table of the current period.

[0095] Furthermore, in some embodiments, the management module 300 is configured to: determine the operation target key-value pair involved in the metadata operation instruction; and query the access association mapping table of the current period based on the operation target key-value pair to obtain the association degree between key-value pairs in at least a portion of the metadata set.

[0096] Furthermore, in some embodiments, when the metadata to be managed is a mapping relationship from logical block address to physical block address, the partitioning module 100 is used to: partition the metadata to be managed based on the numerical range of the logical block address according to a preset partitioning strategy to obtain multiple metadata datasets; or, according to a preset partitioning strategy, after performing a modulo operation on the logical block address, group it according to a preset hash grouping rule to obtain multiple metadata datasets.

[0097] Furthermore, in some embodiments, when performing splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set, the management module 300 is also configured to: monitor the system service operation status; and when the system service operation status is detected to be stopped, suspend the splitting and / or merging operations on key-value pairs in at least a portion of the metadata set based on the correlation between key-value pairs in at least a portion of the metadata set.

[0098] It should be noted that the foregoing explanation of the metadata management method embodiment also applies to the metadata management device of this embodiment, and will not be repeated here.

[0099] According to the metadata management apparatus proposed in this application, the metadata to be managed is divided into multiple metadata datasets; an access association mapping table for the previous period is generated based on the access association degree of key-value pairs in at least a portion of the metadata datasets in the previous period; an access association mapping table for the current period is generated based on the access association mapping table for the previous period; in response to a received metadata operation instruction, the association degree between key-value pairs in at least a portion of the metadata datasets is determined based on the access association mapping table for the current period, and splitting and / or merging operations are performed on the key-value pairs in at least a portion of the metadata datasets. This solves the problems of related technologies where fixed splitting and merging strategies lead to the dispersion of relevant key-value pairs, triggering multiple disk I / O operations and reducing access efficiency, effectively reducing the number of disk I / O operations required for a single metadata operation and significantly improving the access efficiency of the storage system.

[0100] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0101] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0102] When the processor 702 executes the program, it implements the metadata management method provided in the above embodiments.

[0103] Furthermore, electronic devices also include:

[0104] Communication interface 703 is used for communication between memory 701 and processor 702.

[0105] The memory 701 is used to store computer programs that can run on the processor 702.

[0106] The memory 701 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0107] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0108] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0109] The processor 702 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0110] This application also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the metadata management method described above.

[0111] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the metadata management method described above.

[0112] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] The metadata management method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A metadata management method, characterized in that, Includes the following steps: Obtain the metadata to be managed, and divide the metadata to be managed into multiple metadata datasets according to a preset partitioning strategy; Obtain the access correlation of key-value pairs in at least a portion of the metadata set in the previous period, generate an access correlation mapping table for the previous period based on the access correlation of key-value pairs in the at least a portion of the metadata set in the previous period, and generate an access correlation mapping table for the current period based on the access correlation mapping table for the previous period. In response to a received metadata operation instruction, the association degree between key-value pairs in the at least partial metadata set is determined based on the access association mapping table of the current period, and a splitting operation and / or merging operation is performed on the key-value pairs in the at least partial metadata set based on the association degree between the key-value pairs in the at least partial metadata set.

2. The method according to claim 1, characterized in that, The step of performing splitting and / or merging operations on key-value pairs in the at least partial metadata set based on the correlation between key-value pairs in the at least partial metadata set includes: A first target node is identified whose capacity is greater than a first preset capacity in at least a portion of the metadata set nodes. Based on the correlation between key-value pairs in the at least part of the metadata set, identify the first target key-value pair combination in the first target node whose correlation is greater than a first preset correlation threshold; Determine whether splitting the first target node based on a preset default split point splits any first target key-value pair into different child nodes; When the first target node is split based on a preset default split point, and any first target key-value pair combination is split into different child nodes, the preset default split point is moved along the key-value order based on a preset node split strategy so that the first target key-value pair combination is in the same node.

3. The method according to claim 1, characterized in that, The step of performing splitting and / or merging operations on key-value pairs in the at least partial metadata set based on the correlation between key-value pairs in the at least partial metadata set further includes: Identify a second target node whose capacity is less than a second preset capacity in the at least part of the metadata set, and identify the adjacent sibling nodes of the second target node; Based on the correlation between key-value pairs in the at least part of the metadata set, the access correlation between key-value pairs in the second target node and key-value pairs between the adjacent sibling nodes is obtained. Identify a target sibling node whose access correlation with the key-value pair in the second target node is greater than a second preset correlation threshold. Based on a preset node merging strategy, perform a merging operation on the target sibling node and the second target node, or migrate the associated key-value pairs in the target sibling node to the second target node.

4. The method according to claim 1, characterized in that, Also includes: When the background optimization task is started, the target B+ tree corresponding to the target metadata is scanned according to the access association mapping table of the current period. Based on the access association mapping table of the current period, identify the second target key-value pair combination in the target B+ tree whose association degree is greater than the third preset association threshold and which is stored in different nodes; For the second target key-value pair combination whose correlation degree is greater than the third preset correlation threshold and which is stored in different nodes, perform a cross-node key-value pair migration operation to re-aggregate the second target key-value pair combination into the same node.

5. The method according to claim 1, characterized in that, The step of obtaining the access correlation of key-value pairs in at least a portion of the metadata set in the previous period includes: Obtain the access requests to at least a portion of the metadata dataset and the range of key-value pairs corresponding to at least a portion of the access requests within the previous period; The access logs of key-value pairs in the at least partial metadata set are generated based on the access requests of the at least partial metadata set and the key-value pair ranges corresponding to the at least partial access requests. Based on the access log, identify key-value sequences that are accessed consecutively within a preset time window, and obtain the access correlation of key-value pairs in the at least part of the metadata set in the previous period based on the frequency of occurrence of the key-value sequences and the length of consecutive access.

6. The method according to claim 1, characterized in that, The step of generating the access association mapping table for the current period based on the access association mapping table of the previous period includes: Obtain the access relevance of key-value pairs in the at least part of the metadata set in the current period; Based on the access association mapping table of the previous period, the access association degree of the key-value pairs in the at least part of the metadata set in the previous period is obtained; If the difference between the access correlation of key-value pairs in the at least part of the metadata set in the current period and the access correlation in the previous period is greater than a preset update threshold, then the access correlation mapping table of the previous period is updated based on the access correlation of key-value pairs in the at least part of the metadata set in the current period to generate the access correlation mapping table of the current period; otherwise, the access correlation mapping table of the previous period is used as the access correlation mapping table of the current period.

7. The method according to claim 1, characterized in that, The step of determining the association degree between key-value pairs in the at least part of the metadata set in response to the received metadata operation instruction, based on the access association mapping table of the current period, includes: Determine the target key-value pair involved in the metadata operation instruction; Based on the operation target key-value pair, query the access association mapping table for the current period to obtain the association degree between key-value pairs in the at least part of the metadata set.

8. The method according to claim 1, characterized in that, When the metadata to be managed is a mapping relationship from logical block address to physical block address, the metadata to be managed is divided into multiple metadata datasets according to a preset partitioning strategy, including: According to the preset partitioning strategy, the metadata to be managed is partitioned based on the numerical range of the logical block address to obtain the multiple metadata datasets; Alternatively, based on a preset partitioning strategy, after performing a modulo operation on the logical block addresses, the data are grouped according to a preset hash grouping rule to obtain the multiple metadata sets.

9. The method according to claim 1, characterized in that, When performing splitting and / or merging operations on key-value pairs in the at least partial metadata set based on the correlation between key-value pairs in the at least partial metadata set, the method further includes: Monitor the operational status of the system's services; When the system service operation status is detected to be stopped, the splitting and / or merging operations on the key-value pairs in the at least partial metadata set are suspended based on the correlation between the key-value pairs in the at least partial metadata set.

10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, the processor executing the computer program to implement the metadata management method as described in any one of claims 1-9.

Images