Metadata processing method and device, computer equipment and storage medium

By calculating the hash value and preset level of metadata, and directly positioning the target node location, the problem of low access efficiency in traditional tree structures is solved, and the concurrent access efficiency and performance of all-flash systems are improved.

CN118964677APending Publication Date: 2024-11-15INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411022389.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In all-flash data storage systems, the access efficiency of traditional tree structures is low, especially when large-scale concurrent random access of metadata, which leads to increased latency and high CPU consumption, which affects system performance.

Method used

By obtaining the target logical address, calculating its hash value to locate the target tree, and calculating the target node location based on the preset hierarchy and the number of stored key value pairs, efficient metadata processing is achieved, reducing the process of layer-by-layer search and improving access speed.

Benefits of technology

It effectively reduces IO delay, increases concurrent access, reduces CPU consumption, and improves system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964677A_ABST
    Figure CN118964677A_ABST
Patent Text Reader

Abstract

The invention relates to a metadata processing method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a target logic address, and obtaining a target tree where the target logic address is located based on the target logic address; obtaining a target hierarchy of the target tree and the number of target key value pairs stored in the target hierarchy; calculating a target node position of the target logic address in the target tree according to the target hierarchy and the number of the target key-value pairs stored in the target hierarchy; and processing the target metadata corresponding to the target logic address according to the target node position. By adopting the method, the position of the target node can be quickly positioned, and efficient data processing is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a metadata processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In data storage systems, data grains are the basis of data information storage and the smallest unit of data. In recent years, with the development of information technology, massive amounts of data have been generated, but how to effectively manage and process these massive amounts of data has become a prominent issue. For a large amount of stored data, querying and analyzing the data content and meaning can make more effective use of the data. In all-flash data storage, large amounts of highly concurrent data access and query problems are inevitably involved. Effective management of metadata can increase concurrent access and access efficiency. Therefore, an effective management method for multi-concurrent reading and writing of metadata in all-flash is crucial, which can enable large-scale concurrent random access to metadata with higher throughput and lower latency.

[0003] In the prior art, each data volume usually corresponds to a tree structure. Considering the read-write amplification factor, the tree node size should not be too large. When the volume capacity is very large (currently supports a maximum volume capacity of 512T), the corresponding tree layer number will be large, and the average access time will increase exponentially. In addition, the traditional tree access starts from the root node and searches downward step by step. This layer-by-layer search method has a long process and will extend the IO latency. In addition, in the insertion and deletion operations, data insertion, node splitting and data deletion all involve data movement and copying, which will lead to high CPU (Central Processing Unit) consumption and reduced performance. Summary of the invention

[0004] Based on this, it is necessary to provide a metadata processing method, device, computer equipment and storage medium that can quickly locate the target node position and achieve efficient data processing in response to the above technical problems.

[0005] In order to solve the above technical problems, in a first aspect, a metadata processing method is provided, the method comprising:

[0006] Obtain a target logical address, and obtain a target tree where the target logical address is located based on the target logical address;

[0007] Get the target level of the target tree and the number of target key-value pairs stored in the target level;

[0008] Calculate the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level;

[0009] The target metadata corresponding to the target logical address is processed according to the target node position.

[0010] In one embodiment, obtaining a target tree where the target logical address is located based on the target logical address includes:

[0011] In response to receiving a metadata access request carrying a target logical address, obtaining the target logical address carried in the metadata access request;

[0012] A hash value of the target logical address is calculated, a hash value of a target position of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target position of the target logical address.

[0013] In one embodiment, the target level of the target tree is a preset level, and obtaining the target level of the target tree and the number of target key-value pairs stored in the target level includes:

[0014] The preset level of the target tree and the storage space of the target tree are obtained, and the number of key-value pairs that can be stored in the target tree is calculated based on the preset level and the storage space.

[0015] In one embodiment, calculating the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level includes:

[0016] Get the hierarchical address of the target node in the target hierarchy;

[0017] Obtaining an address range of a target key-value pair stored in the target layer based on the number of target key-value pairs stored in the target layer;

[0018] Perform a modulo operation based on the target logical address and the number of target key-value pairs stored in the target level to obtain a calculation result;

[0019] The target node location is determined based on the calculation result and the address range of the target key-value pair stored in the target level.

[0020] In one embodiment, processing the target metadata corresponding to the target logical address according to the target node position includes:

[0021] Determine whether a target key-value pair can be queried from the target node position, where the target key-value pair stores a target logical address and a target physical address with a mapping relationship;

[0022] If the target key-value pair can be found from the target node location, the target physical address in the target key-value pair is read;

[0023] If the target key-value pair cannot be found from the target node position, the pointer information of the upper node of the target node position is obtained layer by layer until the target physical address stored in the parent node is obtained;

[0024] If the target physical address stored in the parent node is an invalid value, a query failure instruction is generated and query failure information is fed back.

[0025] In one of the embodiments, processing the target metadata corresponding to the target logical address according to the target node position further includes:

[0026] Determine whether the target key-value pair is stored in the target node position;

[0027] If the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position;

[0028] If the target key-value pair is stored in the target node position, the target physical address corresponding to the current target logical address is modified.

[0029] In one embodiment, the method further comprises:

[0030] In response to receiving an instruction to write target metadata into a target tree, obtaining a memory occupied space of the target metadata and a memory remaining space of the target tree;

[0031] Determine whether the memory space occupied by the target metadata is greater than the remaining memory space of the target tree;

[0032] If the memory space occupied by the target metadata is larger than the remaining memory space of the target tree, the remaining memory space of all trees in the system is traversed to determine whether there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0033] If there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata, the target metadata is written into the tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0034] If there is no tree whose remaining memory space is larger than the memory space occupied by the target metadata, then the root nodes of all trees in the system are traversed to determine the invalid ratio of the root nodes of all trees;

[0035] If the invalidity ratio of the root nodes of all trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree with invalid root nodes is copied to the new root node.

[0036] In order to solve the above technical problem, in a second aspect, a metadata processing device is provided, the device comprising:

[0037] An acquisition module is used to acquire a target logical address, and based on the target logical address, acquire a target tree where the target logical address is located; acquire a target level of the target tree and the number of target key-value pairs stored in the target level;

[0038] A calculation module, used for calculating the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level;

[0039] The processing module is used to process the target metadata corresponding to the target logical address according to the target node position.

[0040] In order to solve the above technical problem, in the third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: the processor implements the steps of the method described in the first aspect when executing the computer program.

[0041] In order to solve the above technical problem, in a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] Different from the prior art, the present application obtains the target logical address, and based on the target logical address, obtains the target tree where the target logical address is located; obtains the target level of the target tree and the number of target key-value pairs stored in the target level; calculates the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level; processes the target metadata corresponding to the target logical address according to the target node position, so that the target node position can be directly located according to the target level and the number of target key-value pairs stored in the target level, thereby achieving efficient data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the process of accessing data through all-flash storage in the prior art;

[0044] Figure 2 It is a flowchart of a single B+ tree organization method in the prior art;

[0045] Figure 3 It is a flowchart of a B+ tree access method in the prior art;

[0046] Figure 4 A schematic diagram of a metadata processing method in one embodiment;

[0047] Figure 5 A schematic diagram of a process of organizing multiple B+ trees in one embodiment;

[0048] Figure 6 A schematic diagram of a B+ tree structure in another embodiment;

[0049] Figure 7is a structural block diagram of a metadata processing device in one embodiment;

[0050] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] See also Figure 1 In the prior art, when implementing metadata query, a query request carrying a logical address is received, metadata is queried according to the query request, and then the mapping relationship between the logical address and the physical address is found. When querying data, the metadata cache is first accessed. If the metadata cache has corresponding metadata, the metadata is directly checked and then the corresponding metadata is obtained from the metadata cache and returned to the query request. If the metadata cache does not have corresponding metadata, the corresponding metadata is obtained from the SSD disk and then returned to the query request. Finally, the query request accesses the data corresponding to the physical address. It can be seen that the memory resource access efficiency is very important for data reading and writing. The effective management of the resource pool and the rapid application and release of resources are crucial to the metadata access efficiency.

[0053] See also Figure 2 In the traditional solution, each data volume corresponds to a B+ tree structure. Considering the read-write amplification factor, the B+ tree node should not be too large. When the capacity of the data volume is large (currently supports a maximum volume capacity of 512T), the corresponding B+ tree layer number will be large. According to statistics, when the number of B+ tree layers exceeds 4, the average access time increases exponentially.

[0054] See also Figure 3 ,from Figure 3 As can be seen from the figure, the traditional B+ tree access starts from the root node and searches downward step by step. The node is located by binary search. The relationship between the parent and child nodes is usually obtained by flushing the disk address PBA (physical address). Through the PBA (physical address), the hashtable (data table) is queried to obtain the lower-level BtreeNode node; search from the root node step by step until the leaf node. The implementation method of metadata query based on B+ tree can be as follows Figure 3As shown in the figure, the direction indicated by the arrow is the query direction. It can be seen that the fewer the layers, the faster the query speed. Specifically, the metadata of all-flash storage is organized using the B+ tree data structure. When the amount of metadata to be stored is large, the corresponding B+ tree layers will be large. When the number of B+ tree layers exceeds 4, the average access and query time increases exponentially; such layer-by-layer search process is long and prolongs the IO latency. In addition, during the insertion and deletion operations, data insertion, node splitting and data deletion all involve data movement and copying, which results in high CPU consumption and leads to a decrease in system performance.

[0055] In view of the above technical problems, in one embodiment, Figure 4 As shown, a metadata processing method is provided, the method comprising the following steps:

[0056] Step 401, obtain a target logical address, and obtain a target tree where the target logical address is located based on the target logical address.

[0057] The metadata in this application is used to indicate the storage location of data, that is, to indicate the mapping relationship between the logical address and the physical address. In response to receiving a metadata access request carrying a target logical address, the target logical address carried in the metadata access request is obtained; a hash value of the target logical address is calculated, a hash value of a target location of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target location of the target logical address.

[0058] Exemplarily, the hash value of the target logical address can be calculated, and the hash value of the target position can be selected. Assuming that the calculated hash value of the target logical address has n bytes, the hash values ​​of the first few bytes / last few bytes / middle bytes can be obtained as the first hash value, and the tree with the same logical address as the target position and the same value as the first hash value is selected from the tree in the system as the target tree to obtain the target tree address. The target tree in the present application can be a B tree, a B+ tree, etc., and the preferred target tree in the present application is a B+ tree. It can be understood that the target position can be the hash value of bytes of any position and any length, and the present application does not limit this.

[0059] In a feasible implementation, in response to receiving a metadata access request carrying a target logical address, the target logical address carried in the metadata access request is obtained; the byte information of the target logical address is obtained, the hash value of the target logical address is calculated based on the byte information of the target logical address, and the target tree address corresponding to the target logical address is calculated based on the hash value of the target logical address.

[0060] Exemplarily, the target position byte in the target logical address byte can be selected as the target logical address byte to be calculated, the selected target logical address byte to be calculated is hashed to obtain the hash value of the target logical address, any tree in the pre-set trees is selected as the tree to be selected, any logical address to be selected in the tree to be selected is obtained, and the hash value of the target position byte in any logical address to be selected in the tree to be selected is calculated. The tree to be selected whose hash value of the target logical address is the same as the hash value of the target byte in the tree to be selected is used as the target tree. It is understandable that the target position can be a logical address byte at any position, and this application does not limit this. By allocating logical addresses with the same hash value of the target position to the same tree, subsequent search operations are facilitated.

[0061] Step 402, obtaining a target level of the target tree and the number of target key-value pairs stored in the target level.

[0062] After determining the target tree through the above steps, the target level of the target tree and the number of target key-value pairs stored in the target level are obtained. The target level is the lowest level of the target tree, that is, the level of leaf nodes, and the number of key-value pairs stored in the target level is the number of target key-value pairs that the target tree can store.

[0063] The preset level of the target tree and the storage space of the target tree may be obtained, and the number of key-value pairs that can be stored in the target tree may be calculated according to the level of the target tree and the storage space of the target tree.

[0064] Specifically, see Figure 5 , the preset level of the target tree here refers to the number of layers of the target tree. In this application, the number of layers of the tree is set to not exceed 4. Compared with the prior art, in this application, the tree corresponding to the data volume is divided into multiple subtrees (i.e., the target tree in this application), and each subtree carries part of the metadata, so that the number of layers of each tree will not exceed 4; and the space for storing the address of the root node of the tree is only N-1 (assuming N subtrees) 8 bytes more (8 bytes are used to store the PBA address (physical address) of the root node), but its access time can be reduced exponentially.

[0065] After setting the preset level of the tree, you can set the number of target key-value pairs stored in each tree, select any tree as the target tree, determine the number of key-value pairs that the target tree can store based on the obtained level of the target tree and the storage space of the target tree, and set the range of key-value pairs stored in each target tree based on the number of key-value pairs that the target tree can store.

[0066] For example, assuming that the number of key-value pairs that the target tree can store is 64, the logical address alignment is 8K, and the preset level of the target tree is 4 layers, the storage capacity of the target tree is 64*64*64*64*8K=128G.

[0067] Please refer to Table 1 below, which shows the correspondence between the tree levels and the number of nodes in this application. It can be seen that setting the preset tree levels to less than or equal to 4 layers can reduce the capacity of each tree and facilitate rapid positioning of the node.

[0068] Preset Levels Number of nodes Leaf 64*64*64 Level3 64*64 Level2 64 Root 1

[0069] Table 1

[0070] Step 403: Calculate the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level.

[0071] Specifically, the hierarchical address of the target node in the target hierarchy is obtained; based on the number of target key-value pairs stored in the target hierarchy, the address range of the target key-value pairs stored in the target hierarchy is obtained; based on the target logical address and the number of target key-value pairs stored in the target hierarchy, a modulo operation is performed to obtain a calculation result; based on the calculation result and the address range of the target key-value pairs stored in the target hierarchy, the target node position is determined.

[0072] The target node here is the leaf node in the target tree that actually stores the key-value pair. Get the hierarchical address of the target node, that is, get the address of the leaf node in the target tree, so that the target key-value pair can be obtained from the leaf node later. Get the address range of the target key-value pairs stored in the target hierarchy according to the number of target key-value pairs stored in the target hierarchy. See Figure 6 ,like Figure 6 As shown, each leaf node contains a fixed range of key values, namely Figure 6 The key:0-63, key:64-127, key:64*i-64*i+64 shown in the table, each fixed range includes 64 key values, and each leaf node records a key value that is a continuous integer within a fixed range. That is, each leaf node can store 64 key-value pairs. At this time, since the number of key-value pairs is known and the distribution of key-value pairs is known, the structure of the B+ tree can be pre-set in advance according to the B+ tree parameter configuration, for example, in the order of keys from small to large, from left to right, the maximum number of key-value pairs are filled into each leaf node. After the leaf node is determined, the key-value pair distribution of each level of each tree can be determined from bottom to top, until the root node. It should be pointed out here that, except for the leaf nodes, the key-value pairs in other tree nodes are used to record the pointers of their lower nodes. After the B+ tree is built, the B+ tree can be used as a base tree, and when data is dropped to disk, the key-value pair insertion operation can be performed directly according to the base tree structure. No matter how the number of key-value pairs and the order in which they are inserted change, the structure of the tree on disk remains unchanged, and the relative position relationship between the key-value pairs also remains unchanged. It is this structural invariance and the invariance of the relative position relationship that make it possible to design an efficient cache structure.

[0073] Subsequently, in this application, when allocating metadata belonging to different metadata trees, the hash value of the target position character of the metadata logical address is calculated according to the logical address contained in the metadata, and the tree to be selected whose hash value of the logical address is the same as the hash value of the target position byte in the tree to be selected is used as the target tree. In this way, the metadata corresponding to the logical addresses within the same address range can be organized in the same metadata tree, and in this application, it is necessary to save the address range corresponding to the logical address contained in the metadata stored in each metadata tree, so that when a certain logical address needs to be queried, it can be first determined that the metadata tree corresponding to the address range is the metadata tree that needs to implement metadata query, and then the address range to which the logical address belongs is determined, and then the logical address to be queried is located in the determined metadata tree, and the metadata includes a mapping relationship between the logical address and the physical address, so that the user can obtain the data to be queried based on the physical address. In addition, when determining the metadata tree of the address range based on the logical address, the first 16 bits of the logical address (other settings can also be made according to actual needs) bytes can be taken as the assigned key, and then the metadata tree to which the key belongs is searched as the metadata tree of the address range to which the logical address belongs. Specifically, the more bits the logical address has, the fewer the number of metadata contained in each metadata tree, so it corresponds to the number of metadata that needs to be contained in each metadata tree.

[0074] This application plans the location of each key (key value) and the relative position relationship between keys. Each location belongs to a fixed key, regardless of whether the key is used or not. When flushing the disk, the key-value pairs are inserted and dropped to the disk strictly according to this preset structure, which is conducive to the subsequent rapid positioning of the target node position.

[0075] Step 404: Process the target metadata corresponding to the target logical address according to the target node position.

[0076] After the target node position is obtained, the target metadata corresponding to the target logical address can be read and written according to the target node position.

[0077] In one embodiment, when a read processing operation is performed on the target metadata, processing the target metadata corresponding to the target logical address according to the target node position may include: determining whether a target key-value pair can be queried from the target node position, the target key-value pair storing a target logical address and a target physical address with a mapping relationship; if the target key-value pair can be queried from the target node position, then reading the target physical address in the target key-value pair; if the target key-value pair cannot be queried from the target node position, then obtaining the pointer information of the upper node of the target node position layer by layer until the target physical address stored in the parent node is obtained; if the target physical address stored in the parent node is an invalid value, then generating a query failure instruction and feeding back query failure information.

[0078] In a specific embodiment, when a read processing operation is performed on the target metadata, the target logical address of the metadata access request carrying the target logical address can be obtained, and the request can be assigned to the corresponding target tree for access according to the target logical address in the request, wherein the allocation method is to obtain the first n (adjustable) bytes (target position bytes) of the hash value of the target logical address as the assigned key (key value). Exemplarily, the hash value of the selected target position bytes can be hashKey (hash value) = LBA & 0xffff00000000000, and it is assigned to the corresponding tree according to hashKey.

[0079] Request to the corresponding target tree to traverse and access the target physical address corresponding to the target logical address; if it is a read operation, a read query operation, each layer can be directly located, according to the target logical address, according to the rule (key / levels[index] specifies the node of the level, key%64 the position of the node node), calculate the index of the node in the layer and its position in the node, that is, level[index]->node[position]; wherein, key is the key value, Levels[index] is the index of the level of the target tree, according to the key value, determine the target tree corresponding to the target logical address, obtain the preset level of the target tree, the index of the level of the target tree specifies the level, that is, any level in the target tree, based on the target logical address and the number of target key-value pairs stored in the target level, perform a modulo operation to obtain the calculation result; based on the calculation result and the address range of the target key-value pairs stored in the target level, determine the target node position according to the target node position obtained.

[0080] If the target key-value pair can be found from the target node position, that is, the leaf node is hit, then the Value value (target physical address) of the key-value pair stored in the target node position is read;

[0081] If the target key-value pair cannot be found from the target node position, that is, the leaf node is not hit, then NOT_FOUND is returned directly. If the node of the current level[index] (index of the target tree level) is empty, then search up layer by layer until the target physical address stored in the parent node is obtained. The parent node stores the PBA address of the node at night, and then reads the disk and searches again. If the PBA address of the node stored in the parent node is an invalid value (the target physical address stored in the parent node is an invalid value), it means that the node has not yet written data, and a query failure instruction is directly generated, and the query failure information is fed back, that is, NOT_FOUND is returned.

[0082] In another embodiment, when the target metadata is written, the target metadata corresponding to the target logical address is processed according to the target node position, including: determining whether the target key-value pair is stored in the target node position; if the target key-value pair is not stored in the target node position, inserting the key-value pair corresponding to the current target logical address and the target physical address into the target node position; if the target key-value pair is stored in the target node position, modifying the target physical address corresponding to the current target logical address.

[0083] In a specific implementation, when a write operation is performed on the target metadata, the target logical address can be obtained, and the index index of the node in the layer and the position of the node in the node are calculated according to the rule (key / Levels[index] specifies the node number of the level, key%64 the position of the node node), that is, level[index]->kode[position]; wherein key is the key value, Levels[index] is the index of the hierarchy of the target tree, the target tree corresponding to the target logical address is determined according to the key value, the preset hierarchy of the target tree is obtained, the index of the hierarchy of the target tree specifies the number of layers, that is, any hierarchy in the target tree, and a modulo operation is performed based on the target logical address and the number of target key-value pairs stored in the target hierarchy to obtain a calculation result; the target node position is determined based on the calculation result and the address range of the target key-value pairs stored in the target hierarchy according to the target node position obtained.

[0084] Determine whether the target key-value pair is stored in the target node position; if the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position, that is, if not found, insert a key-value pair; if the target key-value pair is stored in the target node position, modify the target physical address corresponding to the current target logical address, if found, modify the value corresponding to the LBA.

[0085] In one embodiment, when the memory metadata reaches a threshold, a disk flush will be performed. During the disk flush, the node will be assigned a PBA address (physical address) on the disk, and its parent node will be calculated according to the rules, and the physical address of the write node at the position within its parent node.

[0086] In this application, when setting the tree, the relative positions of the key and the key in the leaf node are pre-planned, and the key-value pair is inserted in the node position corresponding to the key according to the position of the key. The key-value pair is a mapping relationship between the physical address and the logical address. In this way, the key (logical address) can be directly given to directly locate the node like the bitmap algorithm, instead of searching layer by layer. According to the range distribution of the key, the structure of the tree is set. When flushing the disk, the tree is constructed according to the pre-set structure and the flushing operation is performed. Each layer can be directly located during the query operation; with a smaller processing delay, the overall performance of the system is improved. This method can obtain efficient data access.

[0087] In one embodiment, in response to receiving an instruction to write target metadata into a target tree, the memory occupied space of the target metadata and the remaining memory space of the target tree are obtained; it is determined whether the memory occupied space of the target metadata is greater than the remaining memory space of the target tree; if the memory occupied space of the target metadata is greater than the remaining memory space of the target tree, the remaining memory space of all trees in the system is traversed to determine whether there is any tree whose remaining memory space is greater than the memory occupied space of the target metadata; if there is any tree whose remaining memory space is greater than the memory occupied space of the target metadata, the target metadata is written into a tree whose remaining memory space is greater than the memory occupied space of the target metadata; if there is no tree whose remaining memory space is greater than the memory occupied space of the target metadata, the root nodes of all trees in the system are traversed to determine the invalidity ratio of the root nodes of all trees; if the invalidity ratio of the root nodes of all trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree whose root node is invalid is copied to the new root node.

[0088] Specifically, after receiving the instruction to write the target metadata into the target tree, the memory occupied space of the target metadata is obtained, that is, the size of the memory occupied by the target metadata is obtained, and the remaining memory space of the target tree is obtained, that is, the capacity allocated to the target tree that can currently accommodate data. By calculating whether the memory occupied by the target metadata is greater than the remaining memory space of the target tree, it can be determined whether the target tree can accommodate the target metadata. If the target tree cannot accommodate the target metadata, all trees in the system are traversed to determine whether there is a tree that can accommodate the target metadata. If there is no tree whose remaining memory space is greater than the memory occupied space of the target metadata, it means that all trees in the system cannot accommodate the target metadata, then the root nodes of all trees in the system are traversed, a new root node is created in the tree in the system, and the valid data in the root node is copied to the new root node. In this way, the situation of insufficient storage space can be dealt with.

[0089] In this application, the B+ tree corresponding to the metadata of the data volume is divided into multiple subtrees, and each subtree carries a certain amount of metadata, so that the number of layers of each B+ tree will not exceed 4. Although the space usage of its root node area is slightly increased, its access time can be reduced exponentially; the given key (LBA) can directly locate the node like a bitmap algorithm, instead of searching layer by layer, and the structure of the B+ tree is set according to the range distribution of the key. When flushing the disk, the B+ tree is constructed according to the pre-set structure and the disk flushing operation is performed. During the query operation, each layer can be directly located, resulting in a smaller processing delay.

[0090] It should be understood that although Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] In one embodiment, Figure 7 As shown, a metadata processing device is provided, including: an acquisition module 70, a calculation module 71 and a processing module 72, wherein:

[0092] An acquisition module 70 is used to acquire a target logical address, and based on the target logical address, acquire a target tree where the target logical address is located; acquire a target level of the target tree and the number of target key-value pairs stored in the target level;

[0093] A calculation module 71, configured to calculate a target node position of a target logical address in a target tree according to a target level and the number of target key-value pairs stored in the target level;

[0094] The processing module 72 is used to process the target metadata corresponding to the target logical address according to the target node position.

[0095] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0096] The target tree where the target logical address is obtained based on the target logical address includes:

[0097] In response to receiving a metadata access request carrying a target logical address, obtaining the target logical address carried in the metadata access request;

[0098] A hash value of the target logical address is calculated, a hash value of a target position of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target position of the target logical address.

[0099] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0100] The target level of the target tree is a preset level. Obtaining the target level of the target tree and the number of target key-value pairs stored in the target level includes:

[0101] The preset level of the target tree and the storage space of the target tree are obtained, and the number of key-value pairs that can be stored in the target tree is calculated based on the preset level and the storage space.

[0102] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0103] The target node position of the target tree where the target logical address is calculated based on the target level and the number of target key-value pairs stored in the target level includes:

[0104] Get the hierarchical address of the target node in the target hierarchy;

[0105] Obtaining an address range of a target key-value pair stored in the target layer based on the number of target key-value pairs stored in the target layer;

[0106] Perform a modulo operation based on the target logical address and the number of target key-value pairs stored in the target level to obtain a calculation result;

[0107] The target node location is determined based on the calculation result and the address range of the target key-value pair stored in the target level.

[0108] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0109] Processing the target metadata corresponding to the target logical address according to the target node location includes:

[0110] Determine whether a target key-value pair can be queried from the target node position, where the target key-value pair stores a target logical address and a target physical address with a mapping relationship;

[0111] If the target key-value pair can be found from the target node location, the target physical address in the target key-value pair is read;

[0112] If the target key-value pair cannot be found from the target node position, the pointer information of the upper node of the target node position is obtained layer by layer until the target physical address stored in the parent node is obtained;

[0113] If the target physical address stored in the parent node is an invalid value, a query failure instruction is generated and query failure information is fed back.

[0114] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0115] Processing the target metadata corresponding to the target logical address according to the target node position also includes:

[0116] Determine whether the target key-value pair is stored in the target node position;

[0117] If the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position;

[0118] If the target key-value pair is stored in the target node position, the target physical address corresponding to the current target logical address is modified.

[0119] In one embodiment, another implementation of the metadata processing method that can be implemented by the above device includes the following specific steps:

[0120] The method also includes:

[0121] In response to receiving an instruction to write target metadata into a target tree, obtaining a memory occupied space of the target metadata and a memory remaining space of the target tree;

[0122] Determine whether the memory space occupied by the target metadata is greater than the remaining memory space of the target tree;

[0123] If the memory space occupied by the target metadata is larger than the remaining memory space of the target tree, the remaining memory space of all trees in the system is traversed to determine whether there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0124] If there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata, the target metadata is written into the tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0125] If there is no tree whose remaining memory space is larger than the memory space occupied by the target metadata, then the root nodes of all trees in the system are traversed to determine the invalid ratio of the root nodes of all trees;

[0126] If the invalidity ratio of the root nodes of all trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree with invalid root nodes is copied to the new root node.

[0127] For the specific definition of the metadata processing device, please refer to the definition of the metadata processing method above, which will not be repeated here. Each module in the above metadata processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0128] In one embodiment, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the metadata processing methods provided by the above methods.

[0129] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data applied by the metadata processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a metadata processing method is implemented.

[0130] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0132] Step 401, obtain a target logical address, and obtain a target tree where the target logical address is located based on the target logical address.

[0133] Step 402, obtaining a target level of the target tree and the number of target key-value pairs stored in the target level.

[0134] Step 403: Calculate the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level.

[0135] Step 404: Process the target metadata corresponding to the target logical address according to the target node position.

[0136] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0137] The target tree where the target logical address is obtained based on the target logical address includes:

[0138] In response to receiving a metadata access request carrying a target logical address, obtaining the target logical address carried in the metadata access request;

[0139] A hash value of the target logical address is calculated, a hash value of a target position of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target position of the target logical address.

[0140] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0141] The target level of the target tree is a preset level. Obtaining the target level of the target tree and the number of target key-value pairs stored in the target level includes:

[0142] The preset level of the target tree and the storage space of the target tree are obtained, and the number of key-value pairs that can be stored in the target tree is calculated based on the preset level and the storage space.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] The target node position of the target tree where the target logical address is calculated based on the target level and the number of target key-value pairs stored in the target level includes:

[0145] Get the hierarchical address of the target node in the target hierarchy;

[0146] Obtaining an address range of a target key-value pair stored in the target layer based on the number of target key-value pairs stored in the target layer;

[0147] Perform a modulo operation based on the target logical address and the number of target key-value pairs stored in the target level to obtain a calculation result;

[0148] The target node location is determined based on the calculation result and the address range of the target key-value pair stored in the target level.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] Processing the target metadata corresponding to the target logical address according to the target node location includes:

[0151] Determine whether a target key-value pair can be queried from the target node position, where the target key-value pair stores a target logical address and a target physical address with a mapping relationship;

[0152] If the target key-value pair can be found from the target node location, the target physical address in the target key-value pair is read;

[0153] If the target key-value pair cannot be found from the target node position, the pointer information of the upper node of the target node position is obtained layer by layer until the target physical address stored in the parent node is obtained;

[0154] If the target physical address stored in the parent node is an invalid value, a query failure instruction is generated and query failure information is fed back.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] Processing the target metadata corresponding to the target logical address according to the target node position also includes:

[0157] Determine whether the target key-value pair is stored in the target node position;

[0158] If the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position;

[0159] If the target key-value pair is stored in the target node position, the target physical address corresponding to the current target logical address is modified.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] The method also includes:

[0162] In response to receiving an instruction to write target metadata into a target tree, obtaining a memory occupied space of the target metadata and a memory remaining space of the target tree;

[0163] Determine whether the memory space occupied by the target metadata is greater than the remaining memory space of the target tree;

[0164] If the memory space occupied by the target metadata is larger than the remaining memory space of the target tree, the remaining memory space of all trees in the system is traversed to determine whether there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0165] If there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata, the target metadata is written into the tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0166] If there is no tree whose remaining memory space is larger than the memory space occupied by the target metadata, then the root nodes of all trees in the system are traversed to determine the invalid ratio of the root nodes of all trees;

[0167] If the invalidity ratio of the root nodes of all trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree with invalid root nodes is copied to the new root node.

[0168] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0169] Step 401, obtain a target logical address, and obtain a target tree where the target logical address is located based on the target logical address.

[0170] Step 402, obtaining a target level of the target tree and the number of target key-value pairs stored in the target level.

[0171] Step 403: Calculate the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level.

[0172] Step 404: Process the target metadata corresponding to the target logical address according to the target node position.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] The target tree where the target logical address is obtained based on the target logical address includes:

[0175] In response to receiving a metadata access request carrying a target logical address, obtaining the target logical address carried in the metadata access request;

[0176] A hash value of the target logical address is calculated, a hash value of a target position of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target position of the target logical address.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] The target level of the target tree is a preset level. Obtaining the target level of the target tree and the number of target key-value pairs stored in the target level includes:

[0179] The preset level of the target tree and the storage space of the target tree are obtained, and the number of key-value pairs that can be stored in the target tree is calculated based on the preset level and the storage space.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] The target node position of the target tree where the target logical address is calculated based on the target level and the number of target key-value pairs stored in the target level includes:

[0182] Get the hierarchical address of the target node in the target hierarchy;

[0183] Obtaining an address range of a target key-value pair stored in the target layer based on the number of target key-value pairs stored in the target layer;

[0184] Perform a modulo operation based on the target logical address and the number of target key-value pairs stored in the target level to obtain a calculation result;

[0185] The target node location is determined based on the calculation result and the address range of the target key-value pair stored in the target level.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] Processing the target metadata corresponding to the target logical address according to the target node location includes:

[0188] Determine whether a target key-value pair can be queried from the target node position, where the target key-value pair stores a target logical address and a target physical address with a mapping relationship;

[0189] If the target key-value pair can be found from the target node location, the target physical address in the target key-value pair is read;

[0190] If the target key-value pair cannot be found from the target node position, the pointer information of the upper node of the target node position is obtained layer by layer until the target physical address stored in the parent node is obtained;

[0191] If the target physical address stored in the parent node is an invalid value, a query failure instruction is generated and query failure information is fed back.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] Processing the target metadata corresponding to the target logical address according to the target node position also includes:

[0194] Determine whether the target key-value pair is stored in the target node position;

[0195] If the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position;

[0196] If the target key-value pair is stored in the target node position, the target physical address corresponding to the current target logical address is modified.

[0197] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0198] The method also includes:

[0199] In response to receiving an instruction to write target metadata into a target tree, obtaining a memory occupied space of the target metadata and a memory remaining space of the target tree;

[0200] Determine whether the memory space occupied by the target metadata is greater than the remaining memory space of the target tree;

[0201] If the memory space occupied by the target metadata is larger than the remaining memory space of the target tree, the remaining memory space of all trees in the system is traversed to determine whether there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0202] If there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata, the target metadata is written into the tree whose remaining memory space is larger than the memory space occupied by the target metadata;

[0203] If there is no tree whose remaining memory space is larger than the memory space occupied by the target metadata, then the root nodes of all trees in the system are traversed to determine the invalid ratio of the root nodes of all trees;

[0204] If the invalidity ratio of the root nodes of all trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree with invalid root nodes is copied to the new root node.

[0205] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0206] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A metadata processing method, characterized in that: include: Obtain a target logical address, and based on the target logical address, obtain a target tree where the target logical address is located; Get the target level of the target tree and the number of target key-value pairs stored in the target level; Calculate the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level; The target metadata corresponding to the target logical address is processed according to the target node position.

2. The method according to claim 1, characterized in that: The acquiring the target tree where the target logical address is located based on the target logical address comprises: In response to receiving a metadata access request carrying a target logical address, obtaining the target logical address carried in the metadata access request; A hash value of the target logical address is calculated, a hash value of a target position of the target logical address is obtained, and a target tree address corresponding to the target logical address is calculated based on the hash value of the target position of the target logical address.

3. The method according to claim 1, characterized in that The target level of the target tree is a preset level, and obtaining the target level of the target tree and the number of target key-value pairs stored in the target level includes: A preset level of a target tree and a storage space of the target tree are obtained, and the number of key-value pairs that can be stored in the target tree is calculated based on the preset level and the storage space.

4. The method according to claim 1, characterized in that: The step of calculating the target node position of the target logical address in the target tree according to the target level and the number of target key-value pairs stored in the target level comprises: Get the hierarchical address of the target node in the target hierarchy; Based on the number of target key-value pairs stored in the target level, an address range of the target key-value pairs stored in the target level is obtained; based on the target logical address and the number of target key-value pairs stored in the target level, a modulo operation is performed to obtain a calculation result; The target node location is determined based on the calculation result and the address range of the target key-value pairs stored in the target level.

5. The method according to claim 1, characterized in that The processing of the target metadata corresponding to the target logical address according to the target node position includes: Determine whether a target key-value pair can be queried from the target node position, wherein the target key-value pair stores a target logical address and a target physical address having a mapping relationship; If the target key-value pair can be found from the target node location, the target physical address in the target key-value pair is read; If the target key-value pair cannot be found from the target node position, the pointer information of the upper node of the target node position is obtained layer by layer until the target physical address stored in the parent node is obtained; If the target physical address stored in the parent node is an invalid value, a query failure instruction is generated and query failure information is fed back.

6. The method according to claim 1, characterized in that The processing of the target metadata corresponding to the target logical address according to the target node position further includes: Determine whether the target key-value pair is stored in the target node position; If the target key-value pair is not stored in the target node position, insert the key-value pair corresponding to the current target logical address and the target physical address into the target node position; If the target key-value pair is stored in the target node position, the target physical address corresponding to the current target logical address is modified.

7. The method according to claim 1, characterized in that The method further comprises: In response to receiving an instruction to write target metadata into a target tree, obtaining a memory occupied space of the target metadata and a memory remaining space of the target tree; Determine whether the memory space occupied by the target metadata is greater than the remaining memory space of the target tree; If the memory occupied by the target metadata is larger than the remaining memory space of the target tree, traverse the remaining memory space of all trees in the system to determine whether there is any tree whose remaining memory space is larger than the memory occupied by the target metadata; If there is any tree whose remaining memory space is larger than the memory space occupied by the target metadata, writing the target metadata into the tree whose remaining memory space is larger than the memory space occupied by the target metadata; If there is no memory remaining space of any of the trees that is larger than the memory occupied space of the target metadata, traverse the root nodes of all the trees in the system to determine the invalid ratio of the root nodes of all the trees; If the invalidity ratio of the root nodes of all the trees is greater than a set threshold, a new root node is created, and the valid metadata in the tree with invalid root nodes is copied to the new root node.

8. A metadata processing device, characterized in that: The device comprises: An acquisition module is used to acquire a target logical address, and based on the target logical address, acquire a target tree where the target logical address is located; acquire a target level of the target tree and the number of target key-value pairs stored in the target level; A calculation module, used for calculating a target node position of a target logical address in a target tree according to the target level and the number of target key-value pairs stored in the target level; A processing module is used to process the target metadata corresponding to the target logical address according to the target node position.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Metadata organization method, apparatus and device, and computer readable storage medium

    CN111104435A

  • Metadata access method, device and apparatus and readable storage medium

    CN111125447A

  • Address processing method and device based on storage system, storage system and medium

    CN117573676A