Data processing method and device and server

Abstract

The invention discloses a data processing method and device and a server, and can be applied to the technical field of data storage. The method comprises the following steps: acquiring a first mapping value, wherein the first mapping value belongs to a first mapping range; splitting the first mapping range into a second mapping range and a third mapping range by taking the first mapping value as a first splitting point; and splitting the first fragment into a second fragment and a third fragment by taking the position of the metadata of the object corresponding to the first mapping value in the first fragment as a second splitting point, wherein the mapping value corresponding to the object in the second fragment belongs to a second mapping range, and the mapping value corresponding to the object in the third fragment belongs to a third mapping range; and associating the second mapping range with the second fragment, and associating the third mapping range with the third fragment. According to the method, the resource consumption of fragment splitting can be reduced, and the fragment splitting efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, device and server. Background Art

[0002] In the field of data storage technology, in addition to storing the object's own data, it is also necessary to store the object's metadata. The object's metadata is stored in the index node (inode). The inodes of objects in the same directory are recorded in the same inode distribution table. Among them, the object's inode refers to the inode that stores the object's metadata. The objects in the directory refer to the files and first-level subdirectories in the directory.

[0003] In order to improve the concurrent access level of the inode distribution table, the inode distribution table is split into multiple shards. As the number of inodes recorded in a shard increases, the shard needs to be split to ensure the concurrent access level of the shard.

[0004] Currently, the shard splitting scheme consumes a lot of resources such as processor resources and memory resources, and is time-consuming. Summary of the invention

[0005] The embodiments of the present application propose a data processing method, device and server, which can reduce the resource consumption of shard splitting and improve the efficiency of shard splitting.

[0006] In a first aspect, a data processing method is provided. The method can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a size order of the mapping values ​​corresponding to the multiple objects.

[0007] The method includes: obtaining a first mapping value, the first mapping value belongs to a first mapping range; using the first mapping value as a first splitting point, splitting the first mapping range into a second mapping range and a third mapping range; and using the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point, splitting the first shard into a second shard and a third shard; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; associating the second mapping range with the second shard, and associating the third mapping range with the third shard. The first splitting point is a splitting point for splitting the first mapping range into the second mapping range and the third mapping range, and the second splitting point is a splitting point for splitting the first shard into the second shard and the third shard.

[0008] The method can be used to manage the inode distribution table and shard map of the directory. The first shard is a shard of the inode distribution table of the directory, and the shard map records the association between the first shard and the first mapping range. After the first shard is split into the second shard and the third shard, and the first mapping range is split into the second mapping range and the third mapping range, the association between the second shard and the second mapping range, and the association between the third shard and the third mapping range can be recorded in the shard map.

[0009] The mapping value corresponding to the object is a value obtained by calculating the identifier of the object according to a mapping algorithm (such as a hash range partitioning algorithm, a key range partitioning algorithm, etc.).

[0010] The shard map records the mapping ranges associated with different shards of the inode distribution table. When accessing the metadata of an object, the mapping range is obtained based on the mapping value corresponding to the object. The shard associated with the mapping range is the shard that records the metadata of the object, so the metadata of the object can be accessed in the shard. Due to the association between the mapping range and the shard, that is, the mapping values ​​of objects in different shards belong to different mapping ranges, which requires that different mapping ranges cannot overlap.

[0011] The method provided in the embodiment of the present application can split the mapping range associated with the shard based on the mapping value as the mapping range splitting point when splitting the shard; and split the shard with the position of the metadata of the object corresponding to the mapping value in the shard as the shard splitting point. Since the arrangement order of the metadata of the objects in the shard is consistent with the size order of the mapping values ​​corresponding to the objects, it can be ensured that there is no intersection in the mapping range after the split, and that different shards after the split belong to different mapping ranges. Then, the split shards are respectively associated with the corresponding mapping ranges. In this way, the splitting of the shards can be completed while ensuring that there is no intersection in the mapping range after the split, and that different shards after the split belong to different mapping ranges.

[0012] Since this method does not need to copy metadata or snapshots of metadata, nor does it need incremental writeback, it consumes very little processor resources, memory resources, etc., and the splitting speed is fast and takes very little time, so it can quickly improve the concurrent access level of metadata and eliminate access hot spots. Furthermore, the splitting consumes very little resources and takes very little time, which can ensure that the shards are always small, facilitate the migration of shards between different storage nodes, and enable rapid load balancing and expansion and contraction between storage nodes.

[0013] In a possible implementation, the index structure of the first shard includes multiple leaf nodes, and the leaf nodes record metadata of one or more objects among multiple objects; wherein the arrangement order of the multiple leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the multiple leaf nodes; in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects; taking the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point, the first shard is split into the second shard and the third shard, including: taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into the second leaf node and the third leaf node; wherein the second leaf node and the fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and the fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value. wherein the third splitting point is also called a leaf node splitting point, which is a splitting point used to split the leaf node.

[0014] In this implementation, the arrangement order of the leaf nodes of the shard's index structure is consistent with the size order of the mapping values ​​corresponding to the objects in the leaf nodes, and, in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects. In this way, when the shard is split based on the first mapping value, the position of the metadata of the object corresponding to the first mapping value in the leaf node is used as the splitting point to split the leaf node into two leaf nodes. Among them, the leaf node on the metadata side of the object corresponding to the first mapping value can be used as the leaf node of the index structure of one shard after the split, and the leaf node on the other side can be used as the leaf node of the index structure of another shard after the split. In this way, the splitting of the leaf nodes of the shard's index structure can be completed.

[0015] In one possible implementation, the index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; the first shard is split into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as the second split point, including: based on the key of the object corresponding to the first mapping value and the key range recorded in the first intermediate node, the first intermediate node is split into a second intermediate node and a third intermediate node; wherein the key range recorded in the second intermediate node includes the key of the object in the second leaf node, and the key range recorded in the third intermediate node includes the key of the object in the third leaf node.

[0016] In this implementation, when splitting the shard based on the first mapping value, based on the key corresponding to the first mapping value and the key range to which the first leaf node belongs, the intermediate node recording the key range to which the first leaf node belongs is split into an intermediate node recording the key range to which the second leaf node belongs and an intermediate node recording the key range to which the third leaf node belongs, thereby completing the splitting of the intermediate node.

[0017] In one possible implementation, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the method also includes: obtaining an access request, the access request includes a second mapping value, the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; based on the pointer, querying the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0018] Among them, the metadata of the object corresponding to the second mapping value can be queried in the second leaf node first; if the metadata of the object corresponding to the second mapping value is not found in the second leaf node, based on the pointer, the metadata of the object corresponding to the second mapping value is queried in the third leaf node.

[0019] In this implementation, there is a pointer between the second leaf node and the third leaf node after the split. In this way, when an access request for the first leaf node is received during the splitting of the first shard, if the metadata of the object to be accessed by the access request cannot be found in the second leaf node, the metadata of the object to be accessed by the access request can be queried in the third leaf node through the pointer. In other words, the splitting of the shard does not affect the execution of the access request, so when splitting the shard, there is no need to incrementally write back the shard, nor is there a need to lock the shard. In short, the splitting of the shard will not affect the business or will have little impact on the business.

[0020] In one possible implementation, the fourth shard is associated with a fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the method also includes: merging the fourth mapping range and the first mapping range into a fifth mapping range; and merging the first shard and the fourth shard into a fifth shard; and associating the fifth mapping range with the fifth shard.

[0021] Before the first shard and the fourth shard are merged into the fifth shard, the shard map also records the fourth mapping range and the fourth shard association. After the first shard and the second shard are merged into the fifth shard, the fifth mapping range and the fifth shard association are recorded in the shard map. In this way, through the shard map and the fifth mapping range, the shard (i.e., the fifth shard) where the metadata of the objects in the original first shard and the original fourth shard are located can be queried.

[0022] In this implementation, the mapping ranges associated with the shards to be merged are adjacent. The shards to be merged can be directly merged, and the mapping ranges associated with the shards to be merged can be merged. In the shard map, the association between the merged shards and the merged mapping ranges can be recorded to complete the merging of the shards.

[0023] This implementation does not require copying metadata or metadata snapshots, nor does it require incremental writeback, thereby consuming very little processor resources, memory resources, etc., and the merging speed is fast and the time consumption is extremely short. The merging of shards can be completed quickly, and during the merging period, metadata can be written normally without causing metadata writing interruption.

[0024] In one possible implementation, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; the first shard and the fourth shard are merged into a fifth shard, including: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, the first root node and the second root node are merged to obtain the root node of the index structure of the fifth shard; and, the fifth mapping range is corresponded to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, the fifth mapping range is corresponded to the first root node to obtain the root node of the index structure of the fifth shard; and, a pointer is created from the root node of the index structure of the fifth shard to the second root node.

[0025] In this implementation, if the index structures of the shards to be merged have the same height, the root nodes of the index structures can be directly merged to obtain the root nodes of the index structures of the merged shards. If the index structures of the shards to be merged have different heights, the mapping range corresponding to the root node of the lower index structure can be mapped to the root node of the higher index structure, and a pointer from the root node of the higher index structure to the root node of the lower index structure can be constructed, so that the root node of the lower index structure becomes the intermediate node of the merged index structure.

[0026] In a possible implementation, the size of the object's key is positively correlated with the size of the mapping value corresponding to the object, and the object's key is used to query metadata of the object in the first shard.

[0027] In this way, when the metadata of the object is stored in the shard based on the object's key, it can be ensured that the arrangement order of the object's metadata in the shard is consistent with the size order of the mapping values ​​corresponding to the object.

[0028] In a second aspect, a data processing device is provided, wherein a first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the device comprises: an acquisition module, used to acquire the first mapping value, the first mapping value belongs to the first mapping range; a splitting module, used to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as the first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point; wherein the mapping values ​​corresponding to the objects in the second shard belong to the second mapping range, and the mapping values ​​corresponding to the objects in the third shard belong to the third mapping range; an association module, used to record the association between the second mapping range and the second shard, and the association between the third mapping range and the third shard in the shard map.

[0029] In a possible implementation, the index structure of the first shard includes multiple leaf nodes, and the leaf nodes record metadata of one or more objects among multiple objects; wherein the arrangement order of the multiple leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the multiple leaf nodes; in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects; the splitting module is used to: split the first leaf node into a second leaf node and a third leaf node with the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point; wherein the second leaf node and the fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and the fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value.

[0030] In a possible implementation, the index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; the splitting module is used to: split the first intermediate node into a second intermediate node and a third intermediate node based on the key of the object corresponding to the first mapping value and the key range recorded in the first intermediate node; wherein the key range recorded in the second intermediate node includes the key of the object in the second leaf node, and the key range recorded in the third intermediate node includes the key of the object in the third leaf node. .

[0031] In one possible implementation, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device also includes: a query module; the acquisition module is also used to: obtain an access request, the access request includes a second mapping value, the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; the query module is used to: based on the pointer, query the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0032] In one possible implementation, the fourth shard is associated with the fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein the mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device also includes a merging module; the merging module is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first shard and the fourth shard into a fifth shard; the association module is also used to: associate the fifth mapping range with the fifth shard.

[0033] In one possible implementation, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; the merging module is used to: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merge the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and, correspond the fifth mapping range to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, correspond the fifth mapping range to the first root node to obtain the root node of the index structure of the fifth shard; and, create a pointer from the root node of the index structure of the fifth shard to the second root node.

[0034] In a possible implementation, the size of the object's key is positively correlated with the size of the mapping value corresponding to the object, and the object's key is used to query metadata of the object in the first shard.

[0035] In a third aspect, a storage node is provided, including a processor and a memory; the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method provided in the first aspect.

[0036] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0037] In a fifth aspect, a computer program product is provided, comprising instructions for implementing the method provided in the first aspect.

[0038] In a sixth aspect, a server is provided, comprising a processor and a memory; the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method provided in the first aspect.

[0039] In the seventh aspect, a data processing method is provided, which can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the method includes: taking the first mapping value as the first splitting point, splitting the first mapping range into a second mapping range and a third mapping range; and taking the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point, splitting the first shard into a second shard and a third shard; wherein the first mapping value belongs to the first mapping range, the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; the second mapping range is used to associate with the second shard, and the third mapping range is used to associate with the third shard.

[0040] In an eighth aspect, a data processing device is provided, which can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the device includes: a splitting module, used to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as the first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point; wherein the first mapping value belongs to the first mapping range, the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; the second mapping range is used to associate with the second shard, and the third mapping range is used to associate with the third shard.

[0041] The beneficial effects of the second to eighth aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A A schematic diagram of the structure of a key provided in an embodiment of the present application;

[0043] Figure 1B A schematic diagram of the structure of a storage system provided in an embodiment of the present application;

[0044] Figure 2A schematic diagram of the structure of a storage node provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of an inode distribution table access scheme provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of an inode distribution table access scheme provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a leaf node splitting solution provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of an intermediate node splitting solution provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of a root node splitting solution provided in an embodiment of the present application;

[0052] Fig. 10A A schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0053] Fig. 10B A schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0054] Fig.11 A schematic diagram of a slicing method provided in an embodiment of the present application;

[0055] Fig.12 A schematic diagram of splitting a sharded index structure provided in an embodiment of the present application;

[0056] Fig.13 A flowchart of a data processing method provided in an embodiment of the present application;

[0057] Fig.14 A schematic diagram of merging a sharded index structure provided in an embodiment of the present application;

[0058] Fig.15 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0059] Fig.16A A schematic diagram of the structure of a storage node provided in an embodiment of the present application;

[0060] Fig. 16B A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The scheme provided by the embodiment of the present application will be described below in conjunction with the accompanying drawings. In the embodiment of the present application, "multiple" means two or more, and "multiple" means two or more. "First", "second", etc. are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0062] To facilitate understanding of the solutions provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0063] Object: A general term for files and directories. Objects in a directory refer to files and first-level subdirectories stored in the directory.

[0064] Object metadata: data used to describe the content and properties of an object, such as the name of the object, the size of the object, the modification and access time of the object, the permissions of the object, etc. Object metadata is used for classification, organization, labeling, sorting and searching of objects.

[0065] Inode: is a data structure used to record metadata of objects in a distributed file system. The inode that records the metadata of an object can be called the inode of the object.

[0066] Inode distribution table: a data table used to record the inodes of objects in a directory. The inode distribution table is a horizontal table, that is, one row records the inode of one object.

[0067] Tuple: also known as data row, refers to the row in the inode distribution table. A tuple represents a row in the inode record table, that is, the metadata of an object.

[0068] Shard: Also known as index shard, it is a table shard that includes multiple tuples in the inode distribution table. In other words, a shard is obtained by dividing the inode distribution table, and the inode distribution table consists of multiple shards.

[0069] Mapped value: refers to the mapping value corresponding to the object obtained by mapping the object identifier (such as the name of the object) according to a preset mapping algorithm (such as a hash range partitioning algorithm, a key range partitioning algorithm, etc.). The mapped value corresponding to the object can also be referred to as the mapped value of the object. The mapped value has a size and can be represented by a positive integer.

[0070] Key: An index key, where the index structure maintains a mapping relationship from key to value, thereby supporting fast value search by key. In an embodiment of the present application, the size of an object's key is positively correlated with the size of the mapping value corresponding to the object. The object's key may be composed of the mapping value corresponding to the object and the object's identifier. For example, Figure 1A As shown, the key of an object can be a 2-column key consisting of a mapping value corresponding to the object and an identifier of the object. The mapping value of an object can be called a prefix of the key of the object, and the identifier of the object can be called the body of the key of the object. The size of the key is positively correlated with the mapping value included in the key. In other words, when the mapping values ​​included in the key are different, the size of the key is determined by the mapping value. When the mapping values ​​of the keys are the same, the size of the key is determined by the size of the identifier of the object. Among them, the size of the identifier of the object can be represented by the size order of the lexicographic order corresponding to the identifier.

[0071] Key range: The range of keys of objects in the same data set (for example, the same shard, the same leaf node in the index structure).

[0072] Mapping range: refers to a set of multiple mapping values ​​that are adjacent to each other in sequence. The mapping range can be expressed as a closed interval. The mapping values ​​that make up the mapping range are the endpoints at both ends of the interval and the integers between the two ends. The upper endpoint of the mapping range is the maximum mapping value in the mapping range. Among them, the inode distribution table and the shards are both associated with the mapping range. Among them, the mapping range corresponding to the inode distribution table is composed of all the allocatable mapping values ​​of the inode distribution table. Among them, the total number of allocatable mapping values, and the minimum mapping value and the maximum mapping value among the allocatable mapping values ​​can be preset. The mapping range corresponding to the shard is a sub-range in the mapping range associated with the inode distribution table, and the mapping ranges associated with different shards do not intersect.

[0073] Mapping algorithm: refers to an algorithm that maps an object's key to a mapping value. Common mapping algorithms include hash range partitioning algorithm, key range partitioning algorithm, etc.

[0074] Consistent hashing algorithm: A hash algorithm used to calculate the mapping value of an object, which belongs to the hash range partitioning algorithm. In an embodiment of the present application, the consistent hashing algorithm is an algorithm that performs hash calculation on the identifier (such as the name) of the object based on the total number of assignable mapping values, wherein the obtained hash value is used as the mapping value of the object. The commonly used consistent hashing algorithm is: taking the remainder based on the total number of assignable mapping values. For example, the name of the object is hashed, and the obtained hash value is divided by the total number of assignable mapping values, and the remainder obtained is used as the mapping value of the object.

[0075] Shard map: One inode distribution table corresponds to one shard map, which is used to calculate or query the shard where the metadata of the object in the inode distribution table is located. The shard map records the association between the mapping range and the shard. Specifically, the shard map records the mapping relationship between the mapping range and the shard information (such as the storage address of the shard, identity (ID), etc.). More specifically, the shard map records the start mapping value and the end mapping value of the mapping range, as well as the ID of the shard associated with the mapping value and the storage location of the shard associated with the shard (that is, the identifier of the node where the shard is located). The shard can be named with the shard ID as a suffix to support the positioning of the index shard. The shard map is stored and managed on the disk as a shard. After the shard map is loaded into the memory, it will be organized into an ordered array to support fast binary search.

[0076] Directory table: a data table that records metadata used for path resolution (i.e., metadata of the directory).

[0077] Directory metadata: including directory ID, parent directory ID, directory name, permissions, directory node group ID and other information. The directory table can be replicated to all storage nodes to support low-overhead path resolution. Among them, the detailed information of each directory in the directory table (such as size, modification time, etc.) is maintained in the inode distribution table of the parent directory of the directory. The directory table only maintains information used to resolve the directory path.

[0078] Split point: refers to the split position where the shard or mapping range is split, that is, the shard or mapping range is split at the split point.

[0079] The parent directory of a directory: is the directory where the directory is located, that is, the directory is the first-level subdirectory of its parent directory.

[0080] Node group: consists of one or more storage nodes. The storage nodes in the node group of a directory refer to the storage nodes that store the inode distribution table of the directory, that is, the storage nodes where the shards that make up the inode distribution table are located.

[0081] A distributed file system (DFS) means that the physical storage resources managed by the file system (FS) are not necessarily directly connected to the local storage nodes, but are connected to the storage nodes through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions or volumes.

[0082] Tree index structure: It is a tree-shaped index structure, usually with multiple layers, each layer including at least one page node. A page node represents a storage page (page). In the tree index structure, the page nodes in the bottom layer are called leaf nodes, the page nodes in the previous layer of the leaf nodes can be called intermediate nodes, and the top-level page nodes are called root nodes. Among them, when the tree index structure has only two layers, the root node is also called the intermediate node. Each intermediate node is associated with one or more page nodes of the next layer. Among them, the intermediate node is the parent page node of the page node of the next layer associated with the intermediate node.

[0083] B-link tree: It is a tree index structure and a variant of B+ tree. Each page node in the B-link tree stores a pointer to the right sibling node of the page node. Among them, the minimum key recorded in the right sibling node of the page node is adjacent to the maximum key recorded in the page node. In this way, when a page node is full, for example, the storage space is full (8K page is full), a new right sibling node can be split from the page node, and the metadata of the object corresponding to half of the keys recorded in the page node is recorded to the right sibling node, and a pointer to the right sibling node is set, and then the maximum key of the page node and the pointer to the right sibling node are inserted into the parent page node of the page node. If the parent page node is also full during this process, internal splitting will occur iteratively. The B-link tree can ensure the correctness of other concurrent operations in progress without locking the entire tree during splitting. For example, if the key to be searched is located in the right sibling page node after the split, since the parent page node has not yet inserted a pointer to the right sibling page node, it is still routed to the page node where the split occurred. At this time, it will be found that the key to be searched is greater than the maximum key of the page node, and it will jump to the right sibling page node through the right pointer of the page node.

[0084] Pointer to page node: refers to the identifier of the page node. As mentioned above, a page node is a storage page, and the pointer is the identifier of the storage page. The identifier of the storage page may also be the address of the storage page.

[0085] The key in the page node refers to the key of the object to which the metadata recorded in the page node belongs.

[0086] Multi-version concurrency control (MVCC): MVCC is a technology commonly used in relational databases to implement ACID isolation in transactions. ACID stands for atomicity, consistency, isolation, and durability.

[0087] Write ahead log (WAL): refers to a mechanism in which modifications are not written directly to the database file, but to the database log. If a transaction fails, the record in the database log will be ignored and the modification will be undone. If the transaction succeeds, it will be written back to the database file at a later time and the modification will be committed. The database uses a write ahead log mechanism to ensure persistence and fault recovery.

[0088] The above descriptions are some technical terms that may be involved in the embodiments of the present application, but not all technical terms. For technical terms involved in the embodiments of the present application that are not explained in the specification, reference can be made to the introduction of the prior art.

[0089] As mentioned above, the metadata of an object is used for classification, organization, labeling, sorting, and searching of the object, so the access to the inode distribution table is frequent and diverse. Some accesses may conflict, and it is difficult to perform these accesses on the same shard of the inode distribution table at the same time, resulting in a low level of concurrent access to the inode distribution table. For example, the modification operation on a shard requires locking the shard, so it is difficult to perform other operations on the shard while performing a modification operation on the shard.

[0090] As the business progresses, more and more object metadata is recorded in the same shard. This results in more frequent access to the shard and more conflicting accesses. Therefore, the more object metadata is recorded in a shard, the lower the concurrent access level of the shard.

[0091] In order to improve the concurrent access level of a shard, the shard can be split into two shards. Compared with the shard before the split, the shard after the split records less metadata of objects, which can ensure a higher concurrent access level.

[0092] Furthermore, if the shards deployed on a storage node record a large amount of metadata for objects, the storage node needs to process a large number of access requests, resulting in access hotspots. Therefore, it is necessary to split the shards on the storage node and migrate some of the split shards to other storage nodes to eliminate access hotspots.

[0093] As mentioned above, the mapping ranges associated with the shards do not intersect, so the mapping ranges associated with the shards after the split are also disjoint. The mapping range of the source shard (i.e., the shard before the split) can be set as mapping range A1, and the mapping ranges of the shards B1 and B2 obtained by the split are mapping range A11 and mapping range A12 respectively, and mapping range A11 and mapping range 12 do not intersect. When splitting, it is necessary to record the metadata of the objects corresponding to the mapping values ​​in mapping range A11 from the source shard to shard B1, and the metadata of the objects corresponding to the mapping values ​​in mapping range A12 to shard B2.

[0094] In addition, in order to save storage resources, etc., when the shards are small, they need to be merged. Among them, the small shards mean that the shards record the metadata of fewer objects.

[0095] In one solution, when splitting, create shard B1; then, copy the snapshot of the metadata of the object corresponding to the mapping value in the mapping range A11 in the source shard to shard B1. After the snapshot copy is completed, incremental write-back is required. Specifically, during the snapshot copy, the source shard may have written metadata of a new object. If the mapping value of the new object belongs to the mapping range A11, it is necessary to continue to copy the metadata of the new object to shard B1. In this way, the aforementioned process is iterated until the source shard has written very little metadata during the snapshot copy, or the number of iterations reaches a preset number, the source shard is locked, and the final incremental write-back is performed. After the final incremental write-back is completed and the shard map split is completed, the source shard is unlocked to complete the split. Among them, the metadata of the object corresponding to the mapping value in the mapping range A11 can be deleted from the source shard to obtain shard B2. This solution requires snapshot copying and incremental write-back, which consumes a large amount of processor resources and memory resources. In addition, the locked source shard in this solution may cause a long metadata write interruption.

[0096] The method of merging shards in this solution is similar to the method of splitting shards. Specifically, one shard to be merged is used as the source shard, and a snapshot of the source data in the source shard is copied to another shard to be merged. After the snapshot is completed, incremental write-back is performed, etc. Therefore, the method of merging shards in this solution also consumes a large amount of processor resources and memory resources, and may cause metadata write interruption.

[0097] The embodiment of the present application provides a data processing method. The method records the metadata of the objects in the directory in the shards of the inode distribution table of the directory according to the size of the mapping value corresponding to the object, so that the arrangement order of the metadata of the objects in the shards is consistent with the size order of the mapping value corresponding to the object. When splitting the shard, the mapping range associated with the shard can be split into two mapping ranges with the mapping value as the splitting point S1; and the shard can be split into two shards with the position of the metadata of the object corresponding to the mapping value in the shard as the splitting point S2; then, in the shard map, the association relationship between the split mapping range and the split shard is recorded. In this way, the splitting of the shard can be completed.

[0098] The split point S1 is also called a mapping range split point, which is a split point used to split a mapping range. The split point S1 can be a mapping value. The split point S2 is also called a shard split point, which is a split point used to split a shard. The split point S2 can be the location of the metadata of the object in the shard.

[0099] This solution does not require copying metadata or metadata snapshots, nor does it require incremental writeback, and consumes very little processor resources, memory resources, etc. Since there is no need to copy metadata or metadata snapshots, nor does it require incremental writeback, the splitting speed is fast and the time consumption is extremely short, which can quickly improve the level of concurrent access to metadata and eliminate access hot spots and other problems. In addition, the splitting consumes very few resources and takes very little time, which can ensure that the shards are always small (that is, the metadata of fewer objects is recorded), facilitate the migration of shards between different storage nodes, and enable rapid load balancing and expansion and contraction between storage nodes.

[0100] Next, the data processing method provided in the embodiment of the present application is introduced.

[0101] The data processing method provided in the embodiment of the present application can be applied to the storage system 100. In some embodiments, the storage system 100 can be a distributed file system, such as a Hadoop distributed file system (HDFS) or a Ceph file system (cephFS). In some embodiments, the storage system 100 can be a distributed cache file system, such as Alluxio. In some embodiments, the storage system 100 can be a stand-alone system, that is, the storage system 100 is a single device, such as a single physical device or a virtual device.

[0102] Figure 1BA possible implementation form of the storage system 100 is shown. The storage system 100 may include multiple storage nodes, for example, storage node 110, storage node 120, etc. The storage node in the storage system 100 may be any device or equipment with data computing function, storage function and communication function, such as a server, a virtual machine (VM), a container, etc.

[0103] Storage nodes can act as data nodes to manage and store metadata of objects. Figure 1B As shown, a storage node may store one or more shards and a shard map, wherein the shards and shard map stored by the storage node correspond to the same directory.

[0104] In some embodiments, see Figure 2 , it can be set that storage node 110 stores shard C1 and shard C2. The index structure of shard C1 and shard C2 can be a B-link tree, that is, the page node of the index structure has a pointer to the right brother node. In this way, when splitting the shard, there is no need for incremental writeback and lock shard. The details will be introduced below and will not be repeated here.

[0105] The storage node includes a management module. The management module is used to manage shards and shard maps, for example, splitting shards, and updating shard maps after splitting shards. In some embodiments, the management module can ensure ACID of transactions within a single node through MVCC and WAL. The functions of the management module will be specifically introduced below in conjunction with the method embodiment, and will not be repeated here.

[0106] The storage node may also include a coordination module. Through the coordination module, the storage node can act as a coordinator node to receive access requests sent by the client, and in response to the access request, use the shard map to access the shard to access the metadata of one or more objects in the shard. Specifically, before accessing the data of the object itself, the client needs to access the metadata of the object, such as writing or querying the metadata of the object. Among them, before creating an object, it is necessary to write the metadata of the object in the inode distribution table corresponding to the directory where the object is located (that is, write the metadata of the object to the corresponding shard of the inode distribution table). Before opening an object, it is necessary to query the metadata of the object in the inode distribution table corresponding to the directory where the object is located.

[0107] In some embodiments, Figure 1B As shown in , the storage node stores a directory table, which records the node group ID of the directory. The shards of the directory refer to the shards that make up the inode distribution table of the directory. Figure 3 As shown in the figure, the directory table records the correspondence between the directory ID, directory name, parent directory, and node group ID. For example, the ID of directory a is 1, the ID of the parent directory is 0, and the node group ID is D0; the ID of directory b is 2, the ID of the parent directory is 1, and the node group ID is D1; ​​the ID of directory c is 3, the ID of the parent directory is 1, and the node group ID is D1; ​​the ID of directory d is 4, the ID of the parent directory is 2, and the node group ID is D2. In this way, the storage node can obtain the node group ID of the directory through the parent directory of the directory.

[0108] The directory table is associated with the node composition table. The node composition table records the storage nodes that make up the node group. Figure 3 As shown, the node group with ID D0 is composed of storage node 110 and storage node 120, and the node group with ID D1 is composed of storage node 130 and storage node 140. As described above, the storage nodes in the node group of a directory are the storage nodes of the inode distribution table of the directory. In this way, the storage nodes of the inode distribution table of the directory are obtained through the node group ID of the directory. The storage node of the inode distribution table of the directory refers to the storage node storing the inode distribution table of the directory, that is, the storage node where the shards constituting the inode distribution table are stored.

[0109] Generally speaking, an access request for a directory includes the storage path of the directory. The storage path of the directory includes the ID of the directory and the ID of the parent directory. In this way, the node group ID of the directory is obtained through the ID of the directory and the ID of the parent directory in the storage path. Then, the storage node of the shard of the inode distribution table of the directory is obtained through the node group ID and the node group table of the directory. Then, the access request can be forwarded to the storage node of the inode distribution table of the directory. The storage node of the inode distribution table of the directory can respond to the access request and access the corresponding shard.

[0110] In some embodiments, see Figure 4 , it can be set that the client sends an access request for accessing object abc (i.e., the object named "abc"). The storage node that receives the access request can obtain the storage nodes of the inode distribution table of directory a, i.e., storage node 110 and storage node 120, through the directory table and the node group table. The storage node that receives the access request can send the access request to storage node 110 or storage node 120.

[0111] The access request may be set to be sent to the storage node 110. The coordination module in the storage node 110 may respond to the access request and query the metadata of the object abc in the inode distribution table of the directory a. The details are as follows.

[0112] First, the storage node 110 searches the shard where the metadata of the object abc is located in the shards of the inode distribution table of the directory a based on the mapping value corresponding to the object abc.

[0113] The storage node 110 can calculate the mapping value of the object abc. The access request includes the identifier of the object abc, such as the name of the object "abc". The storage node 110 can input the identifier of the object abc into a preset mapping algorithm so that the mapping algorithm outputs the mapping value corresponding to the object abc. In one example, the identifier of the object can be input into the mapping algorithm to obtain the mapping value corresponding to the object. Exemplarily, the mapping algorithm can be a consistent hashing algorithm.

[0114] The storage node 110 can identify the shard where the metadata of object abc is located based on the shard map of directory a and the mapping value corresponding to object abc. Specifically, it is identified to which mapping range in the shard map the mapping value corresponding to object abc belongs, and based on the identified mapping range, the shard associated with the mapping range is obtained, which is the shard where the metadata of object abc is located. Exemplarily, the mapping value corresponding to object abc can be set equal to 18, and the mapping ranges recorded in the shard map of directory a are: [0, 20], [21, 60], [61, 100], etc., wherein the mapping range [0, 20] is associated with shard C1, the mapping range [21, 60] is associated with shard C2, and the mapping range [61, 100] is associated with shard C3. In this way, it can be identified that the mapping range to which the mapping value of object abc belongs is [0, 20], and then the shard where the metadata of object abc is located is identified as shard C1.

[0115] Next, the storage node 110 queries the metadata of the object abc in the shard C1 based on the key of the object abc.

[0116] If shard C1 is located in storage node 110, the coordination module of storage node 110 queries the metadata of object abc in shard C1 through local access. If shard C1 is located in a storage node other than storage node 110, the coordination module of storage node 110 queries the metadata of object abc in shard C1 through remote access.

[0117] The coordination module can use the key of object abc to find the tuple (i.e., the data row recording the metadata of the object) where the metadata of object abc is located in shard C1. Then, the metadata of object abc is obtained from the tuple obtained by the query, thereby querying the metadata of object abc.

[0118] In other words, the coordination module can obtain an access request, which includes an identifier of an object. Based on the identifier of the object, the coordination module obtains a mapping value corresponding to the object. Based on the mapping value, the coordination module uses the shard map to obtain the shard where the metadata of the object is located. Then, based on a key consisting of the mapping value corresponding to the object and the identifier of the object, the metadata of the object is queried in the shard.

[0119] The coordination module can also record the metadata of the object in the inode distribution table. When it is detected that an object is created in the storage system 100, the coordination module can record the metadata of the object in the inode distribution table of the directory where the object is located. Specifically, the coordination module can obtain the identifier of the object, and obtain the mapping value corresponding to the object based on the identifier of the object. For example, the identifier of the object is input into the consistent hash algorithm so that the consistent hash algorithm outputs the mapping value corresponding to the object. Then, the coordination module identifies the mapping range to which the mapping value belongs based on the shard map, and obtains the shard associated with the mapping range. The coordination module records the metadata of the object in the shard associated with the mapping range.

[0120] Among them, the metadata of the object can be recorded in the shard according to the size of the mapping value corresponding to the object, so that the arrangement order of the metadata of the object in the shard is consistent with the size order of the mapping value corresponding to the object. Specifically, whenever the metadata of the object is recorded in the shard, the metadata of the object to be recorded can be recorded in the shard based on the size order of the mapping value corresponding to the object to be recorded and the mapping value corresponding to the object already recorded in the shard. Among them, the object recorded in the shard refers to the object corresponding to the metadata recorded in the shard, that is, if the metadata of the object is recorded in the shard, then the object belongs to the object recorded in the shard.

[0121] The metadata of the object can be recorded in the shards according to the size of the key of the object. As described above, the key of the object is composed of the mapping value corresponding to the object and the identifier of the object, and when the mapping value of the object is greater than the mapping value of other objects, the key of the object is greater than the keys of other objects. Therefore, the metadata of the object can be recorded in the shards according to the size of the key of the object, and the metadata of the object can be recorded in the shards according to the size of the mapping value corresponding to the object.

[0122] In this way, objects can be recorded on the shards, and the order of the metadata of the objects in the shards is guaranteed to be consistent with the order of the size of the mapping values ​​corresponding to the objects. The order of the metadata of the objects refers to the order of the positions of the metadata of the objects, that is, the order of the tuples where the metadata of the objects are located.

[0123] The above describes the storage system 100 and the related functions of the storage node provided in the embodiment of the present application. Next, in combination with the above description, the data processing method provided in the embodiment of the present application is introduced.

[0124] The storage node in the storage system 100 may execute the method. The storage node splits the shard stored by the storage node through the method. Next, in conjunction with the accompanying drawings, the method is introduced by taking the storage node 110 splitting the shard stored by the storage node 110 as an example.

[0125] The storage node 110 stores a shard 200. The shard 200 may be any shard stored in the storage node 110. For the convenience of description, the shard 200 may be set as a shard of the inode shard table of the directory a.

[0126] The shard map of directory a records the mapping range 300 associated with shard 200. That is, shard 200 is used to store the metadata of the object corresponding to the mapping value in the mapping range 300. It can be set that shard 200 records the metadata of multiple objects. The mapping values ​​corresponding to the multiple objects belong to the mapping range 300. The mapping value corresponding to the object can also be called the mapping value of the object, which is the mapping value output by the mapping algorithm after the identifier of the object is input into the mapping algorithm (such as a consistent hashing algorithm). The mapping value corresponding to the object can be used to query the metadata of the object in shard 200. For details, please refer to the above introduction to the functions of the coordination module, which will not be repeated here.

[0127] The arrangement order of the metadata of the multiple objects in the shard 200 is consistent with the size order of the mapping values ​​corresponding to the multiple objects. In other words, the relative position relationship of the metadata of the multiple objects in the shard 200 is consistent with the size order of the mapping values ​​corresponding to the multiple objects.

[0128] Wherein, whenever the metadata of an object is recorded in the shard 200, the metadata of the object to be recorded can be recorded in the shard 200 based on the mapping value corresponding to the object to be recorded and the mapping value order of the objects recorded in the shard 200. In this way, it can be ensured that the arrangement order of the metadata of multiple objects in the shard 200 and the mapping value order corresponding to the multiple objects are consistent. For details, please refer to the above introduction to the function of the coordination module, which will not be repeated here.

[0129] The arrangement order of the metadata of multiple objects in shard 200 is consistent with the size order of the mapping values ​​corresponding to the multiple objects. Therefore, shard 200 can be split with any mapping value in the mapping range 300 as the split point S1, thereby realizing shard splitting without copying metadata or metadata snapshots, and consuming very little processor resources, memory resources, etc.

[0130] See also Figure 5 , the splitting process for the shard 200 may include the following steps. These steps may be executed by the management module in the storage node 110.

[0131] Step 501 , obtaining a mapping value F1 , wherein the mapping value F1 belongs to a mapping range 300 .

[0132] When it is necessary to split the shard 200, for example, when the number of objects recorded in the shard 200 reaches a preset value, or when an access hotspot appears in the storage node 110, step 501 may be executed. The object recorded in the shard 200 refers to the object to which the metadata recorded in the shard 200 belongs, that is, the shard 200 records the metadata of the object, and the object is the object recorded in the shard 200.

[0133] The mapping value F1 may also be referred to as a split-mapped value, which is a mapping value indicating the split point S1 of the shard 200, that is, the mapping value F1 is used as the split point S1. The mapping value F1 belongs to the mapping range 200, that is, the mapping value F1 is an element in the mapping value set of the mapping range 300. The mapping value F1 may be any mapping value in the mapping range 300. In some embodiments, the size of the mapping value F1 is the midpoint of the mapping range 300. That is, the mapping values ​​in the mapping range 300 are sorted in order of size, and in the obtained sorting result, the mapping value F1 is in the middle position. In some embodiments, the mapping value F1 may be selected in the mapping range 300 based on a uniform slicing algorithm.

[0134] Step 502 a , using the mapping value F1 as the split point S1 , splitting the mapping range 300 into the mapping range 310 and the mapping range 320 .

[0135] The mapping values ​​in the mapping range 300 are sorted in order of size, that is, the arrangement order of the mapping values ​​constituting the mapping range 300 is consistent with the size order of the mapping values. The mapping range 300 is split with the mapping value F1 as the split point S1. The mapping values ​​on one side of the mapping value F1 and the mapping value F1 constitute the mapping range 310, and the mapping values ​​on the other side of the mapping value F1 constitute the mapping range 320. Alternatively, the mapping values ​​on one side of the mapping value F1 constitute the mapping range 310, and the mapping values ​​on the other side of the mapping value F1 and the mapping value F1 constitute the mapping range 320. In this way, the mapping range 300 can be split into the mapping range 310 and the mapping range 320. The mapping values ​​on one side of the mapping value F1 refer to the mapping values ​​less than the mapping value F1, and the mapping values ​​on the other side of the mapping value F1 refer to the mapping values ​​greater than the mapping value F1. Alternatively, the mapping values ​​on one side of the mapping value F1 refer to the mapping values ​​greater than the mapping value F1, and the mapping values ​​on the other side of the mapping value F1 refer to the mapping values ​​less than the mapping value F1.

[0136] The mapping range associated with shard 200, i.e., mapping range 300, is recorded in the shard map of directory a. Splitting mapping range 300 in step 502a specifically refers to splitting the mapping range 300 recorded in the shard map of directory a, i.e., splitting the mapping range 300 recorded in the shard map of directory a into mapping range 310 and mapping range 320.

[0137] Step 502b, taking the position of the metadata of the object corresponding to the mapping value F1 in the shard 200 as the split point S2, the shard 200 is split into the shard 210 and the shard 220. The mapping value corresponding to the object in the shard 210 belongs to the mapping range 310, and the mapping value corresponding to the object in the shard 220 belongs to the mapping range 320.

[0138] As described above, the sorting order of the metadata of the objects in the shard 200 is consistent with the size order of the mapping values ​​corresponding to these objects. The position of the metadata of the object corresponding to the mapping value F1 in the shard 200 is taken as the split point S2, and the shard 200 is split. Among them, the metadata of the objects located on one side of the split point S2 and the metadata of the objects corresponding to the mapping value F1 in the area of ​​the shard 200 constitute the shard 210, and the metadata of the objects located on the other side of the split point S2 in the area of ​​the shard 200 constitute the shard 220. Alternatively, the metadata of the objects located on one side of the split point S2 in the area of ​​the shard 200 constitutes the shard 210, and the metadata of the objects located on the other side of the split point S2 and the metadata of the objects corresponding to the mapping value F1 in the area of ​​the shard 200 constitutes the shard 220.

[0139] Among them, shard 200 includes multiple tuples, each tuple records the metadata of an object. The arrangement order of the tuples is consistent with the size order of the mapping values ​​of the objects to which the metadata recorded in the tuples belong. The tuple where the metadata of the object corresponding to the mapping value F1 is located is taken as the split point S2, and the shard 200 is split. Among them, the tuple located on one side of the split point S2 and the tuple where the metadata of the object corresponding to the mapping value F1 is located constitute shard 210, and the tuple located on the other side of the split point S2 constitutes shard 220. Alternatively, the tuple located on one side of the split point S2 constitutes shard 210, and the tuple located on the other side of the split point S2 and the tuple where the metadata of the object corresponding to the mapping value F1 is located constitutes shard 220.

[0140] In some embodiments, the shard 200 may be organized according to a tree index structure. That is, the index structure of the shard 200 may be a tree structure. The index structure of the shard 200 may be a B tree, a B+ tree, or a B-link tree.

[0141] See also Figure 6, the index structure of shard 200 includes multiple leaf nodes. Each leaf node records one or more object metadata in shard 200. Among them, the arrangement order of the multiple leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the multiple leaf nodes. And, in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects. Among them, the object in the leaf node refers to the object to which the metadata recorded in the leaf node belongs. In other words, if the metadata of an object is recorded in a leaf node, the object is the object in the leaf node.

[0142] Among them, in the leaf nodes of the tree index structure, the metadata of the object corresponds to the key of the object one by one. In the index structure of shard 200, the arrangement order of the leaf nodes is consistent with the size order of the keys of the objects in the leaf nodes. Moreover, in the leaf nodes, the position of the metadata of the object is consistent with the size order of the keys of the object. As described above, the prefix of the key of the object is the mapping value of the object, and the size of the key is positively correlated with the size of the mapping value of the object. Therefore, the arrangement order of the leaf nodes is consistent with the size order of the keys of the objects in the leaf nodes, ensuring that the arrangement order of the leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the leaf nodes. The position of the metadata of the object is consistent with the size order of the keys of the object, ensuring that the position of the metadata of the object is consistent with the size order of the keys of the object.

[0143] In one example, if Figure 6 As shown, the mapping range 300 can be set to [M0, M i+n ],M1、…、M i 、M i+1 …M i+n are all mapping values ​​in the mapping range, and M i+n >M i+1 >M i >M1, in which, according to the order from left to right, the leaf node where the metadata of the object whose key prefix is ​​≤M1 is located is located at the prefix of the key ≤M i The left side of the leaf node where the metadata of the object is located, the prefix of the key ≤ M i The leaf nodes where the metadata of the object is located are located at the prefix of the key ≤ M i+1 The left side of the leaf node where the metadata of the object is located, the prefix of the key ≤ M i+1 The leaf nodes where the metadata of the object is located are located at the prefix of the key ≤ M i+n The metadata of the object is located on the left side of the leaf node. In this way, the arrangement order of the leaf nodes is guaranteed to be consistent with the order of the mapping values ​​corresponding to the objects in the leaf nodes.

[0144] In step 502b, the position of the metadata of the object corresponding to the mapping value F1 in the leaf node G1 is taken as the splitting point S3, and the leaf node G1 is split into the leaf node G2 and the leaf node G3. The splitting point S3 is also called the leaf node splitting point, which is used to split the leaf node. The leaf node G1 is the leaf node where the metadata of the object corresponding to the mapping value F1 is located. Figure 6 As shown, the leaf nodes are arranged from left to right in the order of the keys of the objects in the leaf nodes from small to large. Then the leaf node G2 can be called the left leaf node after the split, and the leaf node G3 can be called the right leaf node after the split.

[0145] The storage area of ​​metadata of objects corresponding to mapping values ​​less than mapping value F1 in leaf node G1 and the storage area of ​​metadata of objects corresponding to mapping value F1 constitute leaf node G2, and the storage area of ​​metadata of objects corresponding to mapping values ​​greater than mapping value F1 constitutes leaf node G3. Leaf node G2 and the leaf nodes on the left side of leaf node G2 are used as leaf nodes in the index structure of shard 210, and leaf node G3 and the leaf nodes on the right side of leaf node G3 are used as leaf nodes in the index structure of shard 220.

[0146] In one example, see Figure 7 , you can set the mapping value F1 to M f , where M i +1≤M f <M i+1 That is, the metadata of the object corresponding to the mapping value F1 is recorded in the leaf node G1. f The metadata of the corresponding object (i.e. the prefix is ​​M f The metadata of the key) is located in the leaf node G1, splitting the leaf node G1 into records with prefix ≤ M f The leaf node of the metadata of the key (i.e., leaf node G2) and the record prefix ≤ M i+1 The leaf node of the metadata of the key (ie, leaf node G3).

[0147] In some embodiments, Figure 7 As shown, after the leaf node G1 is split into the leaf node G2 and the leaf node G3, the leaf node G2 inherits the page node identifier of the leaf node G1 (for example, the address of the storage page of the leaf node G1), so that the page node of the upper layer of the leaf node points to the pointer of the leaf node G1, which points to the leaf node G2. A pointer from the leaf node G2 to the leaf node G3 can be created. Among them, the leaf node G3 records the pointer, so that when the corresponding data is not queried in the leaf node G2, it jumps to the leaf node G3 through the pointer to query in the leaf node G3.

[0148] In an example of this embodiment, the leaf node G2 is provided with a being-split-mapped value field, which is used to record the split mapping value of the current split. Figure 7 For example, the field records the M f Since the pointer of the page node above the leaf node points to the leaf node G1, it points to the leaf node G2, and [M f +1,M i+1 The metadata of the object corresponding to the mapping value in ] is recorded in the leaf node G3 or needs to be recorded in the leaf node G3. f When the leaf node G2 records the metadata of the object, it indicates that the metadata of the object is recorded in or needs to be recorded in the leaf node G3. At this time, through the pointer to the leaf node G3 recorded in the leaf node G2, the query or record is jumped to the leaf node G3.

[0149] In an example of this embodiment, the index structure of the shard 200 is specifically a b-link tree, so that the leaf node G2 can have a pointer to the leaf node G3, thereby realizing the splitting of the shard without the need for incremental write-back.

[0150] In some embodiments, Figure 6 As shown, the index structure of shard 200 includes multiple intermediate nodes, such as intermediate node H1 and intermediate node H2. The multiple intermediate nodes are page nodes of the upper layer of the leaf node of the index structure of shard 200. An intermediate node points to one or more leaf nodes through a pointer. The intermediate node records the key range, which includes the key of the object in each leaf node pointed to by the intermediate node. In other words, the intermediate node records the key range to which the key of the object in each leaf node pointed to by the intermediate node belongs. The arrangement order of the multiple intermediate nodes is consistent with the size order of the key ranges recorded by the multiple intermediate nodes. Among them, the size of the key range refers to the size of the key in the key range, and the size of the key range can be represented by the size of any key in the key range.

[0151] Among them, the prefix of the key of the object in the leaf node pointed to by the intermediate node (that is, the mapping value corresponding to the object) constitutes the sub-mapping range, and the intermediate node corresponds to the sub-mapping range. In other words, the intermediate node corresponds to the sub-mapping range, and the sub-mapping range is composed of the mapping values ​​corresponding to the object in the leaf node pointed to by the intermediate node. As mentioned above, the size of the key is positively correlated with the size of the mapping value. Therefore, the arrangement order of the multiple intermediate nodes of the index structure of the shard 200 is consistent with the size order of the sub-mapping ranges corresponding to the multiple intermediate nodes. Among them, the size of the sub-mapping range is the size of the mapping value in the sub-mapping range, and the size of the sub-mapping range can be represented by the size of any mapping value in the sub-mapping range.

[0152] When splitting the index structure of shard 200, the key prefixed with the mapping value F1 can be used as the split point S4 (also called the intermediate node split point, which is a split point used to split the intermediate nodes), and the multiple intermediate nodes can be directly split into intermediate nodes in the index structure of shard 210 and intermediate nodes in the index structure of shard 220, as follows.

[0153] The key with the prefix of mapping value F1 (i.e., the key of the object corresponding to mapping value F1) can be set to be located in the key range recorded by intermediate node H1, that is, the key range recorded by intermediate node H1 includes the key with the prefix of mapping value F1. In step 502b, based on the key of the object corresponding to mapping value F1 and the key range recorded by intermediate node H1, intermediate node H1 can be split into intermediate node H11 and intermediate node H12. Among them, the key range recorded by intermediate node H11 includes the key of the object in leaf node G2, and the key range recorded by intermediate node H12 includes the key in leaf node G3. Specifically, the key with the prefix of mapping value F1 can be set to be specifically located in the key range J1 recorded by intermediate node H1. With the key with the prefix of mapping value F1 as the split point S4, key range J1 is split into key range J11 and key range J12. Among them, key range J11 includes the key of the object in leaf node G2, and key range J12 includes the key of the object in leaf node G3. The key range recorded in the intermediate node H1 is updated, and the updated intermediate node H1 is used as the intermediate node H11, wherein the key range recorded in the intermediate node H11 includes the key range J11 and the key range smaller than the key range J11 in the key range recorded in the intermediate node H1. The intermediate node H12 is created, and the key range J12 and the key range larger than the key range J12 in the key range recorded in the intermediate node H1 are recorded in the intermediate node H12.

[0154] In other words, the mapping value F1 is in the sub-mapping range corresponding to the intermediate node H1. In step 502b, the sub-mapping range corresponding to the intermediate node H1 is split into the sub-mapping range corresponding to the intermediate node H11 and the sub-mapping range corresponding to the intermediate node H12 with the mapping value F1 as the split point S4'.

[0155] Among them, Figure 6 As shown, the leaf nodes can be arranged from left to right in the order of key range from small to large. Then the middle node H11 can be called the left middle node after the split, and the middle node H12 can be called the right middle node after the split.

[0156] The keys with prefixes less than the mapping value F1 and the keys with prefixes of the mapping value F1 in the key range recorded by the intermediate node H1 can be recorded in the intermediate node H11, and the keys with prefixes greater than the mapping value F1 can be recorded in the intermediate node H12. The intermediate node H11 and the intermediate nodes with key ranges less than the key range of the intermediate node H11 are used as intermediate nodes in the index structure of the shard 210. The intermediate node H12 and the intermediate nodes with key ranges greater than the key range recorded by the intermediate node H12 are used as intermediate nodes in the index structure of the shard 220. In other words, the intermediate nodes with corresponding sub-mapping ranges less than the sub-mapping range corresponding to the intermediate node H11 and the intermediate node H11 are used as intermediate nodes of the index structure of the shard 210; the intermediate nodes with corresponding sub-mapping ranges greater than the sub-mapping range corresponding to the intermediate node H12 and the intermediate node H12 are used as intermediate nodes of the index structure of the shard 220.

[0157] More specifically, see Figure 8 , you can set the mapping value F1 to M f , you can set the key range of the intermediate node H1 record to include the prefix M i +1 key and prefix is ​​M i+1 The key of i +1≤M f <M i+1 , that is, the prefix M f The key record of the intermediate node H1 belongs to the key range recorded by the intermediate node H1. In step 502b, the key record with the prefix M f The key of the intermediate node H1 is the split point S4, splitting the intermediate node H1 into intermediate nodes H11 and H12. The prefix of the key range recorded by the intermediate node H1 is less than M f The key and prefix are M f The key is recorded in the intermediate node H11, and the prefix greater than M in the key range recorded in the intermediate node H1 is f The key is recorded in the intermediate node H12.

[0158] After the intermediate node H1 is split into the intermediate node H11 and the intermediate node H12, a pointer can be created from the intermediate node H12 to the lower node. The prefix of the largest key in the key range recorded by the intermediate node H12 is M f +1, you can create a node from the intermediate node H12 to the prefix ≤ M i+1 A pointer to the leaf node (i.e., leaf node G3) of the metadata of the key.

[0159] The intermediate node H11 inherits the page node identifier of the intermediate node H1, so that the pointer pointing to the intermediate node H1 points to the intermediate node H11. A pointer can be created from the intermediate node H11 to the intermediate node H12. Among them, the intermediate node H11 records the pointer. In this way, when the corresponding key is not queried in the intermediate node H11, the pointer can be used to jump to the intermediate node H12 to query in the intermediate node H12. In this way, the intermediate node can be split without the need for incremental writeback.

[0160] When the index structure of shard 200 includes multiple layers of intermediate nodes, the above method can be referred to to split the intermediate nodes of each layer to obtain the intermediate nodes of the index structure of shard 210 and the intermediate nodes of the index structure of shard 220. They will not be described one by one here.

[0161] In some embodiments, the index structure of the shard 200 also includes a root node. The prefix of the key recorded in the root node is taken in the mapping range associated with the data table 200, that is, in the mapping range 300. Therefore, the root node of the index structure of the shard 200 can be said to correspond to the mapping range 300. The root node is the entrance to the index structure of the shard, and operations such as querying can be performed in the shard through the root node. For the convenience of description, the root node of the index structure of the shard can be called the root node of the shard. In step 502b, the key with the prefix of the mapping value F1 is used as the split point S5 (also called the root node split point, which is a split point used to split the root node), and the root node of the shard 200 is split into the root node of the shard 210 and the root node of the shard 220.

[0162] Specifically, Fig. 9 As shown, the mapping value F1 can be set to M f , the mapping range 300 is the mapping range [M0, M i+n ], that is, M f Belongs to the mapping range [M0, M i+n In step 502b, the prefix M f The key is the split point S5, and the root node of shard 200 is split into the root node of shard 210 and the root node of shard 220. f ] as the mapping range 310, and corresponds to the root node of the data table 210. f +1,M i+n ] as the mapping range 320 and corresponds to the root node of the data table 220.

[0163] The root node of shard 210 retains or inherits the pointer of the root node of shard 200 to the lower node. The root node of shard 220 may be created to point to the intermediate nodes of the index structure of shard 220, such as intermediate node H12, intermediate node H2, etc.

[0164] A pointer may be created from the root node of shard 210 to the root node of shard 220. In this way, when the corresponding mapping value is not found in the root node of shard 210, the pointer may be used to jump to the root node of shard 220 to query in the root node of shard 220. In this way, the root node may be split without performing incremental writeback.

[0165] In this way, the splitting of the index structure of shard 200 can be completed.

[0166] In the above, the metadata of the object corresponding to the split mapping value is located at the left leaf node after the split, and the key prefixed with the split mapping value is located at the left middle node and the root node. In other embodiments, the metadata corresponding to the split mapping value is located at the right leaf node after the split, and the key prefixed with the split mapping value is located at the right middle node and the root node. They will not be introduced one by one here.

[0167] Step 503 , associating the mapping range 310 with the shard 210 , and associating the mapping range 320 with the shard 220 .

[0168] In the shard map, the mapping range 310 and the shard 210 can be recorded to be associated, and the mapping range 320 and the shard 220 can be recorded to be associated. Specifically, the mapping range 300 recorded in the shard map is split into the mapping range 310 and the mapping range 320. By associating the mapping range 310 and the shard 210, and the mapping range 320 and the shard 220 recorded in the shard map, the operation for the object can be forwarded to the shard 210 or the shard 220 for processing based on the shard map and the mapping value of the object. In this way, the shard 210 and the shard 220 obtained by splitting the shard 200 can be used as independent shards, that is, independent shards of the inode distribution table of directory a, for storage or access.

[0169] In some embodiments, see Fig. 10A and Fig. 10B , the shard 210 obtained by splitting the shard 200 has an independent index structure, and the shard 220 also has an independent index structure. In this way, there is no conflict between the access request for the shard 210 and the access request for the shard 220, thereby improving the concurrent execution level of the inode distribution table of the directory.

[0170] In some embodiments, as described above, the left page node obtained by splitting has a pointer to the right page node. After step 503, the left page node can point to the pointer of the right page node to save storage space.

[0171] In some embodiments, after step 503 , storage node 110 may send shard 210 or shard 220 to storage node 120 to store and access shard 210 or shard 220 in storage node 120 to eliminate the access hotspot problem of storage node 110 .

[0172] In summary, the method provided by the embodiment of the present application does not need to copy metadata or snapshots of metadata, nor does it need incremental write-back when splitting shards, so that the consumption of processor resources, memory resources, etc. is extremely small. Since there is no need to copy metadata or snapshots of metadata, nor does it need incremental write-back, the splitting speed is fast and the time consumption is extremely short, so that the concurrent access level of the inode distribution table can be quickly improved, and the access hotspot problem can be eliminated. In addition, during the merge period, metadata can be written normally without causing metadata write interruption. In addition, the splitting consumes very little resources and the splitting time consumption is extremely short, which can ensure that the shards are always small (i.e., the metadata of fewer objects is recorded), and it is convenient for the shards to migrate between different storage nodes, so that the storage nodes can quickly perform load balancing and expansion and contraction. In particular, when splitting the index structure of the shard, only the page node where the split point is located is split, which does not affect the operations for other page nodes, so the splitting operation has little impact on the business.

[0173] Next, in a specific example, the method provided by the embodiment of the present application is introduced. Fig.11 , the mapping range 300 can be set to [21, 60]. Slice 200 records metadata of multiple objects in directory a. For example, object E1, object E2, object E3, object E4, object E5, object E6, object E7, object E8, object E9, object E10, etc. The mapping values ​​corresponding to the multiple objects belong to mapping range 200. Among them, the mapping value corresponding to object E1 is 21, the mapping value corresponding to object E2 is 42, the mapping value corresponding to object E3 is 43, the mapping value corresponding to object E4 is 44, the mapping value corresponding to object E5 is 45, the mapping value corresponding to object E6 is 46, the mapping value corresponding to object E7 is 47, the mapping value corresponding to object E8 is 48, the mapping value corresponding to object E9 is 49, and the mapping value corresponding to object E10 is 55.

[0174] See also Fig.12, the intermediate node H1 of the index structure of shard 200 can be set to record multiple key ranges such as the key range with the prefix of the maximum key being 55, the key range with the prefix of the maximum key being 47, and the key range with the prefix of the maximum key being 43. Moreover, in the intermediate node H1, the arrangement order of these ranges is consistent with the size order of the key ranges. The intermediate node H1 points to multiple leaf nodes, among which the leaf nodes recording the metadata of object E2 and the metadata of object E3 correspond to the key range with the prefix of the maximum key being 43, the leaf node G1 recording the metadata of object E4, the metadata of object E5, the metadata of object E6, and the metadata of object E7 corresponds to the key range with the prefix of the maximum key being 47, and the leaf nodes recording the metadata of object E8, the metadata of object E9, and the metadata of object E10 correspond to the key range with the prefix of the maximum key being 55.

[0175] The split mapping value (i.e., mapping value F1) can be set to 45. With the key with prefix 45 as the split point S4, the key range with the prefix of the maximum key being 47 is split into the key range with the prefix of the maximum key being 45 and the key range with the prefix of the maximum key being 47. Among them, the key ranges smaller than the key range with the prefix of the maximum key being 45 and the key range with the prefix of the maximum key being 45 recorded by the intermediate node H1 are recorded in the intermediate node H11. The key ranges greater than the key range with the prefix of the maximum key being 45 recorded by the intermediate node H1 are recorded in the intermediate node H12. Thus, the intermediate node H1 is split into the intermediate node H11 and the intermediate node H12. Among them, the intermediate node H11 has a pointer to the intermediate node H12.

[0176] 45 corresponds to object E5, and the metadata of object E5 is recorded in leaf node G1. In leaf node G1, the order of the metadata of the object is consistent with the order of the size of the mapping value corresponding to the object. According to the position of the metadata of object E5 in leaf node G1, the leaf node is split into leaf node G2 and leaf node G3. Among them, leaf node G2 records the metadata of object E4 and the metadata of object E5, and leaf node G2 records the metadata of object E6 and the metadata of object E7. In addition, leaf node G2 has a pointer to leaf node G3.

[0177] A pointer is created from the intermediate node H11 to the leaf node G2, and a pointer is created from the intermediate node H12 to the leaf node G3.

[0178] After completing step 503, the pointer from intermediate node H11 to intermediate node H12, the pointer from leaf node G2 to leaf node G3, and other pointers from nodes of the index structure of shard 210 to nodes of the index structure of shard 220 can be deleted.

[0179] Thus, the split of shard 200 is completed, and two independent shards 210 and shard 220 are obtained.

[0180] The embodiment of the present application also provides a data processing method, which can quickly merge the fragments of the inode distribution table. The method is still described by taking the storage node 110 executing the method as an example.

[0181] The storage node stores a shard 400. The shard 400 may be a shard of the inode distribution table of the directory a. That is, the shard 200 and the shard 400 are two different shards of the inode distribution table of the directory a. The shard 400 is associated with a mapping range 500. The mapping range 500 is adjacent to the mapping range 300. Among them, the lower endpoint of the mapping range 500 is adjacent to the upper endpoint of the mapping range 300 (that is, the lower endpoint of the mapping range 500 is greater than the upper endpoint of the mapping range 300, and there is no mapping value between the lower endpoint of the mapping range 500 and the upper endpoint of the mapping range 300), or, the upper endpoint of the mapping range 500 is adjacent to the lower endpoint of the mapping range 300 (that is, the lower endpoint of the mapping range 300 is greater than the upper endpoint of the mapping range 500, and there is no mapping value between the lower endpoint of the mapping range 300 and the upper endpoint of the mapping range 500). The lower endpoint of the mapping range refers to the minimum mapping value in the mapping range, and the upper endpoint refers to the maximum mapping value of the mapping range.

[0182] The shard 400 also records metadata of at least one object in the directory a. The mapping value corresponding to the object of the metadata recorded in the shard 400 belongs to the mapping range 500. Exemplarily, the arrangement order of the metadata of the at least one object in the shard 400 is consistent with the size order of the mapping value corresponding to the at least one object.

[0183] In some scenarios, for example, when both shard 400 and shard 200 are relatively small, shard 400 and shard 300 may be merged. A relatively small shard means that the shard records metadata of relatively few objects.

[0184] See also Fig.13 , the splitting process for the shard 200 may include the following steps. These steps may be executed by the management module in the storage node 110.

[0185] Step 1301a: merge mapping range 300 and mapping range 500 into mapping range 700.

[0186] As described above, the mapping range 300 and the mapping range 500 are adjacent, and the mapping range 300 and the mapping range 500 can be directly merged into a large mapping range to obtain the mapping range 700. Specifically, if the lower endpoint of the mapping range 500 is adjacent to the upper endpoint of the mapping range 300, the lower endpoint of the mapping range 300 is the lower endpoint of the mapping range 700, and the upper endpoint of the mapping range 500 is the upper endpoint of the mapping range 700. If the upper endpoint of the mapping range 500 is adjacent to the lower endpoint of the mapping range 300, the lower endpoint of the mapping range 500 is the lower endpoint of the mapping range 700, and the upper endpoint of the mapping range 300 is the upper endpoint of the mapping range 700.

[0187] Step 1301b, merge fragment 200 and fragment 400 into fragment 800.

[0188] Slice 200 may be directly connected to the tail of slice 400 , or slice 400 may be connected to the tail of slice 200 to obtain slice 800 .

[0189] In some embodiments, both shard 200 and shard 400 have multiple tuples, and the tuples in shard 200 can be recorded directly under the tuples in shard 400, or the tuples in shard 400 can be recorded under shard 200 to obtain shard 800.

[0190] In some embodiments, as described above, the order of arrangement of metadata of objects in shard 200 is consistent with the order of size of mapping values ​​corresponding to the objects, and the order of arrangement of metadata of objects in shard 400 is consistent with the order of size of mapping values ​​corresponding to the objects. Based on the size of mapping range 300 and mapping range 500, shard 200 is connected to the end of shard 400, or shard 400 is connected to the end of shard 200, so that the order of arrangement of metadata of objects in shard 800 is consistent with the order of size of mapping values ​​corresponding to the objects.

[0191] In some embodiments, in step 1301b, the index structure of shard 200 and the index structure of shard 400 are merged to obtain the index structure of shard table 800. The index structure of shard 200 and the index structure of shard 400 use the same tree structure, for example, both are B-trees, B+ trees, or B-link trees.

[0192] The tree index structure is a hierarchical, tree-shaped index structure with multiple layers. According to whether the number of layers of the index structure of shard 200 is the same as the number of layers of the index structure of shard 400, the merging of the index structures can be divided into the following two cases.

[0193] In case 1, the number of layers of the index structure of shard 200 is consistent with the number of layers of the index structure of shard 400. The consistency of the number of layers of the index structure of shard 200 and the number of layers of the index structure of shard 400 can also be called: the high consistency of the index structure of shard 200 and the high consistency of the index structure of shard 400.

[0194] In case 1, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 may be directly merged to obtain the root node of the index structure of shard 800. Merging the root node of the index structure of shard 200 and the root node of the index structure of shard 400 may mean: recording the key in the root node of the index structure of shard 200 into the root node of the index structure of shard 400, or recording the key in the root node of the index structure of shard 400 into the root node of the index structure of shard 200.

[0195] In the embodiment of the present application, the prefix of the key recorded by the page node belongs to which mapping range, and the page node can be said to correspond to the mapping range. The root node of the index structure of shard 800 records the key in the root node of the index structure of shard 200 and the key in the root node of the index structure of shard 200. That is, the prefix of the key in the root node of the index structure of shard 800 is taken in the union of mapping range 300 and mapping range 400. Therefore, the root node of the index structure of shard 800 can be said to correspond to mapping range 300 and mapping range 400.

[0196] If the root node of the index structure of shard 200 and the root node of the index structure of shard 400 are merged, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. The root node of the index structure of shard 200 and the root node of the index structure of shard 200 are not merged, but a new root node is created on the root node of the index structure of shard 200 and the root node of the index structure of shard 400. The new root node is used as the root node of the index structure of shard 800. The root node of the index structure of shard 800 corresponds to the mapping range 300 and the mapping range 500, that is, the prefix of the key recorded by the node of the index structure of shard 800 is the element in the union of the mapping range 300 and the mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become the intermediate nodes of shard 800 .

[0197] Case 2: The number of layers of the index structure of shard 200 is greater than the number of layers of the index structure of shard 400 , that is, the height of the index structure of shard 200 is greater than the height of the index structure of shard 400 .

[0198] In case 2, mapping range 500 is mapped to the root node of the index structure of shard 200. Since the root node of the index structure of shard 200 corresponds to mapping range 300 and mapping range 500, the root node is used as the root node of the index structure of shard 800. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 400 becomes the intermediate node of shard 800.

[0199] If the keys in the root node of the index structure of shard 400 are recorded in the root node of the index structure of shard 200, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. Then, a new root node is created on the root node of the index structure of shard 400 and the root node of the index structure of shard 200. The new root node is used as the root node of the index node of shard 800. The root node of the index structure of shard 800 corresponds to the mapping range 300 and the mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become the intermediate nodes of shard 800.

[0200] Case three: the number of layers of the index structure of shard 400 is greater than the number of layers of the index structure of shard 200 , that is, the height of the index structure of shard 400 is greater than the height of the index structure of shard 200 .

[0201] In case three, mapping range 300 is mapped to the root node of the index structure of shard 400, that is, the key recorded in the root node of the index structure of shard 200 is recorded in the root node of the index structure of shard 400. Since the root node of the index structure of shard 400 corresponds to mapping range 300 and mapping range 500, the root node is used as the root node of the index structure of shard 800. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200. At this time, the root node of the index structure of shard 200 becomes the intermediate node of shard 800.

[0202] If the keys in the root node of the index structure of shard 200 are recorded in the root node of the index structure of shard 400, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. Then, a new root node is created on the root node of the index structure of shard 200 and the root node of the index structure of shard 400. The new root node is used as the root node of the index node of shard 800, corresponding to the mapping range 300 and the mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become the intermediate node of shard 800.

[0203] In some embodiments, the index structure of shard 200 and the index structure of shard 400 are both B-link trees, and the index structure of shard 800 is also required to be a B-link tree. Then in an embodiment, a pointer is created between the node of the index structure of shard 200 and the node of the index structure of shard 400. The details are as follows.

[0204] See also Fig.14 , the mapping range 300 can be set smaller than the mapping range 500, and the index structure of shard 200 can be placed on the left of the index structure of shard 400. In the index structure of shard 200 and the index structure of shard 400, the positions of page nodes are sorted from left to right in the order of the corresponding mapping range from small to large. The page nodes here include leaf nodes and intermediate nodes.

[0205] A pointer may be created from the rightmost leaf node in the index structure of shard 200 to the leftmost leaf node in the index structure of shard 400. In this way, the relationship between the leaf nodes of the index structure of shard 800 meets the requirements of the B-link tree. In one example, Fig.14 As shown, the index structure of shard 200 can be set to include leaf nodes G4 and G5, and the index structure of shard 400 can be set to include leaf nodes G6 and G7. Leaf node G5 is the rightmost leaf node in the index structure of shard 200, and leaf node G6 is the leftmost leaf node in the index structure of shard 400. When merging the index structure of shard 200 and the index structure of shard 400, a pointer from leaf node G5 to leaf node G6 can be created.

[0206] From the leaf node upward, the intermediate nodes of the index structure of the shard 200 can be established layer by layer, and the intermediate node pointers pointing to the index structure of the shard 200 are not repeated here.

[0207] If the height of the index structure of shard 200 is inconsistent with the index height of shard 400, the mapping range associated with the index structure with a lower height is mapped to the root node of the index structure with a higher height (that is, the keys in the root node of the index structure with a lower height are recorded in the root nodes of all structures of the higher height), and a pointer is created from the root node of the index structure with a higher height to the root node of the index structure with a lower height, and a pointer is created from the intermediate node of the index structure with a higher height to the root node of the index structure with a lower height.

[0208] In one example, if Fig.14 As shown, the height of the index structure of shard 200 can be set to 3, that is, the index structure has three layers, wherein the index structure includes intermediate node H1 and intermediate node H2, and intermediate node H2 is the rightmost intermediate node in the index structure. The height of the index structure of shard 200 can also be set to 2, that is, the index structure has two layers, wherein the index structure includes root node I2. In this example, a pointer from intermediate node H2 to root node I2 can be created, and after mapping range 500 to root node I1 (i.e., recording the key value recorded by root node I2 into root node 11), a pointer from root node I1 to root node I2 is created. After the merger, root node I2 becomes intermediate node H3 of the index structure of shard 800, and root node I1 becomes root node I3 of the index structure of shard 800.

[0209] If the height of the index structure of shard 200 is consistent with the index height of shard 400, pointers to the intermediate nodes of the index structure of shard 200 can be established layer by layer from the intermediate nodes of the index structure of shard 200, which will not be repeated here.

[0210] The method of merging the root nodes can refer to the above description of situations one, two, and three, and will not be repeated here.

[0211] Continue reading Fig.13 In step 1302, the mapping range 700 is associated with the shard 800. The association between the mapping range 700 and the shard 800 can be recorded in the shard map. Specifically, the mapping range 300 and the mapping range 500 recorded in the shard map can be merged to obtain the mapping range 700. Then, the mapping range 700 in the shard map is associated with the shard 800. At this point, the merging of the shard 200 and the shard 400 is completed.

[0212] The method provided in the embodiment of the present application does not need to copy metadata or snapshots of metadata, nor does it need incremental writeback when merging shards, thereby consuming very little processor resources, memory resources, etc. Since there is no need to copy metadata or snapshots of metadata, nor does it need incremental writeback, the merging speed is fast, the time consumption is extremely short, and the merging of shards can be completed quickly. Moreover, during the merging period, metadata can be written normally without causing metadata write interruption. In particular, when merging the index structure of the shards, it does not affect access to the shards, so that the merging of the shards has no impact on the business.

[0213] See also Fig.15 , an embodiment of the present application provides a data processing device 1500. The device 1500 is used to manage a first shard, the first shard is associated with a first mapping range, the first shard records metadata of multiple objects in the directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects. Fig.15 As shown, the apparatus 1500 comprises:

[0214] An acquisition module 1510 is configured to acquire a first mapping value, where the first mapping value belongs to the first mapping range;

[0215] The splitting module 1520 is configured to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range;

[0216] The association module 1530 is used to associate the second mapping range with the second fragment, and to associate the third mapping range with the third fragment.

[0217] In some embodiments, the index structure of the first shard includes a plurality of leaf nodes, and the leaf nodes record metadata of one or more objects among the plurality of objects; wherein the arrangement order of the plurality of leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects;

[0218] The splitting module is used for 1520: taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, splitting the first leaf node into a second leaf node and a third leaf node; wherein the second leaf node and the fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and the fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value.

[0219] In an example of this embodiment, the index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; the splitting module 1520 is used to: split the first intermediate node into a second intermediate node and a third intermediate node based on the key of the object corresponding to the first mapping value and the key range recorded in the first intermediate node; wherein the key range recorded in the second intermediate node includes the key of the object in the second leaf node, and the key range recorded in the third intermediate node includes the key of the object in the third leaf node.

[0220] In an example of this embodiment, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device 1500 also includes: a query module 1540; the acquisition module 1510 is also used to: obtain an access request, the access request includes a second mapping value, and the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; the query module 1540 is used to: based on the pointer, query the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0221] In some embodiments, a fourth shard is associated with a fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device 1500 also includes a merging module 1550; the merging module 1550 is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first shard and the fourth shard into a fifth shard; the association module 1530 is also used to: associate the fifth mapping range with the fifth shard.

[0222] In an example of this embodiment, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; the merge module 1550 is used to: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merge the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and, correspond the fifth mapping range to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, correspond the fifth mapping range to the first root node to obtain the root node of the index structure of the fifth shard; and, create a pointer from the root node of the index structure of the fifth shard to the second root node.

[0223] In some embodiments, the size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

[0224] The above mainly introduces the device provided by the embodiment of the present application from the perspective of the method flow. It is understandable that, in order to realize the above functions, the device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0225] The embodiment of the present application provides a storage node 1600. The storage node 1600 may include a processor 1610 and a memory 1620. The memory 1620 stores instructions, which can be executed by the processor 1610. When the instructions are executed by the processor 1610, the storage node 1600 may execute Figure 5 or Fig.13 The method shown.

[0226] In some embodiments, storage node 1600 may be implemented as a server or a virtual computing instance (eg, a virtual machine, a container, etc.).

[0227] like Fig.16A As shown, the processor 1610 and the memory 1620 may be deployed in the same storage node 1600 .

[0228] In some embodiments, storage node 1600 may be implemented as a computing device cluster. Fig. 16B As shown, the storage node 1600 may include computing devices 1600A and 1600B. The computing devices 1600A and 1600B are connected via a network. The processor 1610 may be deployed in the computing device 1600A, and the memory 1620 may be deployed in the computing device 1600B.

[0229] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is executed by a processor, the implementation Figure 5 or Fig.13 The method shown.

[0230] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by the processor or a device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes a computer program that instructs the processor to execute Figure 5 or Fig.13 The method shown.

[0231] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: A first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a size order of the mapping values ​​corresponding to the multiple objects; the method comprises: Obtaining a first mapping value, where the first mapping value belongs to the first mapping range; The first mapping range is split into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and the first shard is split into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; The second mapping range is associated with the second shard, and the third mapping range is associated with the third shard.

2. The method according to claim 1, characterized in that The index structure of the first shard includes a plurality of leaf nodes, and the leaf nodes record metadata of one or more objects among the plurality of objects; wherein the arrangement order of the plurality of leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects; The location of the metadata of the object corresponding to the first mapping value in the first shard is a second split point, and the first shard is split into a second shard and a third shard, including: Taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into a second leaf node and a third leaf node; wherein the second leaf node and a fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and a fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; Among them, the mapping value corresponding to the object in the fourth leaf node is smaller than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is larger than the first mapping value.

3. The method according to claim 2, characterized in that The index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; The location of the metadata of the object corresponding to the first mapping value in the first shard is a second split point, and the first shard is split into a second shard and a third shard, including: Based on the key of the object corresponding to the first mapping value and the key range recorded by the first intermediate node, the first intermediate node is split into a second intermediate node and a third intermediate node; wherein the key range recorded by the second intermediate node includes the key of the object in the second leaf node, and the key range recorded by the third intermediate node includes the key of the object in the third leaf node.

4. The method according to claim 2 or 3, characterized in that: The index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the method further includes: Obtaining an access request, the access request including a second mapping value, where the second mapping value belongs to a mapping value corresponding to an object in the first leaf node; Based on the pointer, metadata of the object corresponding to the second mapping value is queried in the second leaf node and the third leaf node.

5. The method according to any one of claims 1 to 4, characterized in that A fourth shard is associated with a fourth mapping range, the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the method further includes: Merging the fourth mapping range and the first mapping range into a fifth mapping range; and merging the first shard and the fourth shard into a fifth shard; The fifth mapping range is associated with the fifth fragment.

6. The method according to claim 5, characterized in that The first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; The merging the first shard and the fourth shard into a fifth shard comprises: When the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merging the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and corresponding the fifth mapping range to the root node of the index structure of the fifth shard; or, When the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, the fifth mapping range is mapped to the first root node to obtain the root node of the index structure of the fifth shard; and a pointer is created from the root node of the index structure of the fifth shard to the second root node.

7. The method according to any one of claims 1 to 6, characterized in that The size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

8. A data processing device, characterized in that: A first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a size order of the mapping values ​​corresponding to the multiple objects; the device comprises: an acquisition module, configured to acquire a first mapping value, where the first mapping value belongs to the first mapping range; a splitting module, configured to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; An associating module is used to associate the second mapping range with the second fragment, and to associate the third mapping range with the third fragment.

9. The device according to claim 8, characterized in that The index structure of the first shard includes a plurality of leaf nodes, and the leaf nodes record metadata of one or more objects among the plurality of objects; wherein the arrangement order of the plurality of leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects; The split module is used to: Taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into a second leaf node and a third leaf node; wherein the second leaf node and a fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and a fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; Among them, the mapping value corresponding to the object in the fourth leaf node is smaller than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is larger than the first mapping value.

10. The device according to claim 9, characterized in that The index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; The splitting module is used to split the first intermediate node into a second intermediate node and a third intermediate node based on the key of the object corresponding to the first mapping value and the key range recorded by the first intermediate node; wherein the key range recorded by the second intermediate node includes the key of the object in the second leaf node, and the key range recorded by the third intermediate node includes the key of the object in the third leaf node.

11. The device according to claim 9 or 10, characterized in that The index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device also includes: a query module; The acquisition module is further used to: acquire an access request, the access request including a second mapping value, the second mapping value being a mapping value corresponding to an object in the first leaf node; The query module is used to query metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node based on the pointer.

12. The device according to any one of claims 8 to 11, characterized in that A fourth shard is associated with a fourth mapping range, the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device further includes a merging module; The merging module is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first slice and the fourth slice into a fifth slice; The associating module is further used to: associate the fifth mapping range with the fifth fragment.

13. The device according to claim 12, characterized in that The first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; The merge module is used to: When the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merging the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and corresponding the fifth mapping range to the root node of the index structure of the fifth shard; or, When the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, the fifth mapping range is mapped to the first root node to obtain the root node of the index structure of the fifth shard; and a pointer is created from the root node of the index structure of the fifth shard to the second root node.

14. The device according to any one of claims 8 to 13, characterized in that The size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

15. A server, characterized in that: The method comprises a processor and a memory; the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images