Data processing method and apparatus and server

Abstract

A data processing method and apparatus and a server, which are applicable to the technical field of data storage. The method comprises: obtaining a first mapping value, wherein the first mapping value is in a first mapping range; splitting the first mapping range into a second mapping range and a third mapping range by using the first mapping value as a first splitting point; splitting a first shard into a second shard and a third shard by using the location of metadata of an object corresponding to the first mapping value in the first shard as a second splitting point; wherein a mapping value corresponding to an object in the second shard is in the second mapping range, and a mapping value corresponding to an object in the third shard is in the third mapping range; and associating the second mapping range with the second shard, and associating the third mapping range with the third shard. The method can reduce resource consumption of shard splitting and improve efficiency of shard splitting.

Description

Data processing method, device and server

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 27, 2023, with application number 202311279760.6 and application name “A Data Processing Method, Device and Server”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of data storage, and in particular to a data processing method, device and server. Background Art

[0003] In the field of data storage technology, in addition to storing the object's data, it is also necessary to store the object's metadata. Object metadata is stored in inodes. The inodes for objects in the same directory are recorded in the same inode distribution table. An object's inode is the inode that stores the object's metadata. Objects in a directory are files and first-level subdirectories within that directory.

[0004] To improve the concurrent access level of the inode distribution table, the inode distribution table is split into multiple shards. As the number of inodes recorded in a shard increases, the shard needs to be split to ensure the concurrent access level of the shard.

[0005] Currently, the shard splitting scheme consumes a lot of resources such as processor resources and memory resources, and is time-consuming.

[0006] Summary of the Invention

[0007] The embodiments of the present application propose a data processing method, device and server, which can reduce the resource consumption of shard splitting and improve the efficiency of shard splitting.

[0008] In a first aspect, a data processing method is provided. The method can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a magnitude order of the mapping values ​​corresponding to the multiple objects.

[0009] The method includes: obtaining a first mapping value, the first mapping value belonging to a first mapping range; splitting the first mapping range into a second mapping range and a third mapping range using the first mapping value as a first splitting point; and splitting the first shard into a second shard and a third shard using the position of metadata of an object corresponding to the first mapping value in a first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; associating the second mapping range with the second shard, and associating the third mapping range with the third shard. The first splitting point is the splitting point that splits the first mapping range into the second mapping range and the third mapping range, and the second splitting point is the splitting point that splits the first shard into the second shard and the third shard.

[0010] This method can be used to manage the inode distribution table and shard map of the directory. The first shard is a shard of the inode distribution table of the directory, and the shard map records the association between the first shard and the first mapping range. After the first shard is split into the second and third shards, and the first mapping range is split into the second and third mapping ranges, the association between the second shard and the second mapping range, and the association between the third shard and the third mapping range can be recorded in the shard map.

[0011] The mapping value corresponding to the object is a value obtained by calculating the identifier of the object according to a mapping algorithm (such as a hash range partitioning algorithm, a key range partitioning algorithm, etc.).

[0012] The shard map records the mapping ranges associated with different shards of the inode distribution table. When accessing an object's metadata, the mapping range is determined based on the object's corresponding mapping value. The mapping range is associated with the shard that records the object's metadata, allowing access to the object's metadata within that shard. Due to the association between mapping ranges and shards, the mapping values ​​of objects in different shards belong to different mapping ranges, which requires that different mapping ranges cannot overlap.

[0013] The method provided in the embodiment of the present application can split the mapping range associated with the shard based on the mapping value as the mapping range splitting point when splitting the shard; and split the shard with the position of the metadata of the object corresponding to the mapping value in the shard as the shard splitting point. Since the arrangement order of the metadata of the objects in the shard is consistent with the size order of the mapping values ​​corresponding to the objects, this can ensure that there is no overlap in the mapping range after the split, and that different shards after the split belong to different mapping ranges. Then, the split shards are respectively associated with the corresponding mapping ranges. In this way, the splitting of the shard can be completed while ensuring that there is no overlap in the mapping range after the split, and that different shards after the split belong to different mapping ranges.

[0014] Because this method doesn't require copying metadata or metadata snapshots, nor does it require incremental writeback, it consumes minimal processor and memory resources. Furthermore, the splitting process is fast and time-consuming, rapidly increasing concurrent metadata access and eliminating hotspots. Furthermore, the minimal resource consumption and time required for splitting ensure that shards remain small, facilitating the migration of shards between different storage nodes and enabling rapid load balancing and scaling between storage nodes.

[0015] In one possible implementation, the index structure of a first shard includes multiple leaf nodes, each of which records metadata of one or more objects among multiple objects; wherein the arrangement order of the multiple leaf nodes is consistent with the order of the mapping values ​​corresponding to the objects in the multiple leaf nodes; within the leaf nodes, the arrangement order of the metadata of the objects is consistent with the order of the mapping values ​​corresponding to the objects; the first shard is split into a second shard and a third shard using the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point, including: splitting the first leaf node into a second leaf node and a third leaf node using the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point; wherein the second leaf node and a fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and a fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value. The third splitting point, also called a leaf node splitting point, is a splitting point used to split the leaf node.

[0016] In this implementation, the arrangement order of the leaf nodes of the shard's index structure is consistent with the size order of the mapping values ​​corresponding to the objects in the leaf nodes, and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects. In this way, when the shard is split based on the first mapping value, the position of the metadata of the object corresponding to the first mapping value in the leaf node is used as the splitting point to split the leaf node into two leaf nodes. Among them, the leaf node on the metadata side of the object corresponding to the first mapping value can be used as the leaf node of the index structure of one shard after the split, and the leaf node on the other side can be used as the leaf node of the index structure of the other shard after the split. In this way, the splitting of the leaf nodes of the shard's index structure can be completed.

[0017] In one possible implementation, the index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; the first shard is split into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point, including: based on the key of the object corresponding to the first mapping value and the key range recorded in the first intermediate node, the first intermediate node is split into a second intermediate node and a third intermediate node; wherein the key range recorded in the second intermediate node includes the key of the object in the second leaf node, and the key range recorded in the third intermediate node includes the key of the object in the third leaf node.

[0018] In this implementation, when splitting the shard based on the first mapping value, based on the key corresponding to the first mapping value and the key range to which the first leaf node belongs, the intermediate node recording the key range to which the first leaf node belongs is split into an intermediate node recording the key range to which the second leaf node belongs and an intermediate node recording the key range to which the third leaf node belongs, thereby completing the splitting of the intermediate node.

[0019] In one possible implementation, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the method also includes: obtaining an access request, the access request includes a second mapping value, the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; based on the pointer, querying the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0020] Among them, you can first query the metadata of the object corresponding to the second mapping value in the second leaf node; if the metadata of the object corresponding to the second mapping value is not found in the second leaf node, based on the pointer, query the metadata of the object corresponding to the second mapping value in the third leaf node.

[0021] In this implementation, there is a pointer between the second leaf node and the third leaf node after the split. In this way, when an access request for the first leaf node is received during the splitting of the first shard, if the metadata of the object to be accessed by the access request cannot be found in the second leaf node, the metadata of the object to be accessed by the access request can be found in the third leaf node through the pointer. In other words, the splitting of the shard does not affect the execution of the access request, so when splitting the shard, there is no need to perform incremental writeback on the shard, nor is there a need to lock the shard. In short, the splitting of the shard will not affect the business or will have little impact on the business.

[0022] In one possible implementation, the fourth shard is associated with a fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the method also includes: merging the fourth mapping range and the first mapping range into a fifth mapping range; and merging the first shard and the fourth shard into a fifth shard; and associating the fifth mapping range with the fifth shard.

[0023] Before merging the first and fourth shards into the fifth shard, the shard map also records the fourth mapping range and the association with the fourth shard. After merging the first and second shards into the fifth shard, the shard map also records the association with the fifth mapping range and the fifth shard. Thus, using the shard map and the fifth mapping range, one can query the shard (i.e., the fifth shard) containing the metadata for objects in the original first and fourth shards.

[0024] In this implementation, the mapping ranges associated with the shards to be merged are adjacent. The shards to be merged can be directly merged, and the mapping ranges associated with the shards to be merged can be merged. In the shard map, the association between the merged shards and the merged mapping ranges can be recorded to complete the merging of the shards.

[0025] This implementation method does not require copying metadata or metadata snapshots, nor does it require incremental write-back, thus consuming very little processor resources, memory resources, etc. In addition, the merging speed is fast and the time consumption is extremely short. The merging of shards can be completed quickly. Moreover, during the merging period, metadata can be written normally without causing metadata writing interruption.

[0026] In one possible implementation, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; merging the first shard and the fourth shard into the fifth shard includes: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merging the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and, corresponding the fifth mapping range to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, corresponding the fifth mapping range to the first root node to obtain the root node of the index structure of the fifth shard; and, creating a pointer from the root node of the index structure of the fifth shard to the second root node.

[0027] In this implementation, if the index structures of the shards to be merged have the same height, the root nodes of the index structures can be directly merged to obtain the root node of the index structure of the merged shards. If the index structures of the shards to be merged have different heights, the mapping range corresponding to the root node of the lower index structure can be mapped to the root node of the higher index structure, and a pointer is constructed from the root node of the higher index structure to the root node of the lower index structure, so that the root node of the lower index structure becomes the intermediate node of the merged index structure.

[0028] In one possible implementation, the size of the object's key is positively correlated with the size of the mapping value corresponding to the object, and the object's key is used to query metadata of the object in the first shard.

[0029] In this way, when storing the metadata of an object in a shard based on the object's key, it can be ensured that the order of the object's metadata in the shard is consistent with the order of the size of the mapping values ​​corresponding to the object.

[0030] In a second aspect, a data processing device is provided, in which a first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the device includes: an acquisition module for acquiring a first mapping value, the first mapping value belonging to the first mapping range; a splitting module for splitting the first mapping range into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and splitting the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping values ​​corresponding to the objects in the second shard belong to the second mapping range, and the mapping values ​​corresponding to the objects in the third shard belong to the third mapping range; an association module for recording the association between the second mapping range and the second shard, and recording the association between the third mapping range and the third shard in the shard map.

[0031] In one possible implementation, the index structure of the first shard includes multiple leaf nodes, which record metadata of one or more objects among multiple objects; wherein the arrangement order of the multiple leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the multiple leaf nodes; in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects; the splitting module is used to: split the first leaf node into a second leaf node and a third leaf node with the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point; wherein the second leaf node and the fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and the fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value.

[0032] In one possible implementation, the index structure of a first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the keys of objects in the first leaf node. The splitting module is configured to split the first intermediate node into a second intermediate node and a third intermediate node based on the keys of the objects corresponding to the first mapping value and the key range recorded in the first intermediate node. The key range recorded in the second intermediate node includes the keys of objects in the second leaf node, and the key range recorded in the third intermediate node includes the keys of objects in the third leaf node.

[0033] In one possible implementation, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device also includes: a query module; the acquisition module is also used to: obtain an access request, the access request includes a second mapping value, and the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; the query module is used to: based on the pointer, query the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0034] In one possible implementation, the fourth shard is associated with the fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein the mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device also includes a merging module; the merging module is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first shard and the fourth shard into a fifth shard; the association module is also used to: associate the fifth mapping range with the fifth shard.

[0035] In one possible implementation, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; the merging module is used to: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merge the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and, correspond the fifth mapping range to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, correspond the fifth mapping range to the first root node to obtain the root node of the index structure of the fifth shard; and, create a pointer from the root node of the index structure of the fifth shard to the second root node.

[0036] In one possible implementation, the size of the object's key is positively correlated with the size of the mapping value corresponding to the object, and the object's key is used to query metadata of the object in the first shard.

[0037] In a third aspect, a storage node is provided, comprising a processor and a memory; the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method provided in the first aspect.

[0038] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0039] In a fifth aspect, a computer program product is provided, comprising instructions for implementing the method provided in the first aspect.

[0040] In a sixth aspect, a server is provided, comprising a processor and a memory; the memory is used to store computer programs; and the processor is used to execute the computer programs to implement the method provided in the first aspect.

[0041] In the seventh aspect, a data processing method is provided, which can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the method includes: taking the first mapping value as the first splitting point, splitting the first mapping range into a second mapping range and a third mapping range; and taking the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point, splitting the first shard into a second shard and a third shard; wherein the first mapping value belongs to the first mapping range, the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; the second mapping range is used to associate with the second shard, and the third mapping range is used to associate with the third shard.

[0042] In an eighth aspect, a data processing device is provided, which can be used to manage a first shard, wherein the first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the arrangement order of the metadata of the multiple objects in the first shard is consistent with the size order of the mapping values ​​corresponding to the multiple objects; the device includes: a splitting module, used to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as the first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as the second splitting point; wherein the first mapping value belongs to the first mapping range, the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; the second mapping range is used to associate with the second shard, and the third mapping range is used to associate with the third shard.

[0043] The beneficial effects of the second to eighth aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1A is a schematic diagram of the structure of a key provided in an embodiment of the present application;

[0045] FIG1B is a schematic diagram of the structure of a storage system provided in an embodiment of the present application;

[0046] FIG2 is a schematic diagram of the structure of a storage node provided in an embodiment of the present application;

[0047] FIG3 is a schematic diagram of an inode distribution table access solution provided in an embodiment of the present application;

[0048] FIG4 is a schematic diagram of an inode distribution table access solution provided in an embodiment of the present application;

[0049] FIG5 is a flow chart of a data processing method provided in an embodiment of the present application;

[0050] FIG6 is a schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0051] FIG7 is a schematic diagram of a leaf node splitting solution provided in an embodiment of the present application;

[0052] FIG8 is a schematic diagram of an intermediate node splitting solution provided in an embodiment of the present application;

[0053] FIG9 is a schematic diagram of a root node splitting solution provided in an embodiment of the present application;

[0054] FIG10A is a schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0055] FIG10B is a schematic diagram of a sharded index structure provided in an embodiment of the present application;

[0056] FIG11 is a schematic diagram of a sharding method provided in an embodiment of the present application;

[0057] FIG12 is a schematic diagram of splitting a sharded index structure provided in an embodiment of the present application;

[0058] FIG13 is a flow chart of a data processing method provided in an embodiment of the present application;

[0059] FIG14 is a schematic diagram of a merging of a sharded index structure provided in an embodiment of the present application;

[0060] FIG15 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0061] FIG16A is a schematic diagram of the structure of a storage node provided in an embodiment of the present application;

[0062] FIG16B is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following describes the solutions provided by the embodiments of the present application in conjunction with the accompanying drawings. In the embodiments of the present application, "plurality" refers to two or more, and "multiple" refers to two or more. Terms such as "first" and "second" are used only to distinguish similar objects and do not necessarily describe a specific order or quantity of objects.

[0064] To facilitate understanding of the solutions provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0065] Object: A general term for files and directories. Objects in a directory refer to the files and first-level subdirectories stored in that directory.

[0066] Object metadata: This is data used to describe the content and properties of an object, such as the object's name, size, modification and access time, and permissions. Object metadata is used to classify, organize, tag, sort, and search objects.

[0067] Inode: A data structure used to record metadata about objects in a distributed file system. The inode that records the metadata of an object is called the inode of that object.

[0068] Inode distribution table: A data table used to record the inodes of objects in a directory. The inode distribution table is a horizontal table, with one row recording the inode of one object.

[0069] Tuple: also known as data row, refers to the row in the inode distribution table. A tuple represents a row in the inode record table, that is, the metadata of an object.

[0070] Shards: Also known as index shards, these are table shards that include multiple tuples in the inode distribution table. Shards are simply partitions of the inode distribution table, which is composed of multiple shards.

[0071] A mapped value is a value obtained by mapping an object's identifier (e.g., its name) to the object's corresponding value using a predefined mapping algorithm (e.g., a hash range partitioning algorithm or a key range partitioning algorithm). The mapped value for an object is also referred to as the object's mapped value. A mapped value has a size, which can be represented by a positive integer.

[0072] Key: An index key, wherein the index structure maintains a mapping relationship from a key to a value, thereby supporting fast value lookup by key. In an embodiment of the present application, the size of an object's key is positively correlated with the size of the mapping value corresponding to the object. The key of an object may be composed of the mapping value corresponding to the object and the identifier of the object. Exemplarily, as shown in FIG1A , the key of an object may be a 2-column key composed of the mapping value corresponding to the object and the identifier of the object. The mapping value of an object may be referred to as a prefix of the key of the object, and the identifier of the object may be referred to as the body of the key of the object. The size of the key is positively correlated with the mapping value included in the key. In other words, when the mapping values ​​included in the key are different, the size of the key is determined by the mapping value. When the mapping values ​​of the key are the same, the size of the key is determined by the size of the identifier of the object. The size of the identifier of an object may be represented by the size order of the lexicographic order corresponding to the identifier.

[0073] Key range: The range of keys of objects in the same data set (for example, the same shard, the same leaf node in the index structure).

[0074] Mapping range: refers to a set of multiple mapping values ​​that are adjacent in sequence. The mapping range can be expressed as a closed interval, and the mapping values ​​that make up the mapping range are the endpoints at both ends of the interval and the integers between the two ends. The upper endpoint of the mapping range is the maximum mapping value in the mapping range. Among them, the inode distribution table and the shards are both associated with the mapping range. Among them, the mapping range corresponding to the inode distribution table is composed of all the allocatable mapping values ​​of the inode distribution table. Among them, the total number of allocatable mapping values, and the minimum mapping value and the maximum mapping value among the allocatable mapping values ​​can be preset. The mapping range corresponding to the shard is a sub-range in the mapping range associated with the inode distribution table, and the mapping ranges associated with different shards do not intersect.

[0075] Mapping algorithm: refers to an algorithm that maps an object's key to a mapped value. Common mapping algorithms include hash range partitioning algorithm and key range partitioning algorithm.

[0076] Consistent hashing algorithm: a hash algorithm for calculating the mapping value of an object, which belongs to the hash range partitioning algorithm. In an embodiment of the present application, the consistent hashing algorithm is an algorithm that performs a hash calculation on the identifier (such as the name) of the object based on the total number of assignable mapping values, wherein the obtained hash value is used as the mapping value of the object. The commonly used consistent hashing algorithm is: taking the remainder based on the total number of assignable mapping values. For example, the name of the object is hashed, the obtained hash value is divided by the total number of assignable mapping values, and the remainder obtained is used as the mapping value of the object.

[0077] Shard map: One inode distribution table corresponds to a shard map, which is used to calculate or query the shard where the metadata of the object in the inode distribution table is located. The shard map records the association between the mapping range and the shard. Specifically, the shard map records the mapping relationship between the mapping range and the shard information (such as the storage address of the shard, identity (ID), etc.). More specifically, the shard map records the start mapping value and the end mapping value of the mapping range, as well as the ID of the shard associated with the mapping value and the storage location of the shard associated with the shard (that is, the identifier of the node where the shard is located). Shards can be named with the shard ID as a suffix to support the positioning of index shards. The shard map is stored and managed on disk as a shard. After the shard map is loaded into memory, it is organized into an ordered array to support fast binary search.

[0078] Directory table: a data table that records metadata used for path resolution (i.e., directory metadata).

[0079] Directory metadata: This includes information such as the directory ID, parent directory ID, directory name, permissions, and the directory's node group ID. The directory table can be replicated to all storage nodes to support low-overhead path resolution. Detailed information about each directory in the directory table (such as size and modification time) is maintained in the inode distribution table of the directory's parent directory. The directory table only maintains information used to resolve directory paths.

[0080] Split point: refers to the split position where the shard or mapping range is split. That is, the shard or mapping range is split at the split point.

[0081] The parent directory of a directory: is the directory where the directory is located, that is, the directory is the first-level subdirectory of its parent directory.

[0082] Node group: Consists of one or more storage nodes. The storage nodes in a directory's node group are the storage nodes that store the directory's inode distribution table, that is, the storage nodes where the shards that make up the inode distribution table are located.

[0083] A distributed file system (DFS) refers to a file system (FS) that manages physical storage resources that are not necessarily directly connected to local storage nodes, but are connected to storage nodes through a computer network; or it is a complete hierarchical file system formed by combining several different logical disk partitions or volumes.

[0084] Tree index structure: It is a tree-shaped index structure, usually with multiple layers, each layer including at least one page node. A page node represents a storage page (page). In the tree index structure, the page nodes in the bottom layer are called leaf nodes, the page nodes in the layer above the leaf nodes can be called intermediate nodes, and the top-level page nodes are called root nodes. Among them, when the tree index structure has only two layers, the root node is also called the intermediate node. Each intermediate node is associated with one or more page nodes in the next layer. Among them, the intermediate node is the parent page node of the page node in the next layer associated with the intermediate node.

[0085] B-link tree (B-link-tree): is a tree-shaped index structure, which is a variant of B+ tree (B+tree). Each page node in the B-link tree stores a pointer to the right sibling node of the page node. Among them, the minimum key recorded in the right sibling node of the page node is adjacent to the maximum key recorded in the page node. In this way, when a page node is full, for example, the storage space is full (8K page is full), a new right sibling node can be split from the page node, and the metadata of the object corresponding to half of the keys recorded in the page node is recorded to the right sibling node, and a pointer to the right sibling node is set. Then, the maximum key of the page node and the pointer to the right sibling node are inserted into the parent page node of the page node. If the parent page node is also full during this process, internal splitting will occur iteratively. The B-link tree can ensure the correctness of other ongoing concurrent operations without locking the entire tree during a split. For example, if the key to be searched is located on the right sibling page node after the split, the parent page node will still be routed to the split page node because the parent page node has not yet inserted a pointer to the right sibling page node. At this time, it will be found that the key to be searched is greater than the maximum key of the page node, and the page node will jump to the right sibling page node through the right pointer of the page node.

[0086] Pointer to page node: refers to the identifier of the page node. As mentioned above, a page node is a storage page, and the pointer is the identifier of the storage page. The identifier of the storage page can also be the address of the storage page.

[0087] The key in the page node refers to the key of the object to which the metadata recorded in the page node belongs.

[0088] Multi-version concurrency control (MVCC): MVCC is a technology commonly used in relational databases to implement transaction isolation in the ACID (atomicity, consistency, isolation, and durability) framework.

[0089] Write-ahead logging (WAL) is a mechanism whereby changes are written to the database log rather than directly to the database file. If a transaction fails, the record in the database log is ignored, undoing the changes. If the transaction succeeds, the record is written back to the database file at a later time, committing the changes. Databases use write-ahead logging to ensure persistence and failure recovery.

[0090] The above descriptions are some of the technical terms that may be involved in the embodiments of the present application, but not all of them. For technical terms that are not explained in the specification and that are involved in the embodiments of the present application, please refer to the description of the prior art.

[0091] As mentioned above, object metadata is used for object classification, organization, tagging, sorting, and searching. Therefore, access to the inode distribution table is frequent and diverse. Some accesses may conflict, making it difficult to access the same shard of the inode distribution table simultaneously, resulting in low concurrent access to the inode distribution table. For example, modifying a shard requires locking the shard, making it difficult to perform other operations on the same shard while modifying it.

[0092] As business progresses, more and more object metadata is recorded in the same shard. This results in more frequent access to that shard and more conflicting accesses. Therefore, the more object metadata is recorded in a shard, the lower the concurrent access level of that shard.

[0093] To improve the concurrent access level of a shard, you can split it into two shards. Compared to the original shard, the split shard records less object metadata, ensuring a higher level of concurrent access.

[0094] Furthermore, if a shard deployed on a storage node records metadata for a large number of objects, the storage node will need to process a large number of access requests, leading to access hotspots. Therefore, it is necessary to split the shards on the storage node and migrate some of the split shards to other storage nodes to eliminate access hotspots.

[0095] As mentioned above, if the mapping ranges associated with the shards do not intersect, then the mapping ranges associated with the shards after the split will also not intersect. The mapping range of the source shard (i.e., the shard before the split) can be set to mapping range A1, and the mapping ranges of the split shards B1 and B2 are mapping range A11 and mapping range A12, respectively. Mapping range A11 and mapping range 12 do not intersect. When splitting, the metadata of the objects corresponding to the mapping values ​​in mapping range A11 in the source shard need to be recorded in shard B1, and the metadata of the objects corresponding to the mapping values ​​in mapping range A12 need to be recorded in shard B2.

[0096] In addition, to save storage resources, when shards are small, they need to be merged. Small shards refer to shards that record metadata for fewer objects.

[0097] In one solution, during a split, shard B1 is created. Then, a snapshot of the metadata for the objects corresponding to the mapping values ​​in mapping range A11 in the source shard is copied to shard B1. After the snapshot copy is complete, an incremental writeback is required. Specifically, during the snapshot copy, metadata for new objects may be written to the source shard. If the mapping values ​​for the new objects fall within mapping range A11, the metadata for the new objects must be copied to shard B1. This process is repeated until minimal metadata is written to the source shard during the snapshot copy, or the number of iterations reaches a preset number. The source shard is then locked, and a final incremental writeback is performed. After the final incremental writeback is complete and the shard map split is complete, the source shard is unlocked, completing the split. The metadata for the objects corresponding to the mapping values ​​in mapping range A11 can be deleted from the source shard, resulting in shard B2. This solution requires both snapshot copying and incremental writeback, which consumes significant processor and memory resources. Furthermore, locking the source shard in this solution can result in lengthy metadata write interruptions.

[0098] This solution merges shards in a similar way to splitting them. Specifically, one shard to be merged is used as the source shard, a snapshot of the source data in the source shard is copied to the other shard to be merged, and after the snapshot is completed, an incremental write-back is performed, etc. Therefore, this solution also consumes a large amount of processor and memory resources and may cause metadata write interruptions.

[0099] An embodiment of the present application provides a data processing method. This method records the metadata of the objects in the directory in the shards of the inode distribution table of the directory according to the size of the mapping value corresponding to the object, so that the order of the metadata of the objects in the shards is consistent with the order of the size of the mapping value corresponding to the object. When splitting a shard, the mapping range associated with the shard can be split into two mapping ranges with the mapping value as the split point S1; and the shard can be split into two shards with the position of the metadata of the object corresponding to the mapping value in the shard as the split point S2; then, in the shard map, the association relationship between the split mapping range and the split shards is recorded. In this way, the splitting of the shard can be completed.

[0100] Split point S1, also known as a mapping range split point, is used to split a mapping range. Specifically, split point S1 can be a mapping value. Split point S2, also known as a shard split point, is used to split a shard. Specifically, split point S2 can be the location of the object's metadata within the shard.

[0101] This solution does not require copying metadata or metadata snapshots, nor does it require incremental writebacks, and consumes very little processor resources, memory resources, etc. Since there is no need to copy metadata or metadata snapshots, nor does it require incremental writebacks, the splitting speed is fast and the time consumption is extremely short, which can quickly increase the level of concurrent access to metadata and eliminate access hotspots and other problems. In addition, the splitting consumes very few resources and takes very little time, which can ensure that the shards are always small (that is, the metadata of fewer objects is recorded), facilitate the migration of shards between different storage nodes, and enable rapid load balancing and scaling between storage nodes.

[0102] Next, the data processing method provided in the embodiment of the present application is introduced.

[0103] The data processing method provided in the embodiments of the present application can be applied to the storage system 100. In some embodiments, the storage system 100 can be a distributed file system, such as a Hadoop distributed file system (HDFS) or a Ceph file system (cephFS). In some embodiments, the storage system 100 can be a distributed cache file system, such as Alluxio. In some embodiments, the storage system 100 can be a stand-alone system, that is, the storage system 100 is a single device, such as a single physical device or a virtual device.

[0104] FIG1B illustrates a possible implementation of a storage system 100. The storage system 100 may include multiple storage nodes, such as storage node 110 and storage node 120. The storage nodes in the storage system 100 may be any device or equipment with data computing, storage, and communication functions, such as a server, a virtual machine (VM), or a container.

[0105] A storage node can act as a data node, managing and storing metadata for objects. As shown in Figure 1B , a storage node can store one or more shards and a shard map. The shards and shard map stored by the storage node correspond to the same directory.

[0106] In some embodiments, referring to Figure 2 , storage node 110 can be configured to store shards C1 and C2. The index structure for shards C1 and C2 can be a B-link tree, meaning that page nodes in the index structure have pointers to their right-hand sibling nodes. This eliminates the need for incremental writeback and shard locks when splitting a shard. This will be discussed in detail below and will not be repeated here.

[0107] The storage node includes a management module. This module is used to manage shards and the shard map, for example, splitting a shard and updating the shard map after a shard split. In some embodiments, the management module can use MVCC and WAL to ensure ACID compliance for transactions within a single node. The functions of the management module will be described in detail below in conjunction with the method embodiments and will not be further elaborated here.

[0108] The storage node may also include a coordination module. Through the coordination module, the storage node can act as a coordinator node to receive access requests sent by the client, and in response to the access request, use the shard map to access the shard to access the metadata of one or more objects in the shard. Specifically, before accessing the data of the object itself, the client needs to access the metadata of the object, such as writing or querying the metadata of the object. Among them, before creating an object, it is necessary to write the metadata of the object in the inode distribution table corresponding to the directory where the object is located (that is, write the metadata of the object to the corresponding shard of the inode distribution table). Before opening an object, it is necessary to query the metadata of the object in the inode distribution table corresponding to the directory where the object is located.

[0109] In some embodiments, as shown in Figure 1B, the storage node stores a directory table, which records the node group ID of the directory. Among them, the fragmentation of the directory refers to the fragmentation of the inode distribution table that constitutes the directory. As shown in Figure 3, the directory table records the correspondence between the directory ID, the directory name, the parent directory, and the node group ID. For example, the ID of directory a is 1, the ID of the parent directory is 0, and the node group ID is D0; the ID of directory b is 2, the ID of the parent directory is 1, and the node group ID is D1; ​​the ID of directory c is 3, the ID of the parent directory is 1, and the node group ID is D1; ​​the ID of directory d is 4, the ID of the parent directory is 2, and the node group ID is D2. In this way, the storage node can obtain the node group ID of the directory through the parent directory of the directory.

[0110] The directory table is associated with a node composition table. The node composition table records the storage nodes that make up the node group. As shown in Figure 3, the node group with ID D0 consists of storage node 110 and storage node 120, and the node group with ID D1 consists of storage node 130 and storage node 140. As described above, the storage nodes in the directory's node group are the storage nodes of the directory's inode distribution table. In this way, the storage nodes of the directory's inode distribution table are obtained through the directory's node group ID. The storage nodes of the directory's inode distribution table refer to the storage nodes that store the directory's inode distribution table, that is, the storage nodes where the shards that make up the inode distribution table are located.

[0111] Generally speaking, an access request for a directory includes the directory's storage path. The directory's storage path includes the directory's ID and parent directory ID. Thus, the directory's node group ID is obtained using the directory's ID and parent directory ID in the storage path. Then, using the directory's node group ID and node group table, the storage nodes for the shards of the directory's inode distribution table are obtained. The access request can then be forwarded to the storage nodes for the directory's inode distribution table. The storage nodes for the directory's inode distribution table can respond to the access request and access the corresponding shards.

[0112] In some embodiments, referring to FIG. 4 , a client may issue an access request for object abc (i.e., an object named "abc"). Upon receiving this access request, the storage node may obtain the storage nodes of the inode distribution table for directory a, i.e., storage node 110 and storage node 120, from the directory table and node group table. Upon receiving this access request, the storage node may send the access request to storage node 110 or storage node 120.

[0113] The access request may be sent to the storage node 110. The coordination module in the storage node 110 may respond to the access request and query the metadata of the object abc in the inode distribution table of the directory a.

[0114] First, the storage node 110 searches the shards of the inode distribution table of the directory a for the shard where the metadata of the object abc is located based on the mapping value corresponding to the object abc.

[0115] The storage node 110 can calculate the mapping value of the object abc. The access request includes the identifier of the object abc, such as the name of the object "abc". The storage node 110 can input the identifier of the object abc into a preset mapping algorithm, so that the mapping algorithm outputs the mapping value corresponding to the object abc. In one example, the identifier of the object can be input into the mapping algorithm to obtain the mapping value corresponding to the object. Exemplarily, the mapping algorithm can be a consistent hashing algorithm.

[0116] The storage node 110 can identify the shard where the metadata of object abc is located based on the shard map of directory a and the mapping value corresponding to object abc. Specifically, it identifies which mapping range in the shard map the mapping value corresponding to object abc belongs to, and based on the identified mapping range, obtains the shard associated with the mapping range, which is the shard where the metadata of object abc is located. Exemplarily, the mapping value corresponding to object abc can be set to 18, and the mapping ranges recorded in the shard map of directory a are: [0, 20], [21, 60], [61, 100], etc., wherein the mapping range [0, 20] is associated with shard C1, the mapping range [21, 60] is associated with shard C2, and the mapping range [61, 100] is associated with shard C3. In this way, it can be identified that the mapping range to which the mapping value of object abc belongs is [0, 20], and then the shard where the metadata of object abc is located is identified as shard C1.

[0117] Next, the storage node 110 queries the metadata of the object abc in the shard C1 based on the key of the object abc.

[0118] If shard C1 is located on storage node 110, the coordination module of storage node 110 queries the metadata of object abc in shard C1 through local access. If shard C1 is located on a storage node other than storage node 110, the coordination module of storage node 110 queries the metadata of object abc in shard C1 through remote access.

[0119] The coordination module can use the key of object abc to find the tuple containing the metadata of object abc in shard C1 (i.e., the data row that records the metadata of the object). Then, the metadata of object abc is obtained from the tuple obtained by the query, thereby querying the metadata of object abc.

[0120] In other words, the coordination module receives an access request that includes an object's identifier. Based on the object's identifier, the coordination module retrieves the corresponding mapping value. Using this mapping value, the coordination module uses the sharding graph to determine the shard where the object's metadata resides. Then, based on a key consisting of the object's mapping value and the object's identifier, the coordination module queries the shard for the object's metadata.

[0121] The coordination module can also record the metadata of the object in the inode distribution table. When it is detected that an object is created in the storage system 100, the coordination module can record the metadata of the object in the inode distribution table of the directory where the object is located. Specifically, the coordination module can obtain the identifier of the object and obtain the mapping value corresponding to the object based on the identifier of the object. For example, the identifier of the object is input into the consistent hash algorithm so that the consistent hash algorithm outputs the mapping value corresponding to the object. Then, the coordination module identifies the mapping range to which the mapping value belongs based on the shard map and obtains the shard associated with the mapping range. The coordination module records the metadata of the object in the shard associated with the mapping range.

[0122] The metadata of the object can be recorded in the shard according to the size of the mapping value corresponding to the object, so that the arrangement order of the metadata of the object in the shard is consistent with the size order of the mapping value corresponding to the object. Specifically, whenever the metadata of the object is recorded in the shard, the metadata of the object to be recorded can be recorded in the shard based on the size order of the mapping value corresponding to the object to be recorded and the mapping value corresponding to the object already recorded in the shard. The object recorded in the shard refers to the object corresponding to the metadata recorded in the shard, that is, if the metadata of the object is recorded in the shard, then the object belongs to the object recorded in the shard.

[0123] The metadata of an object can be recorded in a shard based on the size of the object's key. As described above, an object's key consists of the object's corresponding mapping value and the object's identifier. Furthermore, when an object's mapping value is greater than the mapping values ​​of other objects, the object's key is greater than the keys of other objects. Therefore, recording the metadata of an object in a shard based on the size of its key can achieve the goal of recording the metadata of an object in a shard based on the size of its corresponding mapping value.

[0124] In this way, objects can be recorded on shards, and the order of the object metadata in the shards is guaranteed to be consistent with the order of the corresponding mapping values. The order of the object metadata refers to the order of the location of the object metadata, that is, the order of the tuples containing the object metadata.

[0125] The above describes the storage system 100 and the related functions of the storage node provided in the embodiment of the present application. Next, in combination with the above description, the data processing method provided in the embodiment of the present application is introduced.

[0126] The storage node in the storage system 100 can execute the method. The storage node splits the shard stored by the storage node through the method. Next, with reference to the accompanying drawings, the method is described by taking the storage node 110 splitting the shard stored by the storage node 110 as an example.

[0127] The storage node 110 stores a fragment 200. The fragment 200 may be any fragment stored in the storage node 110. For ease of description, the fragment 200 may be a fragment of the inode fragment table of the directory a.

[0128] The shard map of directory a records the mapping range 300 associated with shard 200. That is, shard 200 is used to store the metadata of the object corresponding to the mapping value in the mapping range 300. Shard 200 can be set to record the metadata of multiple objects. The mapping values ​​corresponding to the multiple objects belong to the mapping range 300. The mapping value corresponding to the object can also be called the mapping value of the object. It is the mapping value output by the mapping algorithm after the object's identifier is input into the mapping algorithm (such as the consistent hashing algorithm). The mapping value corresponding to the object can be used to query the metadata of the object in shard 200. For details, please refer to the above introduction to the functions of the coordination module, which will not be repeated here.

[0129] The order in which the metadata of the multiple objects are arranged in the shard 200 is consistent with the order in which the mapping values ​​corresponding to the multiple objects are arranged. In other words, the relative position relationship of the metadata of the multiple objects in the shard 200 is consistent with the order in which the mapping values ​​corresponding to the multiple objects are arranged.

[0130] Whenever metadata for an object is recorded in shard 200, the metadata for the object to be recorded can be recorded in shard 200 based on the order of the mapping values ​​corresponding to the object to be recorded and the mapping values ​​of the objects already recorded in shard 200. This ensures that the order of the metadata for multiple objects in shard 200 is consistent with the order of the mapping values ​​corresponding to the multiple objects. For details, please refer to the above description of the functions of the coordination module and will not be repeated here.

[0131] The arrangement order of the metadata of multiple objects in shard 200 is consistent with the size order of the mapping values ​​corresponding to the multiple objects. Therefore, any mapping value in the mapping range 300 can be used as the splitting point S1 to split shard 200, thereby realizing shard splitting without copying metadata or metadata snapshots, and consuming very little processor resources, memory resources, etc.

[0132] 5 , the splitting process for the shard 200 may include the following steps, which may be executed by the management module in the storage node 110 .

[0133] Step 501 : Obtain a mapping value F1 , where the mapping value F1 belongs to a mapping range 300 .

[0134] When it is necessary to split shard 200, for example, when the number of objects recorded in shard 200 reaches a preset value, or when an access hotspot appears on storage node 110, step 501 may be executed. The object recorded in shard 200 refers to the object to which the metadata recorded in shard 200 belongs, that is, shard 200 records the metadata of the object, and the object is the object recorded in shard 200.

[0135] Mapping value F1, also known as a split-mapped value, is a mapping value that indicates split point S1 of shard 200. That is, mapping value F1 serves as split point S1. Mapping value F1 belongs to mapping range 200, that is, mapping value F1 is an element in a mapping value set called mapping range 300. Mapping value F1 can be any mapping value within mapping range 300. In some embodiments, the size of mapping value F1 is the midpoint of mapping range 300. That is, the mapping values ​​within mapping range 300 are sorted by size, and in the resulting sorting result, mapping value F1 is positioned in the middle. In some embodiments, mapping value F1 can be selected within mapping range 300 based on a uniform splitting algorithm.

[0136] In step 502 a , the mapping range 300 is split into the mapping range 310 and the mapping range 320 using the mapping value F1 as the splitting point S1 .

[0137] The mapping values ​​in mapping range 300 are sorted by size, that is, the order of the mapping values ​​that make up mapping range 300 is consistent with the order of the mapping values. Mapping range 300 is split using mapping value F1 as split point S1. The mapping values ​​on one side of mapping value F1 and mapping value F1 make up mapping range 310, while the mapping values ​​on the other side of mapping value F1 make up mapping range 320. Alternatively, the mapping values ​​on one side of mapping value F1 make up mapping range 310, while the mapping values ​​on the other side of mapping value F1 and mapping value F1 make up mapping range 320. In this way, mapping range 300 can be split into mapping range 310 and mapping range 320. The mapping values ​​on one side of mapping value F1 are smaller than mapping value F1, while the mapping values ​​on the other side of mapping value F1 are larger than mapping value F1. Alternatively, the mapping values ​​on one side of mapping value F1 are larger than mapping value F1, while the mapping values ​​on the other side of mapping value F1 are smaller than mapping value F1.

[0138] The mapping range associated with shard 200, i.e., mapping range 300, is recorded in the shard map of directory a. Splitting mapping range 300 in step 502a specifically refers to splitting mapping range 300 recorded in the shard map of directory a, i.e., splitting mapping range 300 recorded in the shard map of directory a into mapping range 310 and mapping range 320.

[0139] Step 502b: Split shard 200 into shard 210 and shard 220, using the location of the metadata of the object corresponding to mapping value F1 in shard 200 as split point S2. The mapping values ​​corresponding to the objects in shard 210 belong to mapping range 310, and the mapping values ​​corresponding to the objects in shard 220 belong to mapping range 320.

[0140] As described above, the sorting order of the metadata of the objects in shard 200 is consistent with the order of the size of the mapping values ​​corresponding to these objects. The position of the metadata of the object corresponding to the mapping value F1 in shard 200 is used as the split point S2, and shard 200 is split. The metadata of the objects located on one side of the split point S2 and the metadata of the objects corresponding to the mapping value F1 in the region of shard 200 constitute shard 210, and the metadata of the objects located on the other side of the split point S2 in the region of shard 200 constitute shard 220. Alternatively, the metadata of the objects located on one side of the split point S2 in the region of shard 200 constitute shard 210, and the metadata of the objects located on the other side of the split point S2 and the metadata of the objects corresponding to the mapping value F1 in the region of shard 200 constitute shard 220.

[0141] Among them, shard 200 includes multiple tuples, each tuple records the metadata of an object. The arrangement order of the tuples is consistent with the size order of the mapping values ​​of the objects to which the metadata recorded in the tuples belong. The tuple containing the metadata of the object corresponding to the mapping value F1 is used as the split point S2, and the shard 200 is split. Among them, the tuples located on one side of the split point S2 and the tuple containing the metadata of the object corresponding to the mapping value F1 constitute shard 210, and the tuples located on the other side of the split point S2 constitute shard 220. Alternatively, the tuples located on one side of the split point S2 constitute shard 210, and the tuples located on the other side of the split point S2 and the tuple containing the metadata of the object corresponding to the mapping value F1 constitute shard 220.

[0142] In some embodiments, the shards 200 may be organized in a tree-like index structure. In other words, the index structure of the shards 200 may be a tree-like structure, such as a B-tree, a B+ tree, or a B-link tree.

[0143] Referring to Figure 6, the index structure of shard 200 includes multiple leaf nodes. Each leaf node records one or more object metadata in shard 200. The arrangement order of the multiple leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the multiple leaf nodes. In addition, in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of the mapping values ​​corresponding to the objects. The object in the leaf node refers to the object to which the metadata recorded in the leaf node belongs. In other words, if the metadata of an object is recorded in a leaf node, the object is the object in the leaf node.

[0144] Among them, in the leaf nodes of the tree index structure, the metadata of the object corresponds to the key of the object one-to-one. In the index structure of shard 200, the arrangement order of the leaf nodes is consistent with the size order of the keys of the objects in the leaf nodes. Moreover, in the leaf nodes, the position of the metadata of the object is consistent with the size order of the keys of the object. As mentioned above, the prefix of the key of the object is the mapping value of the object, and the size of the key is positively correlated with the size of the mapping value of the object. Therefore, the arrangement order of the leaf nodes is consistent with the size order of the keys of the objects in the leaf nodes, which ensures that the arrangement order of the leaf nodes is consistent with the size order of the mapping values ​​corresponding to the objects in the leaf nodes. The position of the metadata of the object is consistent with the size order of the keys of the object, which ensures that the position of the metadata of the object is consistent with the size order of the keys of the object.

[0145] In one example, as shown in FIG6 , the mapping range 300 may be set to [M0, M i+n ], M1, …, M i 、M i+1 …M i+n are all mapping values ​​in the mapping range, and M i+n >M i+1 >M i >M1 Among them, according to the order from left to right, the leaf node where the metadata of the object with the prefix of the key ≤ M1 is located is located at the prefix of the key ≤ M i The metadata of the object is located on the left side of the leaf node, the prefix of the key is ≤ M i The leaf node where the metadata of the object is located is located at the prefix of the key ≤ M i+1 The metadata of the object is located on the left side of the leaf node, the prefix of the key is ≤ M i+1 The leaf node where the metadata of the object is located is located at the prefix of the key ≤ M i+n The metadata of the object is located on the left side of the leaf node. In this way, the arrangement order of the leaf nodes is guaranteed to be consistent with the order of the mapping values ​​corresponding to the objects in the leaf nodes.

[0146] In step 502b, the position of the metadata of the object corresponding to the mapping value F1 in the leaf node G1 is used as the splitting point S3, and the leaf node G1 is split into the leaf node G2 and the leaf node G3. The splitting point S3 is also called the leaf node splitting point and is the splitting point used to split the leaf node. The leaf node G1 is the leaf node where the metadata of the object corresponding to the mapping value F1 is located. As shown in Figure 6, the leaf nodes are arranged from left to right in the order of the keys of the objects in the leaf nodes from small to large. Then, the leaf node G2 can be called the left leaf node after the split, and the leaf node G3 can be called the right leaf node after the split.

[0147] The metadata storage area for objects corresponding to mapping values ​​less than mapping value F1 and the metadata storage area for objects corresponding to mapping value F1 in leaf node G1 constitute leaf node G2, and the metadata storage area for objects corresponding to mapping values ​​greater than mapping value F1 constitutes leaf node G3. Leaf node G2 and the leaf nodes to the left of leaf node G2 serve as leaf nodes in the index structure of shard 210, and leaf node G3 and the leaf nodes to the right of leaf node G3 serve as leaf nodes in the index structure of shard 220.

[0148] In one example, referring to FIG. 7 , the mapping value F1 may be set to M f , where M i +1≤M f <M i+1 That is, the metadata of the object corresponding to the mapping value F1 is recorded in the leaf node G1. In step 502b, the metadata of the object corresponding to the mapping value F1 is recorded in the leaf node G1. f The metadata of the corresponding object (i.e. the prefix is ​​M f The metadata of the key) is located in the leaf node G1, splitting the leaf node G1 into record prefix ≤ M f The leaf node of the metadata of the key (i.e. leaf node G2) and the record prefix ≤ M i+1 The leaf node of the metadata of the key (ie, leaf node G3).

[0149] In some embodiments, as shown in FIG7 , after leaf node G1 is split into leaf node G2 and leaf node G3, leaf node G2 inherits the page node identifier of leaf node G1 (e.g., the address of the storage page of leaf node G1), so that the page node of the upper layer of the leaf node points to the pointer of leaf node G1, which in turn points to leaf node G2. A pointer can be created from leaf node G2 to leaf node G3. Leaf node G3 records the pointer, so that when no corresponding data is found in leaf node G2, the pointer is used to jump to leaf node G3 to query in leaf node G3.

[0150] In an example of this embodiment, the leaf node G2 is provided with a being-split-mapped value field, which is used to record the split mapping value of the current split. Taking the case shown in FIG7 as an example, this field records M f Since the pointer of the page node above the leaf node points to the leaf node G1, it points to the leaf node G2, and [M f +1, M i+1 The metadata of the object corresponding to the mapping value in ] is recorded in the leaf node G3 or needs to be recorded in the leaf node G3. When the mapping value of the object to be queried or recorded is greater than M f When , it indicates that the metadata of the object is recorded in or needs to be recorded in leaf node G3. At this time, through the pointer to leaf node G3 recorded by leaf node G2, jump to leaf node G3 for query or record.

[0151] In an example of this embodiment, the index structure of the shard 200 is specifically a b-link tree, so that the leaf node G2 can have a pointer to the leaf node G3, thereby realizing the splitting of the shard without the need for incremental write-back.

[0152] In some embodiments, as shown in FIG6 , the index structure of the shard 200 includes a plurality of intermediate nodes, such as an intermediate node H1 and an intermediate node H2. The plurality of intermediate nodes are page nodes of the upper layer of the leaf node of the index structure of the shard 200. An intermediate node points to one or more leaf nodes through a pointer. The intermediate node records a key range, which includes the key of the object in each leaf node pointed to by the intermediate node. In other words, the intermediate node records the key range to which the key of the object in each leaf node pointed to by the intermediate node belongs. The arrangement order of the plurality of intermediate nodes is consistent with the size order of the key ranges recorded by the plurality of intermediate nodes. Among them, the size of the key range refers to the size of the key in the key range, and the size of the key range can be represented by the size of any key in the key range.

[0153] Among them, the prefix of the key of the object in the leaf node pointed to by the intermediate node (that is, the mapping value corresponding to the object) constitutes the sub-mapping range, and the intermediate node corresponds to the sub-mapping range. In other words, the intermediate node corresponds to the sub-mapping range, and the sub-mapping range is composed of the mapping values ​​corresponding to the object in the leaf node pointed to by the intermediate node. As mentioned above, the size of the key is positively correlated with the size of the mapping value. Therefore, the arrangement order of the multiple intermediate nodes of the index structure of the shard 200 is consistent with the size order of the sub-mapping ranges corresponding to the multiple intermediate nodes. Among them, the size of the sub-mapping range is the size of the mapping value in the sub-mapping range, and the size of the sub-mapping range can be represented by the size of any mapping value in the sub-mapping range.

[0154] When splitting the index structure of shard 200, the key prefixed with the mapping value F1 can be used as the split point S4 (also called the intermediate node split point, which is a split point used to split the intermediate nodes), and the multiple intermediate nodes can be directly split into intermediate nodes in the index structure of shard 210 and intermediate nodes in the index structure of shard 220, as follows.

[0155] The key prefixed with the mapping value F1 (i.e., the key of the object corresponding to the mapping value F1) can be set to be located in the key range recorded by the intermediate node H1, that is, the key range recorded by the intermediate node H1 includes the key prefixed with the mapping value F1. In step 502b, the intermediate node H1 can be split into the intermediate node H11 and the intermediate node H12 based on the key of the object corresponding to the mapping value F1 and the key range recorded by the intermediate node H1. Among them, the key range recorded by the intermediate node H11 includes the key of the object in the leaf node G2, and the key range recorded by the intermediate node H12 includes the key in the leaf node G3. Specifically, the key prefixed with the mapping value F1 can be set to be specifically located in the key range J1 recorded by the intermediate node H1. With the key prefixed with the mapping value F1 as the split point S4, the key range J1 is split into the key range J11 and the key range J12. Among them, the key range J11 includes the key of the object in the leaf node G2, and the key range J12 includes the key of the object in the leaf node G3. The key range recorded in intermediate node H1 is updated, and the updated intermediate node H1 serves as intermediate node H11. The key range recorded in intermediate node H11 includes key range J11 and key ranges smaller than key range J11 in the key range recorded in intermediate node H1. Intermediate node H12 is created, and key range J12 and key ranges larger than key range J12 in the key range recorded in intermediate node H1 are recorded in intermediate node H12.

[0156] In other words, the mapping value F1 is located in the sub-mapping range corresponding to the intermediate node H1. In step 502b, the sub-mapping range corresponding to the intermediate node H1 is split into the sub-mapping range corresponding to the intermediate node H11 and the sub-mapping range corresponding to the intermediate node H12 using the mapping value F1 as the splitting point S4'.

[0157] 6 , the leaf nodes can be arranged from left to right in the order of key range from small to large. Then the middle node H11 can be called the left middle node after the split, and the middle node H12 can be called the right middle node after the split.

[0158] The keys with a prefix less than the mapped value F1 and the keys with a prefix equal to the mapped value F1 in the key range recorded by the intermediate node H1 can be recorded in the intermediate node H11, and the keys with a prefix greater than the mapped value F1 can be recorded in the intermediate node H12. The intermediate node H11 and the intermediate nodes with a key range less than the key range of the intermediate node H11 are used as intermediate nodes in the index structure of the shard 210. The intermediate node H12 and the intermediate nodes with a key range greater than the key range recorded by the intermediate node H12 are used as intermediate nodes in the index structure of the shard 220. In other words, the intermediate nodes whose corresponding sub-mapping ranges are less than the sub-mapping range corresponding to the intermediate node H11 and the intermediate node H11 are used as intermediate nodes in the index structure of the shard 210; the intermediate nodes whose corresponding sub-mapping ranges are greater than the sub-mapping range corresponding to the intermediate node H12 and the intermediate node H12 are used as intermediate nodes in the index structure of the shard 220.

[0159] More specifically, referring to FIG8 , the mapping value F1 can be set to M f , you can set the key range of the intermediate node H1 record to include the prefix M i +1 key and prefix is ​​M i+1 Among them, M i +1≤M f <M i+1 , that is, the prefix M f The key record of the intermediate node H1 belongs to the key range of the intermediate node H1. In step 502b, the key record of the intermediate node H1 is prefixed with M. f The key of the intermediate node H1 is the split point S4, splitting the intermediate node H1 into intermediate nodes H11 and H12. Among them, the prefix of the key range recorded by the intermediate node H1 is less than M f The key and prefix are M f The key is recorded in the intermediate node H11, and the prefix of the key range recorded in the intermediate node H1 is greater than M f The key is recorded in the intermediate node H12.

[0160] Among them, after the intermediate node H1 is split into the intermediate node H11 and the intermediate node H12, a pointer can be created from the intermediate node H12 to the lower node. The prefix of the largest key in the key range recorded by the intermediate node H12 is M f +1, you can create a node from the intermediate node H12 to the prefix ≤ M i+1 A pointer to the leaf node (i.e., leaf node G3) of the metadata of the key.

[0161] Intermediate node H11 inherits the page node identifier of intermediate node H1, so that the pointer to intermediate node H1 points to intermediate node H11. A pointer can be created from intermediate node H11 to intermediate node H12. Intermediate node H11 records this pointer. In this way, when the corresponding key is not found in intermediate node H11, the pointer can be used to jump to intermediate node H12 to query in intermediate node H12. In this way, intermediate nodes can be split without the need for incremental writeback.

[0162] When the index structure of shard 200 includes multiple layers of intermediate nodes, the above method can be referred to, and the intermediate nodes of each layer can be split to obtain the intermediate nodes of the index structure of shard 210 and the intermediate nodes of the index structure of shard 220. Detailed description is omitted here.

[0163] In some embodiments, the index structure of shard 200 also includes a root node. The prefix of the key recorded in the root node is a value taken in the mapping range associated with data table 200, that is, a value taken in the mapping range 300. Therefore, the root node of the index structure of shard 200 can be said to correspond to the mapping range 300. The root node is the entry to the index structure of the shard, and operations such as querying can be performed in the shard through the root node. For convenience of description, the root node of the index structure of the shard can be referred to as the root node of the shard. In step 502b, the key with the prefix of the mapping value F1 is used as the split point S5 (also called the root node split point, which is the split point used to split the root node), and the root node of shard 200 is split into the root node of shard 210 and the root node of shard 220.

[0164] Specifically, as shown in FIG9 , the mapping value F1 can be set to M f , the mapping range 300 is the mapping range [M0, M i+n ], that is, M f Belongs to the mapping range [M0, M i+n ]. In step 502b, the prefix is ​​M f The key is the split point S5, splitting the root node of shard 200 into the root node of shard 210 and the root node of shard 220. f ] as the mapping range 310, and corresponds to the root node of the data table 210. f +1, M i+n ] as the mapping range 320 and corresponds to the root node of the data table 220.

[0165] The root node of shard 210 retains or inherits the pointer to the lower-level node from the root node of shard 200. The root node of shard 220 can be created to point to the intermediate nodes of the index structure of shard 220, such as intermediate node H12, intermediate node H2, etc.

[0166] A pointer can be created from the root node of shard 210 to the root node of shard 220. This way, if the corresponding mapping value is not found in the root node of shard 210, the pointer can be used to jump to the root node of shard 220 for querying. This allows the root node to be split without performing incremental writeback.

[0167] In this way, the splitting of the index structure of shard 200 can be completed.

[0168] In the above description, the metadata of the object corresponding to the split mapping value is located in the left leaf node after the split, and the key prefixed with the split mapping value is located in the left middle node and the root node. In other embodiments, the metadata corresponding to the split mapping value is located in the right leaf node after the split, and the key prefixed with the split mapping value is located in the right middle node and the root node. These will not be described one by one here.

[0169] Step 503 : Associating the mapping range 310 with the shard 210 , and associating the mapping range 320 with the shard 220 .

[0170] In the shard map, the association between mapping range 310 and shard 210 can be recorded, and the association between mapping range 320 and shard 220 can be recorded. Specifically, mapping range 300 recorded in the shard map is split into mapping range 310 and mapping range 320. By associating mapping range 310 with shard 210 and mapping range 320 with shard 220 recorded in the shard map, operations on the object can be forwarded to shard 210 or shard 220 for processing based on the shard map and the mapping value of the object. In this way, shard 210 and shard 220 obtained by splitting shard 200 can be stored or accessed as independent shards, that is, independent shards of the inode distribution table of directory a.

[0171] In some embodiments, referring to Figures 10A and 10B , shard 210, which is obtained by splitting shard 200, has an independent index structure, and shard 220 also has an independent index structure. Thus, access requests to shard 210 and shard 220 do not conflict with each other, thereby improving the concurrent execution level of the directory's inode distribution table.

[0172] In some embodiments, as described above, the left page node obtained by splitting has a pointer to the right page node. After step 503, the pointer of the left page node to the right page node can be used to save storage space.

[0173] In some embodiments, after step 503 , storage node 110 may send shard 210 or shard 220 to storage node 120 to store and access shard 210 or shard 220 in storage node 120 to eliminate the access hotspot problem of storage node 110 .

[0174] In summary, the method provided by the embodiment of the present application does not require copying metadata or a snapshot of metadata, nor does it require incremental write-back when splitting shards, thereby consuming very little processor resources, memory resources, etc. Since there is no need to copy metadata or a snapshot of metadata, nor does it require incremental write-back, the splitting speed is fast and the time consumption is extremely short, thereby quickly improving the concurrent access level of the inode distribution table and eliminating access hot spots and other problems. Moreover, during the merge, metadata can be written normally without causing metadata write interruptions. In addition, the splitting consumes very little resources and takes very little time, which can ensure that the shards are always small (i.e., metadata that records fewer objects), facilitates the migration of shards between different storage nodes, and enables rapid load balancing and expansion and contraction between storage nodes. In particular, when splitting the index structure of the shards, only the page node where the split point is located is split, which does not affect operations on other page nodes. Therefore, the splitting operation has little impact on the business.

[0175] Next, the method provided by the embodiment of the present application is introduced in a specific example. Referring to Figure 11, the mapping range 300 can be set specifically to [21, 60]. Slice 200 records the metadata of multiple objects in directory a. For example, object E1, object E2, object E3, object E4, object E5, object E6, object E7, object E8, object E9, object E10, etc. The mapping values ​​corresponding to these multiple objects belong to mapping range 200. Among them, the mapping value corresponding to object E1 is 21, the mapping value corresponding to object E2 is 42, the mapping value corresponding to object E3 is 43, the mapping value corresponding to object E4 is 44, the mapping value corresponding to object E5 is 45, the mapping value corresponding to object E6 is 46, the mapping value corresponding to object E7 is 47, the mapping value corresponding to object E8 is 48, the mapping value corresponding to object E9 is 49, and the mapping value corresponding to object E10 is 55.

[0176] Referring to Figure 12, the intermediate node H1 of the index structure of shard 200 can be configured to record multiple key ranges, such as a key range with a maximum key prefix of 55, a key range with a maximum key prefix of 47, and a key range with a maximum key prefix of 43. Furthermore, in intermediate node H1, the order in which these ranges are arranged is consistent with the order in which the key ranges are ranked. Intermediate node H1 points to multiple leaf nodes, wherein the leaf nodes recording metadata for objects E2 and E3 correspond to the key range with a maximum key prefix of 43. The leaf node G1 recording metadata for objects E4, E5, E6, and E7 corresponds to the key range with a maximum key prefix of 47. The leaf nodes recording metadata for objects E8, E9, and E10 correspond to the key range with a maximum key prefix of 55.

[0177] The split mapping value (i.e., mapping value F1) can be set to 45. With the key prefixed with 45 as the split point S4, the key range with the prefix of the maximum key being 47 is split into the key range with the prefix of the maximum key being 45 and the key range with the prefix of the maximum key being 47. The key ranges smaller than the key range with the prefix of the maximum key being 45 and the key range with the prefix of the maximum key being 45 recorded by the intermediate node H1 are recorded in the intermediate node H11. The key ranges larger than the key range with the prefix of the maximum key being 45 recorded by the intermediate node H1 are recorded in the intermediate node H12. Thus, the intermediate node H1 is split into the intermediate node H11 and the intermediate node H12. The intermediate node H11 has a pointer to the intermediate node H12.

[0178] 45 corresponds to object E5, and the metadata for object E5 is recorded in leaf node G1. In leaf node G1, the order of the object metadata is consistent with the order of the corresponding mapping values. Based on the position of object E5's metadata in leaf node G1, the leaf node is split into leaf node G2 and leaf node G3. Leaf node G2 records the metadata for objects E4 and E5, and leaf node G2 records the metadata for objects E6 and E7. Leaf node G2 also has a pointer to leaf node G3.

[0179] A pointer is created from the intermediate node H11 to the leaf node G2, and a pointer is created from the intermediate node H12 to the leaf node G3.

[0180] After completing step 503, the pointer from intermediate node H11 to intermediate node H12, the pointer from leaf node G2 to leaf node G3, and other pointers from nodes of the index structure of shard 210 to nodes of the index structure of shard 220 can be deleted.

[0181] Thus, the split of shard 200 is completed, resulting in two independent shards 210 and shard 220.

[0182] The present application also provides a data processing method that can quickly merge shards of an inode distribution table. Taking the storage node 110 as an example, the method will be described.

[0183] The storage node stores shard 400. Shard 400 may be a shard of the inode distribution table of directory a. That is, shard 200 and shard 400 are two different shards of the inode distribution table of directory a. Shard 400 is associated with mapping range 500. Mapping range 500 is adjacent to mapping range 300. The lower endpoint of mapping range 500 is adjacent to the upper endpoint of mapping range 300 (i.e., the lower endpoint of mapping range 500 is greater than the upper endpoint of mapping range 300, and there is no mapping value between the lower endpoint of mapping range 500 and the upper endpoint of mapping range 300), or the upper endpoint of mapping range 500 is adjacent to the lower endpoint of mapping range 300 (i.e., the lower endpoint of mapping range 300 is greater than the upper endpoint of mapping range 500, and there is no mapping value between the lower endpoint of mapping range 300 and the upper endpoint of mapping range 500). The lower endpoint of a mapping range refers to the minimum mapping value in the mapping range, and the upper endpoint refers to the maximum mapping value in the mapping range.

[0184] Shard 400 also records metadata for at least one object in directory a. The mapping value corresponding to the object in the metadata recorded in shard 400 belongs to mapping range 500. Exemplarily, the order in which the metadata for the at least one object is arranged in shard 400 is consistent with the order in which the mapping values ​​corresponding to the at least one object are ranked.

[0185] In some scenarios, for example, when both shard 400 and shard 200 are relatively small, shard 400 and shard 300 may be merged. A relatively small shard means that the shard records metadata of relatively few objects.

[0186] 13 , the splitting process for shard 200 may include the following steps, which may be executed by the management module in storage node 110 .

[0187] Step 1301 a : merge mapping range 300 and mapping range 500 into mapping range 700 .

[0188] As described above, mapping range 300 and mapping range 500 are adjacent, so mapping range 300 and mapping range 500 can be directly merged into a larger mapping range, resulting in mapping range 700. Specifically, if the lower endpoint of mapping range 500 is adjacent to the upper endpoint of mapping range 300, then the lower endpoint of mapping range 300 is used as the lower endpoint of mapping range 700, and the upper endpoint of mapping range 500 is used as the upper endpoint of mapping range 700. If the upper endpoint of mapping range 500 is adjacent to the lower endpoint of mapping range 300, then the lower endpoint of mapping range 500 is used as the lower endpoint of mapping range 700, and the upper endpoint of mapping range 300 is used as the upper endpoint of mapping range 700.

[0189] Step 1301b: merge fragment 200 and fragment 400 into fragment 800.

[0190] Slice 200 can be directly connected to the end of slice 400, or slice 400 can be connected to the end of slice 200 to obtain slice 800.

[0191] In some embodiments, both shard 200 and shard 400 have multiple tuples. The tuples in shard 200 can be recorded directly under the tuples in shard 400, or the tuples in shard 400 can be recorded under shard 200 to obtain shard 800.

[0192] In some embodiments, as described above, the order in which the metadata of objects in shard 200 is arranged is consistent with the order in which the mapped values ​​corresponding to the objects are ordered, and the order in which the metadata of objects in shard 400 is arranged is consistent with the order in which the mapped values ​​corresponding to the objects are ordered. Based on the sizes of mapping range 300 and mapping range 500, shard 200 is connected to the end of shard 400, or shard 400 is connected to the end of shard 200, so that the order in which the metadata of objects in shard 800 is arranged is consistent with the order in which the mapped values ​​corresponding to the objects are ordered.

[0193] In some embodiments, in step 1301b, the index structure of shard 200 and the index structure of shard 400 are merged to obtain the index structure of shard table 800. The index structure of shard 200 and the index structure of shard 400 use the same tree structure, such as a B-tree, a B+ tree, or a B-link tree.

[0194] The tree index structure is a hierarchical, tree-shaped index structure with multiple layers. Depending on whether the number of layers of the index structure of shard 200 is the same as the number of layers of the index structure of shard 400, the merging of the index structures can be divided into the following two cases.

[0195] In case 1, the number of layers of the index structure of shard 200 is consistent with the number of layers of the index structure of shard 400. The consistency of the number of layers of the index structure of shard 200 and the number of layers of the index structure of shard 400 can also be referred to as the high consistency of the index structure of shard 200 and the high consistency of the index structure of shard 400.

[0196] In case 1, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 can be directly merged to obtain the root node of the index structure of shard 800. Merging the root node of the index structure of shard 200 and the root node of the index structure of shard 400 can mean recording the key in the root node of the index structure of shard 200 into the root node of the index structure of shard 400, or recording the key in the root node of the index structure of shard 400 into the root node of the index structure of shard 200.

[0197] In an embodiment of the present application, the prefix of the key recorded by the page node belongs to which mapping range, and the page node can be said to correspond to the mapping range. The root node of the index structure of shard 800 records the key in the root node of the index structure of shard 200 and the key in the root node of the index structure of shard 200. That is, the prefix of the key in the root node of the index structure of shard 800 is taken in the union of mapping range 300 and mapping range 400. Therefore, the root node of the index structure of shard 800 can be said to correspond to mapping range 300 and mapping range 400.

[0198] If the root node of the index structure of shard 200 and the root node of the index structure of shard 400 are merged, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. Therefore, the root node of the index structure of shard 200 and the root node of the index structure of shard 200 are not merged, but a new root node is created above the root node of the index structure of shard 200 and the root node of the index structure of shard 400. This new root node is used as the root node of the index structure of shard 800. The root node of the index structure of shard 800 corresponds to mapping range 300 and mapping range 500, that is, the prefix of the key recorded by the node of the index structure of shard 800 is the element in the union of mapping range 300 and mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become the intermediate nodes of shard 800.

[0199] Case 2: The number of layers of the index structure of shard 200 is greater than the number of layers of the index structure of shard 400 , that is, the height of the index structure of shard 200 is greater than the height of the index structure of shard 400 .

[0200] In the second scenario, mapping range 500 is mapped to the root node of the index structure of shard 200. Since the root node of the index structure of shard 200 corresponds to mapping range 300 and mapping range 500, this root node is used as the root node of the index structure of shard 800. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this point, the root node of the index structure of shard 400 becomes an intermediate node of shard 800.

[0201] If the keys in the root node of the index structure of shard 400 are recorded in the root node of the index structure of shard 200, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. A new root node is then created above the root node of the index structure of shard 400 and the root node of the index structure of shard 200. The new root node is used as the root node of the index node of shard 800. The root node of the index structure of shard 800 corresponds to mapping range 300 and mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this time, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become intermediate nodes of shard 800.

[0202] Case three: the number of layers of the index structure of shard 400 is greater than the number of layers of the index structure of shard 200 , that is, the height of the index structure of shard 400 is greater than the height of the index structure of shard 200 .

[0203] In scenario three, mapping range 300 is mapped to the root node of shard 400's index structure. Specifically, the key recorded in the root node of shard 200's index structure is recorded in the root node of shard 400's index structure. Since the root node of shard 400's index structure corresponds to mapping range 300 and mapping range 500, this root node is used as the root node of shard 800's index structure. A pointer is then created from the root node of shard 800's index structure to the root node of shard 200's index structure. At this point, the root node of shard 200's index structure becomes an intermediate node of shard 800.

[0204] If the keys in the root node of the index structure of shard 200 are recorded in the root node of the index structure of shard 400, the number of keys that the root node needs to record will exceed the maximum number that the root node can accommodate. A new root node is then created above the root node of the index structure of shard 200 and the root node of the index structure of shard 400. The new root node is used as the root node of the index node of shard 800, corresponding to mapping range 300 and mapping range 500. Then, a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 200, and a pointer is created from the root node of the index structure of shard 800 to the root node of the index structure of shard 400. At this point, the root node of the index structure of shard 200 and the root node of the index structure of shard 400 become intermediate nodes of shard 800.

[0205] In some embodiments, the index structures of shard 200 and shard 400 are both B-link trees, and the index structure of shard 800 is also required to be a B-link tree. In this embodiment, pointers are created between the nodes of the index structure of shard 200 and the nodes of the index structure of shard 400. The details are as follows.

[0206] Referring to Figure 14 , mapping range 300 can be set smaller than mapping range 500, and the index structure of shard 200 can be placed to the left of the index structure of shard 400. In the index structures of shards 200 and 400, the positions of page nodes are sorted from left to right in ascending order of the corresponding mapping ranges. Page nodes here include leaf nodes and intermediate nodes.

[0207] A pointer can be created from the rightmost leaf node in the index structure of shard 200 to the leftmost leaf node in the index structure of shard 400. In this way, the relationship between the leaf nodes of the index structure of shard 800 meets the requirements of the B-link tree. In one example, as shown in Figure 14, the index structure of shard 200 can be set to include leaf node G4 and leaf node G5, and the index structure of shard 400 can be set to include leaf node G6 and leaf node G7. Among them, leaf node G5 is the rightmost leaf node in the index structure of shard 200, and leaf node G6 is the leftmost leaf node in the index structure of shard 400. When the index structure of shard 200 and the index structure of shard 400 are merged, a pointer can be created from leaf node G5 to leaf node G6.

[0208] From the leaf node upwards, the intermediate nodes of the index structure of the shard 200 can be established layer by layer, and the intermediate node pointers pointing to the index structure of the shard 200 are not described here one by one.

[0209] If the height of the index structure of shard 200 is inconsistent with the index height of shard 400, the mapping range associated with the index structure with lower height will be mapped to the root node of the index structure with higher height (that is, the key in the root node of the index structure with lower height will be recorded in the root nodes of all structures of the higher height), and a pointer will be created from the root node of the index structure with higher height to the root node of the index structure with lower height, as well as a pointer will be created from the intermediate node of the index structure with higher height to the root node of the index structure with lower height.

[0210] In one example, as shown in Figure 14, the height of the index structure of shard 200 can be set to 3, that is, the index structure has three layers, wherein the index structure includes intermediate node H1 and intermediate node H2, and intermediate node H2 is the rightmost intermediate node in the index structure. The height of the index structure of shard 200 can also be set to 2, that is, the index structure has two layers, wherein the index structure includes root node I2. In this example, a pointer from intermediate node H2 to root node I2 can be created, and after mapping range 500 to root node I1 (i.e., recording the key value recorded by root node I2 into root node 11), a pointer from root node I1 to root node I2 is created. After the merger, root node I2 becomes intermediate node H3 of the index structure of shard 800, and root node I1 becomes root node I3 of the index structure of shard 800.

[0211] If the height of the index structure of shard 200 is consistent with the index height of shard 400, pointers to the intermediate nodes of the index structure of shard 200 can be established layer by layer from the intermediate nodes of the index structure of shard 200, which will not be repeated here.

[0212] The method of merging the root nodes can refer to the above description of situations one, two, and three, and will not be repeated here.

[0213] Continuing with Figure 13 , in step 1302 , mapping range 700 is associated with shard 800 . The association between mapping range 700 and shard 800 can be recorded in the shard map. Specifically, mapping range 300 and mapping range 500, as recorded in the shard map, can be merged to obtain mapping range 700. Then, mapping range 700 in the shard map is associated with shard 800. This completes the merge of shards 200 and 400.

[0214] The method provided in the embodiment of the present application does not require copying metadata or metadata snapshots, nor does it require incremental writeback when merging shards, thereby consuming very little processor resources, memory resources, etc. Since there is no need to copy metadata or metadata snapshots, nor does it require incremental writeback, the merging speed is fast and the time consumption is extremely short, and the merging of shards can be completed quickly. Furthermore, during the merge, metadata can be written normally without causing metadata write interruptions. In particular, when merging the index structure of the shards, access to the shards is not affected, so that the merging of the shards has no impact on the business.

[0215] Referring to FIG. 15 , an embodiment of the present application provides a data processing device 1500 . Device 1500 is configured to manage a first shard, which is associated with a first mapping range. The first shard records metadata for multiple objects in the directory; wherein the mapping values ​​corresponding to the multiple objects belong to the first mapping range, and the order in which the metadata for the multiple objects are arranged in the first shard is consistent with the order in which the mapping values ​​corresponding to the multiple objects are arranged. As shown in FIG. 15 , device 1500 includes:

[0216] An acquiring module 1510 is configured to acquire a first mapping value, where the first mapping value belongs to the first mapping range;

[0217] Splitting module 1520 is configured to split the first mapping range into a second mapping range and a third mapping range using the first mapping value as a first splitting point; and split the first shard into a second shard and a third shard using the location of metadata of objects corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping values ​​corresponding to the objects in the second shard belong to the second mapping range, and the mapping values ​​corresponding to the objects in the third shard belong to the third mapping range;

[0218] The associating module 1530 is configured to associate the second mapping range with the second fragment, and associate the third mapping range with the third fragment.

[0219] In some embodiments, the index structure of the first shard includes a plurality of leaf nodes, wherein the leaf nodes record metadata of one or more objects among the plurality of objects; wherein an arrangement order of the plurality of leaf nodes is consistent with a size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and within the leaf nodes, an arrangement order of the metadata of the objects is consistent with a size order of mapping values ​​corresponding to the objects;

[0220] The splitting module is used for 1520: taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into a second leaf node and a third leaf node; wherein the second leaf node and the fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and the fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; wherein the mapping value corresponding to the object in the fourth leaf node is less than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is greater than the first mapping value.

[0221] In an example of this embodiment, the index structure of the first shard includes a first intermediate node, and the key range recorded by the first intermediate node includes the key of the object in the first leaf node; the splitting module 1520 is used to: split the first intermediate node into a second intermediate node and a third intermediate node based on the key of the object corresponding to the first mapping value and the key range recorded by the first intermediate node; wherein the key range recorded by the second intermediate node includes the key of the object in the second leaf node, and the key range recorded by the third intermediate node includes the key of the object in the third leaf node.

[0222] In an example of this embodiment, the index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device 1500 also includes: a query module 1540; the acquisition module 1510 is also used to: obtain an access request, the access request includes a second mapping value, and the second mapping value belongs to the mapping value corresponding to the object in the first leaf node; the query module 1540 is used to: based on the pointer, query the metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node.

[0223] In some embodiments, a fourth shard is associated with a fourth mapping range, and the fourth shard records metadata of at least one object in the directory, wherein the mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device 1500 also includes a merging module 1550; the merging module 1550 is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first shard and the fourth shard into a fifth shard; the association module 1530 is also used to: associate the fifth mapping range with the fifth shard.

[0224] In an example of this embodiment, the first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; the merging module 1550 is used to: when the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merge the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and, correspond the fifth mapping range to the root node of the index structure of the fifth shard; or, when the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, correspond the fifth mapping range to the first root node to obtain the root node of the index structure of the fifth shard; and, create a pointer from the root node of the index structure of the fifth shard to the second root node.

[0225] In some embodiments, the size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

[0226] The above mainly introduces the device provided by the embodiment of the present application from the perspective of the method flow. It is understandable that, in order to realize the above functions, the device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0227] An embodiment of the present application provides a storage node 1600. Storage node 1600 may include a processor 1610 and a memory 1620. Memory 1620 stores instructions that are executable by processor 1610. When processor 1610 executes the instructions, storage node 1600 may perform the method shown in FIG5 or FIG13.

[0228] In some embodiments, the storage node 1600 may be implemented as a server or a virtual computing instance (eg, a virtual machine, a container, etc.).

[0229] As shown in FIG. 16A , the processor 1610 and the memory 1620 may be deployed in the same storage node 1600 .

[0230] In some embodiments, storage node 1600 may be implemented as a computing device cluster. For example, as shown in FIG16B , storage node 1600 may include computing devices 1600A and 1600B. Computing devices 1600A and 1600B are connected via a network. Processor 1610 may be deployed in computing device 1600A, and memory 1620 may be deployed in computing device 1600B.

[0231] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is executed by a processor, the method shown in FIG. 5 or FIG. 13 is implemented.

[0232] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the processor or a device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes a computer program that instructs the processor to execute the method shown in Figure 5 or Figure 13.

[0233] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: A first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a size order of the mapping values ​​corresponding to the multiple objects; the method comprises: Obtaining a first mapping value, where the first mapping value belongs to the first mapping range; The first mapping range is split into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and the first shard is split into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; The second mapping range is associated with the second shard, and the third mapping range is associated with the third shard.

2. The method according to claim 1, characterized in that The index structure of the first shard includes a plurality of leaf nodes, and the leaf nodes record metadata of one or more objects among the plurality of objects; wherein the arrangement order of the plurality of leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects; The location of the metadata of the object corresponding to the first mapping value in the first shard is a second split point, and the first shard is split into a second shard and a third shard, including: Taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into a second leaf node and a third leaf node; wherein the second leaf node and a fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and a fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; Among them, the mapping value corresponding to the object in the fourth leaf node is smaller than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is larger than the first mapping value.

3. The method according to claim 2, characterized in that The index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; The location of the metadata of the object corresponding to the first mapping value in the first shard is a second split point, and the first shard is split into a second shard and a third shard, including: Based on the key of the object corresponding to the first mapping value and the key range recorded by the first intermediate node, the first intermediate node is split into a second intermediate node and a third intermediate node; wherein the key range recorded by the second intermediate node includes the key of the object in the second leaf node, and the key range recorded by the third intermediate node includes the key of the object in the third leaf node.

4. The method according to claim 2 or 3, characterized in that: The index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the method further includes: Obtaining an access request, the access request including a second mapping value, where the second mapping value belongs to a mapping value corresponding to an object in the first leaf node; Based on the pointer, metadata of the object corresponding to the second mapping value is queried in the second leaf node and the third leaf node.

5. The method according to any one of claims 1 to 4, characterized in that A fourth shard is associated with a fourth mapping range, the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the method further includes: Merging the fourth mapping range and the first mapping range into a fifth mapping range; and merging the first shard and the fourth shard into a fifth shard; The fifth mapping range is associated with the fifth fragment.

6. The method according to claim 5, characterized in that The first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; The merging the first shard and the fourth shard into a fifth shard comprises: When the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merging the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and corresponding the fifth mapping range to the root node of the index structure of the fifth shard; or, When the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, the fifth mapping range is mapped to Go to the first root node to obtain the root node of the index structure of the fifth shard; and create a pointer from the root node of the index structure of the fifth shard to the second root node.

7. The method according to any one of claims 1 to 6, characterized in that The size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

8. A data processing device, characterized in that: A first shard is associated with a first mapping range, and the first shard records metadata of multiple objects in a directory; wherein mapping values ​​corresponding to the multiple objects belong to the first mapping range, and an arrangement order of the metadata of the multiple objects in the first shard is consistent with a size order of the mapping values ​​corresponding to the multiple objects; the device comprises: an acquisition module, configured to acquire a first mapping value, where the first mapping value belongs to the first mapping range; a splitting module, configured to split the first mapping range into a second mapping range and a third mapping range with the first mapping value as a first splitting point; and split the first shard into a second shard and a third shard with the position of the metadata of the object corresponding to the first mapping value in the first shard as a second splitting point; wherein the mapping value corresponding to the object in the second shard belongs to the second mapping range, and the mapping value corresponding to the object in the third shard belongs to the third mapping range; An associating module is used to associate the second mapping range with the second fragment, and to associate the third mapping range with the third fragment.

9. The device according to claim 8, characterized in that The index structure of the first shard includes a plurality of leaf nodes, and the leaf nodes record metadata of one or more objects among the plurality of objects; wherein the arrangement order of the plurality of leaf nodes is consistent with the size order of mapping values ​​corresponding to the objects in the plurality of leaf nodes; and in the leaf nodes, the arrangement order of the metadata of the objects is consistent with the size order of mapping values ​​corresponding to the objects; The split module is used to: Taking the position of the metadata of the object corresponding to the first mapping value in the first leaf node as the third splitting point, the first leaf node is split into a second leaf node and a third leaf node; wherein the second leaf node and a fourth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the second shard, and the third leaf node and a fifth leaf node among the multiple leaf nodes are used as leaf nodes in the index structure of the third shard; Among them, the mapping value corresponding to the object in the fourth leaf node is smaller than the first mapping value, and the mapping value corresponding to the object in the fifth leaf node is larger than the first mapping value.

10. The device according to claim 9, characterized in that The index structure of the first shard includes a first intermediate node, and the key range recorded in the first intermediate node includes the key of the object in the first leaf node; The splitting module is used to split the first intermediate node into a second intermediate node and a third intermediate node based on the key of the object corresponding to the first mapping value and the key range recorded by the first intermediate node; wherein the key range recorded by the second intermediate node includes the key of the object in the second leaf node, and the key range recorded by the third intermediate node includes the key of the object in the third leaf node.

11. The device according to claim 9 or 10, characterized in that The index structure of the second shard includes a pointer from the second leaf node to the third leaf node; the device also includes: a query module; The acquisition module is further used to: acquire an access request, the access request including a second mapping value, the second mapping value being a mapping value corresponding to an object in the first leaf node; The query module is used to query metadata of the object corresponding to the second mapping value in the second leaf node and the third leaf node based on the pointer.

12. The device according to any one of claims 8 to 11, characterized in that A fourth shard is associated with a fourth mapping range, the fourth shard records metadata of at least one object in the directory, wherein a mapping value of the at least one object belongs to the fourth mapping range, and the fourth mapping range is adjacent to the first mapping range; the device further includes a merging module; The merging module is used to: merge the fourth mapping range and the first mapping range into a fifth mapping range; and merge the first slice and the fourth slice into a fifth slice; The associating module is further used to: associate the fifth mapping range with the fifth fragment.

13. The device according to claim 12, characterized in that The first root node of the index structure of the first shard corresponds to the first mapping range, and the second root node of the index structure of the fourth shard corresponds to the fourth mapping range; The merge module is used to: When the height of the index structure of the first shard is consistent with the height of the index structure of the fourth shard, merging the first root node and the second root node to obtain the root node of the index structure of the fifth shard; and corresponding the fifth mapping range to the root node of the index structure of the fifth shard; or, When the height of the index structure of the first shard is greater than the height of the index structure of the fourth shard, the fifth mapping range is mapped to the first root node to obtain the root node of the index structure of the fifth shard; and a pointer is created from the root node of the index structure of the fifth shard to the second root node.

14. The device according to any one of claims 8 to 13, characterized in that The size of the key of the object is positively correlated with the size of the mapping value corresponding to the object, and the key of the object is used to query the metadata of the object in the first shard.

15. A server, characterized in that: The method comprises a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images