Metadata processing method and device for distributed file system

Abstract

The invention provides a metadata processing method and device for a distributed file system, relates to the technical field of artificial intelligence, in particular to the technical field of cloud computing, cloud storage and cloud databases, and can be applied to an intelligent cloud scene. One specific embodiment of the method comprises the following steps: in response to the situation that the data volume of metadata of the distributed file system is smaller than a preset threshold value, storing a metadata storage layer table and a path analysis acceleration layer table of the distributed file system on the same original fragment; in response to the situation that the data size of metadata of the distributed file system is not smaller than a preset threshold value, the original fragment is split into a metadata storage layer fragment and a path analysis acceleration layer fragment, the metadata storage layer fragment is used for storing a metadata storage layer table, and the path analysis acceleration layer fragment is used for storing a path analysis acceleration layer table; and scheduling the metadata storage layer tables on the metadata storage layer fragments to different data fragments.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of cloud computing, cloud storage and cloud database technology, and can be applied in intelligent cloud scenarios. Background Art

[0002] The data in a distributed file system can include file data, which can include file content. Furthermore, the data in a distributed file system can also include metadata. Metadata can include all data in the distributed file system's namespace, including directories, directory attributes, and file attributes.

[0003] In actual use, metadata in distributed file systems needs to be processed. However, existing distributed file systems either have excellent performance for small data volumes but insufficient scalability, or have strong scalability for large data volumes but poor performance for small data volumes. Summary of the Invention

[0004] The embodiments of the present disclosure provide a metadata processing method, apparatus, device, storage medium, and program product for a distributed file system.

[0005] In a first aspect, an embodiment of the present disclosure proposes a metadata processing method for a distributed file system, comprising: in response to the data volume of the metadata of the distributed file system being less than a preset threshold, storing the metadata storage layer table and the path resolution acceleration layer table of the distributed file system on the same original shard, wherein the metadata storage layer table is used to store the metadata of the distributed file system, and the path resolution acceleration layer table is used to store the path of the metadata of the distributed file system; in response to the data volume of the metadata of the distributed file system being not less than a preset threshold, splitting the original shard into metadata storage layer shards and path resolution acceleration layer shards, wherein the metadata storage layer shards store the metadata storage layer table, and the path resolution acceleration layer shards store the path resolution acceleration layer table; and scheduling the metadata storage layer table on the metadata storage layer shards to different data shards.

[0006] In a second aspect, an embodiment of the present disclosure proposes a metadata processing device for a distributed file system, comprising: a storage module, configured to store the metadata storage layer table and the path resolution acceleration layer table of the distributed file system on the same original shard in response to the data volume of the metadata of the distributed file system being less than a preset threshold, wherein the metadata storage layer table is used to store the metadata of the distributed file system, and the path resolution acceleration layer table is used to store the path of the metadata of the distributed file system; a splitting module, configured to split the original shard into metadata storage layer shards and path resolution acceleration layer shards in response to the data volume of the metadata of the distributed file system being not less than a preset threshold, wherein the metadata storage layer shards store the metadata storage layer table, and the path resolution acceleration layer shards store the path resolution acceleration layer table; a scheduling module, configured to schedule the metadata storage layer table on the metadata storage layer shards to different data shards.

[0007] In a third aspect, an embodiment of the present disclosure proposes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0010] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects, and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. Among them:

[0012] Figure 1 is a flowchart of an embodiment of a metadata processing method for a distributed file system according to the present disclosure;

[0013] Figure 2 It is a schematic diagram of a single-point metadata architecture;

[0014] Figure 3 It is a schematic diagram of the subtree partitioning metadata architecture;

[0015] Figure 4 It is a schematic diagram of the metadata architecture that integrates directory partitioning and subtree partitioning;

[0016] Figure 5 is a flowchart of another embodiment of a metadata processing method for a distributed file system according to the present disclosure;

[0017] Figure 6 This is a schematic diagram of the original table;

[0018] Figure 7 This is a schematic diagram of the splitting of the original table;

[0019] Figure 8 is a structural diagram of an embodiment of a metadata processing device for a distributed file system according to the present disclosure;

[0020] Figure 9 It is a block diagram of an electronic device used to implement the metadata processing method of the distributed file system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] Figure 1 A process 100 of an embodiment of a metadata processing method for a distributed file system according to the present disclosure is shown. The metadata processing method for a distributed file system includes the following steps:

[0024] Step 101: In response to the data volume of the metadata of the distributed file system being less than a preset threshold, the metadata storage layer table and the path resolution acceleration layer table of the distributed file system are stored on the same original shard.

[0025] In this embodiment, in a small data scale scenario, the amount of metadata of the distributed file system is less than a preset threshold. In this case, the execution subject of the distributed file system metadata processing method can store the metadata storage layer (Dentry) table and the path resolution acceleration layer (Dtree) table of the distributed file system on the same original shard.

[0026] The metadata processing method for a distributed file system is typically executed by a server. The server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. This is not specifically limited here.

[0027] A distributed file system (DFS) manages physical storage resources that aren't necessarily directly connected to local nodes but rather are connected via a computer network. Alternatively, it manages a complete, hierarchical file system composed of several logical disk partitions or volumes. A DFS provides a logical, tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files across the network.

[0028] The metadata in a distributed file system may include all data in the namespace of the distributed file system, such as directories, directory attributes, and file attributes.

[0029] In small-scale data scenarios, the number of files in a distributed file system is relatively small (no more than a few hundred million), and a single-point metadata service can fully meet the needs. In this case, the metadata storage layer table and the path resolution acceleration layer table of the distributed file system can be stored on the same original shard.

[0030] The metadata storage layer table can be used to store metadata for the distributed file system, thereby distributing the hierarchical namespace within the metadata storage layer table within the distributed database. The metadata storage layer table stores all data in the distributed file system namespace, including directories, directory attributes, and file attributes. Because the volume of the distributed file system's metadata is less than a preset threshold, the metadata storage layer table is not split and is stored entirely on a single original shard.

[0031] The path resolution acceleration layer table can be used to store the path of the metadata of the distributed file system. In order to reduce the number of RPCs (Remote Procedure Calls) performed during the path search process, the path resolution acceleration layer table is introduced. The path resolution acceleration layer table is a table in the database that additionally stores a copy of the directory information of the hierarchical namespace. In the path resolution acceleration layer table, each record contains only the information necessary to perform path search, ensuring that even in a hierarchical namespace of tens of billions, the size of the path resolution acceleration layer table will not exceed 100GB. Therefore, the path resolution acceleration layer table can be stored entirely in one original shard without the need for any sharding, allowing search requests to resolve long paths through a single RPC call to the path resolution acceleration layer table.

[0032] Furthermore, to ensure high system availability, all data changes are first written to the WAL (Write-Ahead Logging) and replicated within the corresponding Raft group. Because the path resolution acceleration layer table does not require range scans and the data volume can fit entirely in memory, a full-memory hash engine is used. The path resolution acceleration layer table acts as a storage shard, implementing complex file system-level operations through UDFs (User-Defined Functions). Once these operations are defined in the upper-layer application, they can be integrated into the path resolution acceleration layer table as plug-ins.

[0033] Figure 2 A schematic diagram of a single-point metadata architecture is shown. Figure 2 As shown, a distributed file system 200 with a single-point metadata architecture may include a data warehouse 201, machine learning 202, a client 203, a network 204, a metadata service 205, and a data service 206. Data service 206 may be used to store file data in distributed file system 200. Metadata service 205 may be used to store metadata in distributed file system 200. The metadata of distributed file system 200 is stored on the same shard. The root node is empty, and its child nodes A and B are used to store metadata for distributed file system 200. The root node, child nodes A, and child nodes B are all on the same shard.

[0034] In some embodiments, in small-scale scenarios, a single-point metadata architecture can be used to store the metadata storage layer table and the path resolution acceleration layer table on the same original shard. At this time, for the processing request of the received target metadata, the path resolution acceleration layer table can be searched on the original shard to determine the target path; based on the target path, the metadata storage layer table can be searched on the original shard, and the target metadata can be processed at the found location. The processing may include but is not limited to: writing, modifying, deleting, etc. Taking writing as an example, since the metadata storage layer table and the path resolution acceleration layer table are stored on the same original shard, the location of the target metadata on the original shard can be queried through only one PRC call, and the target metadata can be written to the original shard through one write, thereby improving the performance in small-scale scenarios.

[0035] Step 102: In response to the data volume of the metadata of the distributed file system being not less than a preset threshold, the original shard is split into a metadata storage layer shard and a path resolution acceleration layer shard.

[0036] In this embodiment, as the data scale grows, the amount of metadata in the distributed file system becomes greater than a preset threshold, making it impossible for a single machine to store both the metadata storage layer table and the path resolution acceleration layer table. In this case, the execution entity can split the original shard into a metadata storage layer shard and a path resolution acceleration layer shard.

[0037] The metadata storage layer shards can be used to store metadata storage layer tables. The path resolution acceleration layer shards can be used to store path resolution acceleration layer tables. By introducing a full-path index shard for each file system, all directory information is maintained on a single server, optimizing the file path resolution process and improving efficiency.

[0038] Step 103: Schedule the metadata storage layer table on the metadata storage layer shard to different data shards.

[0039] In this embodiment, as the data scale continues to grow, the execution entity can further split the metadata storage layer shards and schedule the metadata storage layer tables to different data shards. In addition, these data shards can be further split to carry even larger data scales.

[0040] When splitting metadata storage layer shards, records with the same parent node field can be restricted to the same shard. This ensures that files, directories, and parent directory attributes within the same directory are stored on the same shard, limiting file operations to a single table and improving the efficiency of related operations. Partitioning technology not only supports the expansion of hierarchical namespaces, but also ensures metadata consistency through database transactions and enhances system reliability through data replication. Because metadata storage layer tables need to support range scans, a single-machine storage engine of the lsm-tree type can be used.

[0041] Figure 3 Schematic diagram of the subtree partitioning metadata architecture is shown. Figure 3 As shown, the distributed file system 300 with a subtree partitioning metadata architecture may include a data warehouse 301, machine learning 302, a client 303, a network 304, a metadata service 305, and a data service 306. The data service 306 may be used to store file data in the distributed file system 300. The metadata service 305 may be used to store metadata in the distributed file system 300. The metadata of the distributed file system 300 may be partitioned using subtrees to distribute the metadata in the distributed file system 300 to different nodes, which can roughly evenly distribute the metadata in terms of data volume. The root node is empty, and the root node and its child node A are on shard 1. The root node's child node B, its child node C, and its child node D are on shard 2.

[0042] A distributed file system with a directory-partitioned metadata architecture can integrate distributed file system and database technologies into the distributed file system metadata domain, architecturally dividing metadata services into two layers: the database layer and the metadata proxy layer. The database layer is responsible for data storage, typically using NewSQL, ensuring data persistence while providing distributed transaction capabilities. Metadata is distributed across multiple shards of the database system by directory. The metadata proxy layer can provide POSIX or HDFS interfaces externally and internally convert data in the hierarchical namespace into records in a table system, utilizing transactions to ensure operational correctness.

[0043] Figure 4 Figure 2 shows a schematic diagram of the metadata architecture that integrates directory partitioning and subtree partitioning. Figure 4 As shown, distributed file system 400 may include metadata storage layer shards 401, path resolution acceleration layer shards 402, namespace library 403, master server 404, and time server 405. Metadata storage layer shards 401 can be used to store metadata storage layer tables. Path resolution acceleration layer shards 402 can be used to store path resolution acceleration layer tables. Metadata of distributed file system 400 is stored in the metadata storage layer tables. The paths of metadata of distributed file system 400 are stored in the path resolution acceleration layer tables. When metadata storage layer shards 401 are split, records with the same parent node field can be restricted to the same shard. Specifically, the root node is empty, and the root node's child nodes A and B are on the same shard. Child nodes C and D of child node A are on the same shard. The path resolution acceleration layer table is on a shard and is used to record the paths of nodes A, B, C, and D. Namespace library 403 can implement logic related to directory tree semantics, including but not limited to file operations, directory reads, directory modifications, directory statistics, searches, and rename locks. Taking the directory statistics operation as an example, the namespace library first queries the node identifier of the parent path through an RPC call to the path resolution acceleration layer shard 402, and then queries the directory attribute information through another RPC call to the metadata storage layer shard 401.

[0044] In some embodiments, as the data scale grows, a distributed architecture can be adopted to split the original shards into metadata storage layer shards and path resolution acceleration layer shards. The metadata storage layer table is stored on the metadata storage layer shards, and the path resolution acceleration layer table is stored on the path resolution acceleration layer shards, and further scheduled to different data shards. At this time, for the processing request of the received target metadata, the path resolution acceleration layer table can be searched on the path resolution acceleration layer shards to determine the target path. Based on the target path, the metadata storage layer table is searched on the metadata storage layer shards, and the target metadata is processed at the found location. The path resolution acceleration layer table is completely stored on the path resolution acceleration layer shards, and the target path can be found through a single PRC call to the path resolution acceleration layer shards, thereby reducing the number of PRC calls made during the path search process.

[0045] Taking the case where the processing request is a write request as an example, the target path is the path of the parent node of the target metadata. At this time, based on the path of the parent node of the target metadata, the metadata storage layer table can be searched to determine the parent node of the target metadata. A child node is created for the parent node of the target metadata, the target metadata is written to the child node, and the path of the target metadata is updated to the path resolution acceleration layer table. Since the metadata storage layer table and the path resolution acceleration layer table are stored on different shards, first a PRC call is made to the path resolution acceleration layer shard to query the path of the parent node of the target metadata, and another PRC call is made to the metadata storage layer shard to query the parent node of the target metadata, and then the target metadata is written to the metadata storage layer shard once, and the path of the target metadata is written to the path resolution acceleration layer shard once.

[0046] Taking the example of a non-write processing request, the target path is the path of the target metadata. Based on the path of the target metadata, the metadata storage layer table is searched to determine the node of the target metadata. The target metadata is then processed on the node of the target metadata. This processing may include, but is not limited to, modification, deletion, and so on. Because the metadata storage layer table and the path resolution acceleration layer table are stored on different shards, a PRC call is first made to the path resolution acceleration layer shard to query the path of the target metadata node, and another PRC call is made to the metadata storage layer shard to query the target metadata.

[0047] The disclosed embodiment provides a distributed file system hierarchical namespace solution with a stand-alone and distributed integrated architecture, which has the ability to seamlessly expand from small scale to large scale. It can not only achieve extremely low latency in the initial stage by adopting the stand-alone mode, but also smoothly switch to the efficient distributed mode as the business grows. For the stand-alone mode, in a small-scale scenario, it is possible to use the stand-alone namespace system to place the metadata of a user file system in a shard, achieving low latency of hundreds of microseconds, thereby optimizing performance. For the distributed mode, as the business scale expands, when the scale grows to a point where it cannot be placed on a single machine, it can be seamlessly migrated to a distributed architecture to achieve horizontal expansion and meet the growing processing needs.

[0048] Continue to refer Figure 5 , which shows a process 500 of another embodiment of a metadata processing method for a distributed file system according to the present disclosure. The metadata processing method for a distributed file system includes the following steps:

[0049] Step 501: In response to the data volume of metadata of the distributed file system being less than a preset threshold, a database identifier is allocated to each file system in the distributed file system.

[0050] In this embodiment, in a small data scale scenario, the amount of metadata of the distributed file system is less than a preset threshold. In this case, the execution subject of the distributed file system metadata processing method can assign a database identifier (db_id) to each file system in the distributed file system.

[0051] The metadata processing method for a distributed file system is typically executed by a server. The server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. This is not specifically limited here.

[0052] A distributed file system (DFS) manages physical storage resources that aren't necessarily directly connected to local nodes but rather are connected via a computer network. Alternatively, it manages a complete, hierarchical file system composed of several logical disk partitions or volumes. A DFS provides a logical, tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files across the network.

[0053] By assigning a unique database identifier to each file system in the distributed file system, a logically isolated environment can be created for each file system. Each file system can contain a metadata storage layer table and a path resolution acceleration layer table, ensuring that the data in these two tables is in the same ordered encoding space and can be stored in the same original shard.

[0054] Step 502: Create an original table and set the range of the original table to be from the minimum database identifier to the maximum database identifier.

[0055] In this embodiment, when creating a database, the execution entity may create an original table (Table) and set the range of the original table to be from the minimum database identifier (db_id_MIN) to the maximum database identifier (db_id_MAX).

[0056] The metadata storage layer and path resolution acceleration layer tables of the file system belonging to the same original table are stored on the same original shard. Setting the range of the original table to the key range ([db_id_MIN, db_id_MAX]) ensures that this range is sufficient to cover the data of the entire distributed file system, ensuring that all data in the metadata storage layer and path resolution acceleration layer tables falls within this original shard.

[0057] Figure 6 A schematic diagram of the original table is shown. Figure 6 As shown, Table 1 of distributed file system FS1 includes the FS1Dentry table and the FS1Dtree table. Table 2 of distributed file system FS2 includes the FS2Dentry table and the FS2Dtree table. A primitive table is created for the namespace library of the distributed file system, and the range of the primitive table is set from the minimum database identifier to the maximum database identifier, ensuring that all Dentry tables and Dtree tables of the distributed file system are in the same shard.

[0058] Step 503: In response to the data volume of the metadata of the distributed file system being not less than a preset threshold, the original shard is split at the logical dividing point between the metadata storage layer table and the path resolution acceleration layer table to obtain metadata storage layer shards and path resolution acceleration layer shards.

[0059] In this embodiment, as the data scale grows, the amount of metadata in the distributed file system becomes no less than a preset threshold. At this point, a single machine cannot store both the metadata storage layer table and the path resolution acceleration layer table simultaneously. The aforementioned execution entity can split the original shard at the logical demarcation point between the metadata storage layer table and the path resolution acceleration layer table, resulting in metadata storage layer shards and path resolution acceleration layer shards.

[0060] TafDB's table splitting feature allows you to split tables at the logical demarcation point db_id_dentry_MAX between metadata storage tier tables and path resolution acceleration tier tables. The metadata storage tier shard contains metadata storage tier tables, with a range of [db_id_MIN, db_id_dentry_MAX]. The path resolution acceleration tier shard contains path resolution acceleration tier tables, with a range of [db_id_dentry_MAX, db_id_MAX).

[0061] Step 504: Divide the metadata storage layer table into subtrees, and schedule metadata on child nodes belonging to the same layer and the same parent node to the same data shard.

[0062] In this embodiment, the execution entity may divide the metadata storage layer table into subtrees, and schedule metadata on child nodes belonging to the same layer and the same parent node to the same data shard.

[0063] When splitting metadata storage layer shards, records with the same parent node field can be restricted to the same shard. This ensures that files, directories, and parent directory attributes within the same directory are stored on the same shard, limiting file operations to a single table and improving the efficiency of related operations. Partitioning technology not only supports the expansion of hierarchical namespaces, but also ensures metadata consistency through database transactions and enhances system reliability through data replication. Because metadata storage layer tables need to support range scans, a single-machine storage engine of the lsm-tree type can be used.

[0064] Figure 7 A schematic diagram of the original table split is shown. Figure 7 As shown, table 1 of the distributed file system FS1 may include an FS1 Dentry table and an FS1 Dtree table. The Dentry table and the Dtree table are split at a logical demarcation point, with the FS1Dentry table on one shard and the FS1Dtree table on another shard.

[0065] The disclosed embodiment provides a distributed file system hierarchical namespace solution with a stand-alone and distributed integrated architecture, which has the ability to seamlessly expand from small scale to large scale. It can not only achieve extremely low latency in the initial stage by adopting the stand-alone mode, but also smoothly switch to the efficient distributed mode as the business grows. For the stand-alone mode, in a small-scale scenario, it is possible to use the stand-alone namespace system to place the metadata of a user file system in a shard, achieving low latency of hundreds of microseconds, thereby optimizing performance. For the distributed mode, as the business scale expands, when the scale grows to a point where it cannot be placed on a single machine, it can be seamlessly migrated to a distributed architecture to achieve horizontal expansion and meet the growing processing needs.

[0066] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a metadata processing device for a distributed file system. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0067] like Figure 8 As shown, the metadata processing device 800 of the distributed file system of this embodiment may include: a storage module 801, a splitting module 802, and a scheduling module 803. The storage module 801 is configured to, in response to the amount of metadata of the distributed file system being less than a preset threshold, store the metadata storage layer table and the path resolution acceleration layer table of the distributed file system on the same original shard, wherein the metadata storage layer table is used to store the metadata of the distributed file system, and the path resolution acceleration layer table is used to store the path of the metadata of the distributed file system; the splitting module 802 is configured to, in response to the amount of metadata of the distributed file system being not less than a preset threshold, split the original shard into a metadata storage layer shard and a path resolution acceleration layer shard, wherein the metadata storage layer shard stores the metadata storage layer table, and the path resolution acceleration layer shard stores the path resolution acceleration layer table; and the scheduling module 803 is configured to schedule the metadata storage layer table on the metadata storage layer shard to different data shards.

[0068] In this embodiment, in the metadata processing device 800 of the distributed file system, the specific processing of the storage module 801, the splitting module 802 and the scheduling module 803 and the technical effects thereof can be referred to in the respective Figure 1 The relevant descriptions of steps 101-103 in the corresponding embodiment are not repeated here.

[0069] In some optional implementations of this embodiment, the storage module 801 is further configured to: assign a database identifier to each file system in the distributed file system; create an original table and set the range of the original table to be from the minimum database identifier to the maximum database identifier, wherein the metadata storage layer table and the path resolution acceleration layer table of the file system belonging to the same original table are stored on the same original shard.

[0070] In some optional implementations of this embodiment, the splitting module 802 is further configured to: split the original shards at the logical dividing point between the metadata storage layer table and the path resolution acceleration layer table to obtain metadata storage layer shards and path resolution acceleration layer shards, wherein the range of the metadata storage layer shards is from the minimum database identifier to the identifier of the logical dividing point, and the range of the path resolution acceleration layer shards is from the identifier of the logical dividing point to the maximum database identifier.

[0071] In some optional implementations of this embodiment, the scheduling module 803 is further configured to: divide the metadata storage layer table into subtrees, and schedule the metadata on the child nodes belonging to the same layer and the same parent node to the same data shard.

[0072] In some optional implementations of this embodiment, the metadata processing device 800 of the distributed file system also includes: a receiving module, configured to receive a processing request for target metadata; a first determination module, configured to, in response to the metadata storage layer table and the path resolution acceleration layer table being stored on the same original shard, search the path resolution acceleration layer table on the original shard based on the processing request to determine the target path; a first processing module, configured to search the metadata storage layer table on the original shard based on the target path, and process the target metadata at the found location.

[0073] In some optional implementations of this embodiment, the metadata processing device 800 of the distributed file system also includes: a second determination module, configured to respond to the metadata storage layer table being stored on the metadata storage layer table and the path resolution acceleration layer table being stored on the path resolution acceleration layer table, and based on the processing request, search the path resolution acceleration layer table on the path resolution acceleration layer shard to determine the target path; a second processing module, configured to search the metadata storage layer table on the metadata storage layer shard based on the target path, and process the target metadata at the found location.

[0074] In some optional implementations of this embodiment, the metadata processing device 800 of the distributed file system also includes: the processing request is a write request, the target path is the path of the parent node of the target metadata; and the second processing module is further configured to: based on the path of the parent node of the target metadata, search the metadata storage layer table to determine the parent node of the target metadata; create a child node for the parent node of the target metadata, write the target metadata into the child node, and update the path of the target metadata to the path resolution acceleration layer table.

[0075] In some optional implementations of this embodiment, the metadata processing device 800 of the distributed file system also includes: the processing request is a non-write request, the target path is the path of the target metadata; and the second processing module is further configured to: based on the path of the target metadata, search the metadata storage layer table to determine the node of the target metadata; and process the target metadata on the node of the target metadata.

[0076] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0077] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0078] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0079] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0080] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0081] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the metadata processing method for a distributed file system. For example, in some embodiments, the metadata processing method for a distributed file system can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the metadata processing method for a distributed file system described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the metadata processing method for a distributed file system by any other suitable means (e.g., via firmware).

[0082] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0083] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0086] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0087] A computer system may include clients and servers. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0088] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0089] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A metadata processing method for a distributed file system, comprising: In response to the data volume of the metadata of the distributed file system being less than a preset threshold, storing the metadata storage layer table and the path resolution acceleration layer table of the distributed file system on the same original shard, wherein the metadata storage layer table is used to store the metadata of the distributed file system, and the path resolution acceleration layer table is used to store the path of the metadata of the distributed file system; In response to the data volume of the metadata of the distributed file system being not less than the preset threshold, splitting the original shard into a metadata storage layer shard and a path resolution acceleration layer shard, wherein the metadata storage layer shard is used to store the metadata storage layer table, and the path resolution acceleration layer shard is used to store the path resolution acceleration layer table; The metadata storage layer table on the metadata storage layer shard is scheduled to different data shards.

2. The method according to claim 1, wherein: The step of storing the metadata storage layer table and the path resolution acceleration layer table of the distributed file system on the same original shard includes: Assigning a database identifier to each file system in the distributed file system; An original table is created, and a range of the original table is set from a minimum database identifier to a maximum database identifier, wherein a metadata storage layer table and a path resolution acceleration layer table of a file system belonging to the same original table are stored on the same original shard.

3. The method according to claim 2, wherein: The step of splitting the original shards into metadata storage layer shards and path resolution acceleration layer shards includes: At the logical dividing point between the metadata storage layer table and the path resolution acceleration layer table, the original shard is split to obtain the metadata storage layer shard and the path resolution acceleration layer shard, wherein the metadata storage layer shard ranges from the minimum database identifier to the identifier of the logical dividing point, and the path resolution acceleration layer shard ranges from the identifier of the logical dividing point to the maximum database identifier.

4. The method according to claim 1, wherein: The step of scheduling the metadata storage layer table on the metadata storage layer shard to different data shards includes: The metadata storage layer table is divided into subtrees, and metadata on child nodes belonging to the same layer and the same parent node are scheduled to the same data shard.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: receiving a processing request for target metadata; In response to the metadata storage layer table and the path resolution acceleration layer table being stored on the same original slice, based on the processing request, searching the path resolution acceleration layer table on the original slice to determine the target path; Based on the target path, the metadata storage layer table is searched on the original shard, and the target metadata is processed at the searched position.

6. The method according to claim 5, wherein: The method further comprises: In response to the metadata storage layer table being stored on the metadata storage layer table, the path resolution acceleration layer table being stored on the path resolution acceleration layer table, searching the path resolution acceleration layer table on the path resolution acceleration layer slice based on the processing request to determine a target path; Based on the target path, the metadata storage layer table is searched on the metadata storage layer shard, and the target metadata is processed at the searched position.

7. The method according to claim 6, wherein: The processing request is a write request, and the target path is the path of the parent node of the target metadata; as well as The step of searching the metadata storage layer table on the metadata storage layer shard based on the target path and processing the target metadata at the searched position includes: Based on the path of the parent node of the target metadata, searching the metadata storage layer table to determine the parent node of the target metadata; A child node is created for the parent node of the target metadata, the target metadata is written into the child node, and the path of the target metadata is updated into the path resolution acceleration layer table.

8. The method according to claim 7, wherein: The processing request is a non-write request, and the target path is a path of the target metadata; as well as The step of searching the metadata storage layer table on the metadata storage layer shard based on the target path and processing the target metadata at the searched position includes: Based on the path of the target metadata, searching the metadata storage layer table to determine the node of the target metadata; The target metadata is processed on a node of the target metadata.

9. A metadata processing device for a distributed file system, comprising: A storage module, configured to store a metadata storage layer table and a path resolution acceleration layer table of the distributed file system on the same original shard in response to the data volume of the metadata of the distributed file system being less than a preset threshold, wherein the metadata storage layer table is used to store the metadata of the distributed file system, and the path resolution acceleration layer table is used to store the path of the metadata of the distributed file system; a splitting module configured to split the original shard into a metadata storage layer shard and a path resolution acceleration layer shard in response to the data volume of the metadata of the distributed file system being not less than the preset threshold, wherein the metadata storage layer shard stores the metadata storage layer table, and the path resolution acceleration layer shard stores the path resolution acceleration layer table; The scheduling module is configured to schedule the metadata storage layer table on the metadata storage layer shard to different data shards.

10. The device according to claim 9, wherein: The storage module is further configured to: Assigning a database identifier to each file system in the distributed file system; An original table is created, and a range of the original table is set from a minimum database identifier to a maximum database identifier, wherein a metadata storage layer table and a path resolution acceleration layer table of a file system belonging to the same original table are stored on the same original shard.

11. The device according to claim 10, wherein: The splitting module is further configured to: At the logical dividing point between the metadata storage layer table and the path resolution acceleration layer table, the original shard is split to obtain the metadata storage layer shard and the path resolution acceleration layer shard, wherein the metadata storage layer shard ranges from the minimum database identifier to the identifier of the logical dividing point, and the path resolution acceleration layer shard ranges from the identifier of the logical dividing point to the maximum database identifier.

12. The device according to claim 9, wherein: The scheduling module is further configured to: The metadata storage layer table is divided into subtrees, and metadata on child nodes belonging to the same layer and the same parent node are scheduled to the same data shard.

13. The device according to any one of claims 9 to 12, wherein: The device also includes: A receiving module configured to receive a processing request for target metadata; A first determination module is configured to, in response to the metadata storage layer table and the path resolution acceleration layer table being stored on the same original shard, search the path resolution acceleration layer table on the original shard based on the processing request to determine a target path; The first processing module is configured to search the metadata storage layer table on the original shard based on the target path, and process the target metadata at the searched position.

14. The device according to claim 13, wherein: The device also includes: A second determination module is configured to, in response to the metadata storage layer table being stored on the metadata storage layer table and the path resolution acceleration layer table being stored on the path resolution acceleration layer table, search the path resolution acceleration layer table on the path resolution acceleration layer slice based on the processing request to determine the target path; The second processing module is configured to search the metadata storage layer table on the metadata storage layer shard based on the target path, and process the target metadata at the searched position.

15. The device according to claim 14, wherein: The processing request is a write request, and the target path is a path of a parent node of the target metadata; and The second processing module is further configured to: Based on the path of the parent node of the target metadata, searching the metadata storage layer table to determine the parent node of the target metadata; A child node is created for the parent node of the target metadata, the target metadata is written into the child node, and the path of the target metadata is updated into the path resolution acceleration layer table.

16. The device according to claim 15, wherein: The processing request is a non-write request, and the target path is a path of the target metadata; and The second processing module is further configured to: Based on the path of the target metadata, searching the metadata storage layer table to determine the node of the target metadata; The target metadata is processed on a node of the target metadata.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images