Metadata management method and related equipment

Abstract

The embodiment of the invention discloses a metadata management method and related equipment, belongs to the technical field of distributed file systems, and is used for balancing metadata management capacity and concurrent access performance and improving metadata management efficiency. The method comprises a management node applied to a distributed file system, the distributed storage system further comprises a metadata node, and the metadata node is used for storing metadata. The method comprises the following steps: acquiring path feature information from a data warehouse metadata service, wherein the path feature information is used for representing a directory hierarchical relationship followed by a metadata node when the metadata node stores metadata; determining routing node information based on the path feature information; sending the routing node information to the client; the routing node information is used for the client to determine a target metadata node corresponding to the access request, and the access request is used for performing access operation on the target metadata node.

Description

Technical Field

[0001] This application relates to the field of distributed file system technology, and in particular to a metadata management method and related equipment. Background Technology

[0002] A distributed file system (DFS) refers to a file system that manages physical resources that can be interconnected via a network to form a complete file system. In other words, a distributed file system connects a large number of scattered computing devices together through a network, forming a huge computing device cluster, allowing each device to fully utilize its value. Distributed file systems have advantages in storage capacity, storage capacity growth rate, and data security. Therefore, distributed storage systems can provide highly reliable, scalable, fault-tolerant, and load-balanced storage solutions for large-scale data management and processing scenarios.

[0003] However, when the number of files in a distributed file system reaches a certain threshold, the performance of distributed concurrent access drops sharply, affecting business stability. Therefore, the technical problem of balancing metadata management capacity and concurrent access performance, and improving metadata management efficiency in distributed file systems, urgently needs to be solved. Summary of the Invention

[0004] This application provides a metadata management method and related equipment that can balance metadata management capacity and concurrent access performance, thereby improving metadata management efficiency.

[0005] Firstly, a metadata management method is provided, which is applied to the management node in a distributed file system. The distributed storage system also includes metadata nodes, which are used to store metadata. The method includes: obtaining path feature information from the data warehouse metadata service, whereby the path feature information is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata; determining routing node information based on the path feature information; and sending the routing node information to the client. The routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform access operations on the target metadata node.

[0006] Understandably, the management node determines routing node information based on path feature information from the data warehouse metadata service, and uses this routing node information for unified management of metadata nodes. This routing node information is sent to the client, allowing the client to determine the target metadata node and complete the access operation. This linearly increases the capacity and access capabilities of metadata management, thereby improving the efficiency of metadata management.

[0007] In one possible implementation, path feature information includes a storage path used to represent the metadata when it is stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node; where N and M are positive integers greater than 1, and N is greater than or equal to M.

[0008] It is understandable that, since a routing node information includes the root path node in a storage path and at least M level child path nodes under the root path node, the routing node information can be dynamically determined based on the capacity of the metadata and the storage path, thereby improving the accuracy of the routing node information.

[0009] In one possible implementation, at least N-level sub-path nodes include a first sub-path node under the root path node and a second sub-path node under the first sub-path node; the root path node is used to represent the database corresponding to the metadata node; the first sub-path node is used to represent a partition in the database; and the first sub-path node is used to represent a table in the partition.

[0010] In one possible implementation, before sending the routing node information to the client, the method further includes: performing structured processing on the routing node information to obtain a prefix tree; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M level child path nodes; performing serialization processing on the prefix tree to obtain the serialization result corresponding to the prefix tree; the serialization result supports transmission over the network; sending the routing node information to the client includes: sending the serialization result to the client.

[0011] Understandably, by structuring the routing node information and generating a prefix tree, storage space for the routing node information can be saved, and the matching speed can be improved. Furthermore, by serializing the prefix tree, the corresponding prefix tree can be transmitted over the network, improving the usability of the routing node information.

[0012] In one possible implementation, the position of each routing node's information on the hash ring is determined by performing consistent hashing on the routing node information. Metadata nodes satisfy the consistent hash function and are distributed on the hash ring. Based on the position of each routing node's information on the hash ring, the mapping relationship between the routing node information and the metadata nodes is determined. Specifically, the first metadata node adjacent to the routing node information on the hash ring in a clockwise direction is the metadata node corresponding to the routing node information. This mapping relationship is used to verify client access requests for the target metadata node.

[0013] Understandably, by performing consistent hashing on the routing node information through the management node, the mapping relationship between each routing node's information and the metadata node is determined. This mapping relationship is used by the client to verify access requests to the target metadata node, thereby improving the accuracy of client access.

[0014] Secondly, a metadata management method is provided, which is executed by the client and includes: receiving routing node information sent by a management node; the management node is a node in a distributed file system, which also includes metadata nodes for storing metadata; the routing node information is generated by the management node based on path feature information; the path feature information is created by a data warehouse metadata service and is used to represent the directory hierarchy followed by the metadata nodes when storing metadata; obtaining an access request; the access request includes access path information; determining the target metadata node based on the access path information and the routing node information; and performing an access operation on the target metadata node.

[0015] Understandably, the client can determine the target metadata node based on the received routing node information and access path information. The client can then access the target metadata node. When accessing the target metadata node, the client only needs to use the cached routing node information to determine the target metadata node, which improves the performance of concurrent metadata access and thus enhances the efficiency of metadata management.

[0016] In one possible implementation, the access path information is matched with the information of each routing node to determine the matching result. If the matching result indicates that the target routing node information is matched according to the access path information, the target routing node information is subjected to consistent hashing to obtain the position of the target routing node information on the hash ring. Based on the position of the target routing node information on the hash ring, the target metadata node is determined. The target metadata node is the first metadata node adjacent to the target routing node information on the hash ring in a clockwise direction. The metadata nodes in the distributed storage system satisfy the consistent hash function and are distributed on the hash ring.

[0017] Understandably, the client can improve matching efficiency by matching the access path information with the routing node information. Furthermore, by performing consistent hashing on the target routing node information to determine its position on the hash ring, the target metadata node can be identified, thus improving the efficiency of identifying the target metadata node.

[0018] In one possible implementation, if the matching result indicates that no routing node information is matched based on the access path information, the root path information in the access path information is subjected to consistent hashing to obtain the position of the root path information on the hash ring; based on the position of the root path information on the hash ring, the target metadata node is determined; wherein, the target metadata node is the first metadata node adjacent to the root path information on the hash ring in a clockwise direction.

[0019] Understandably, if the matching result indicates that no routing node information is matched based on the access path information, the client can perform consistent hashing on the root path information in the access path information to obtain the position of the root path information on the hash ring, thereby determining the target metadata node and improving the stability of the client access operation.

[0020] In one possible implementation, path feature information includes a storage path representing the metadata stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node; access path information includes a root path node in a storage path and at least K levels of sub-path nodes under the root path node; wherein N, M, and K are positive integers greater than 1, and N is greater than or equal to M.

[0021] It is understandable that, since a routing node information includes the root path node in a storage path and at least M level child path nodes under the root path node, the routing node information can be dynamically determined based on the capacity of the metadata and the storage path, thereby improving the accuracy of the routing node information.

[0022] In one possible implementation, the serialization result sent by the management node is received; the serialization result is obtained by serializing the prefix tree, and the serialization result supports transmission over the network; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M level child path nodes; the serialization result is deserialized to obtain the prefix tree; and the routing node information is obtained based on the prefix tree.

[0023] Understandably, the client receives the serialized result sent by the management node. This serialized result is obtained by serializing the prefix tree, allowing the prefix tree corresponding to the routing node information to be transmitted over the network, thus improving the usability of the routing node information. Simultaneously, by generating the prefix tree from the routing node information through structured processing, storage space for the routing node information can be saved, and the matching speed of the routing node information can be improved.

[0024] Thirdly, embodiments of this application provide a metadata management device. Embodiments of this application can divide the metadata management device into functional modules according to the method provided in the first aspect above. For example, each function can be divided into its own functional modules, or two or more functions can be inherited into a single processing module. For instance, embodiments of this application can divide the metadata management device into an acquisition module, a processing module, and a sending module according to their functions. Descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the first aspect above or its corresponding possible implementations, and will not be repeated here.

[0025] Fourthly, embodiments of this application provide a metadata management device. Based on the method provided in the second aspect above, this application can divide the metadata management device into functional modules. For example, each function can be divided into its own functional modules, or two or more functions can be inherited into a single processing module. For instance, embodiments of this application can divide the metadata management device into a receiving module, an acquiring module, a processing module, and an access module according to their functions. Descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the second aspect above or its corresponding possible implementations, and will not be repeated here.

[0026] Fifthly, embodiments of this application provide a distributed file system, which includes a management node and a metadata node. The management node is used to execute the metadata management method provided in the first aspect; the metadata node is used to store metadata.

[0027] In a sixth aspect, embodiments of this application provide a server, the server including: a processor and a memory for storing processor-executable instructions;

[0028] The processor is configured to execute instructions that cause the server to perform the metadata management method provided in the first aspect above.

[0029] In a seventh aspect, embodiments of this application provide a computing device, which includes a processor and a memory for storing processor-executable instructions; the processor is configured to execute instructions, causing the computing device to perform the metadata management method provided in the second aspect above.

[0030] Eighthly, embodiments of this application provide a computer program product containing instructions that, when executed by a management node, cause the management node to perform the metadata management method provided in the first aspect.

[0031] Ninthly, embodiments of this application provide a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the metadata management method provided in the second aspect above.

[0032] In a tenth aspect, embodiments of this application provide a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the metadata management method provided in the first aspect above.

[0033] In one aspect, embodiments of this application provide a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the metadata management method provided in the second aspect above.

[0034] For a detailed description of the third to eleventh aspects and their various implementations in the embodiments of this application, please refer to the detailed descriptions in the first aspect, the second aspect and their various implementations; and for a detailed description of the beneficial effects of the third to eleventh aspects and their various implementations, please refer to the beneficial effect analysis in the first aspect and the various implementations of the second aspect, which will not be repeated here.

[0035] These or other aspects of the embodiments of this application will become more apparent in the following description. Attached Figure Description

[0036] Figure 1 This paper illustrates a schematic diagram of a general architecture for a distributed file system provided in an embodiment of this application.

[0037] Figure 2 This illustration shows a schematic diagram of an Alluxio file management software architecture provided in an embodiment of this application;

[0038] Figure 3 This illustration shows a schematic diagram of an access flow for the Alluxio distributed data orchestration system provided in an embodiment of this application.

[0039] Figure 4 This illustration shows a schematic diagram of a structured file management software architecture provided in an embodiment of this application;

[0040] Figure 5 This illustration shows an access flow diagram of a structured distributed file system provided in an embodiment of this application;

[0041] Figure 6 This illustration shows a schematic diagram of a gateway federated distributed file system architecture provided in an embodiment of this application;

[0042] Figure 7This illustration shows a gateway federated distributed file system access process according to an embodiment of this application;

[0043] Figure 8 This illustration shows a schematic diagram of a distributed file architecture system provided in an embodiment of this application;

[0044] Figure 9 This illustration shows a flowchart of a metadata management method provided in an embodiment of this application;

[0045] Figure 10 This illustration shows a schematic diagram of a prefix tree 600 provided in an embodiment of this application;

[0046] Figure 11 This illustration shows a mapping relationship between metadata nodes and routing nodes provided in an embodiment of this application;

[0047] Figure 12 This illustration shows a flowchart of a metadata management method provided in an embodiment of this application;

[0048] Figure 13 This illustration shows a structural schematic diagram of a metadata management device 800 provided in an embodiment of this application;

[0049] Figure 14 This illustration shows a structural schematic diagram of a metadata management device 1000 provided in an embodiment of this application;

[0050] Figure 15 This illustration shows a hardware schematic diagram of a computing device provided in an embodiment of this application;

[0051] Figure 16 This illustration shows a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0052] Figure 17 A schematic diagram illustrating a possible cluster implementation method provided in an embodiment of this application is shown. Detailed Implementation

[0053] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to define a specific order.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] First, the application scenarios of the embodiments of this application will be introduced by way of example.

[0056] Distributed file systems are fundamental components of data processing systems. During the loading, computation, and querying processes, a large number of file metadata operations are performed. Therefore, the storage capacity and concurrency latency are typically important metrics for distributed file systems. In existing distributed file systems, once the amount of file metadata reaches a specified threshold, the performance of distributed concurrent access deteriorates sharply, impacting business stability.

[0057] Figure 1 A schematic diagram of a general architecture for a distributed file system provided in an embodiment of this application is shown. For example... Figure 1 As shown in the schematic diagram 100 of the general architecture of the distributed file system, there are computing devices 110, servers 120, and storage arrays 130. The computing devices 110, servers 120, and storage arrays 130 are connected via a network.

[0058] The computing device 110 runs a client for accessing the distributed file system; specifically, it can be a desktop computer, laptop computer, or smartphone. The server 120 stores and manages metadata in the distributed file system and handles client access requests; specifically, the server 120 can be a central server, an edge server, or a local server in a local data center. The storage array 130 stores file data in the distributed file system. The storage array 130 consists of a large number of storage units, which can be volatile memory, non-volatile memory, hard disks, or flash drives.

[0059] In related technologies, for example Figure 1 The distributed file system shown can tier the storage of metadata and perform hot and cold storage to increase the storage capacity of metadata, thus reducing file metadata capacity and operation latency.

[0060] For example, Alluxio can be used for hierarchical storage of metadata. Alluxio is an open-source distributed data orchestration system. It can classify and manage metadata based on its frequency and access requirements, and store it in different storage tiers according to management strategies. For instance, frequently accessed metadata can be stored in high-speed storage, while less frequently accessed metadata can be stored in lower-cost storage. Specifically, Figure 2 A schematic diagram 200 of an Alluxio file management software architecture provided in an embodiment of this application is shown. Figure 2As shown, the Alluxio file management software architecture includes a client 210, a data storage node 220, an object storage 230, a distributed file system 240, and a metadata storage node 250. The Alluxio distributed data orchestration system utilizes the high-performance key-value storage engine RockDB to store metadata, ensuring write and random read performance through a Log Structured Merge Tree (LSM). For hot metadata, the storage architecture of object storage 230 is adopted. Object storage 230 is a data storage structure that treats data as independent data objects, each including the data itself and metadata. Data storage node 220 is used to store data; metadata storage node 250 is used to store and manage metadata; the client 210 can run on a computing device to receive access requests and perform access operations.

[0061] Figure 3 This illustration shows a schematic diagram of an access flow for the Alluxio distributed data orchestration system provided in an embodiment of this application. Figure 3 As shown, client 210 runs on computing device 110 and is used for access operations. The Alluxio system runs on server 120. Computing device 110 can initialize the client and establish a connection with the metadata node in the Alluxio system. Computing device 110 receives access requests through client 210 (S101) and determines the access path based on the access request (S102). Computing device 110 sends the access path to server 120 (S103). Server 120 can access the target metadata node through the Alluxio system call interface (S104) and complete the access request through the target metadata node (S105).

[0062] However, the above method is limited by the memory and disk space constraints of server 120, and cannot meet the linear growth of metadata storage capacity. At the same time, due to the limited load capacity of server 120's central processing unit (CPU), it cannot handle high-concurrency file access operations.

[0063] In some examples, for distributed file systems with a structured architecture, metadata storage and computation can be separated, thereby increasing the storage capacity of metadata. Figure 4 This illustration shows a schematic diagram of a structured file management software architecture provided by an embodiment of this application. For example... Figure 4As shown in the diagram 300, the structured file management software architecture comprises three main parts: a customer database, a block store, and metadata storage. The metadata storage includes a naming layer, a file layer, a block layer, and a key-value storage system. The naming layer, file layer, and block layer communicate with the key-value storage system over a network. The naming layer, file layer, and block layer storage fragment the metadata, abstracting it into a simple key-value model. Due to the fragmentation of metadata, the structured distributed file system relies on a strongly consistent key-value store and its transactions to ensure metadata consistency. The block store is used to store metadata.

[0064] Figure 5 This illustration shows a schematic diagram of the access flow for a distributed file system structured according to an embodiment of this application. For example... Figure 5 As shown, client 210 runs on computing device 110 and is used for access operations. The structured distributed file system runs on server 120. Database server 310 is used to run and manage the metadata database; computing device 110 can initialize the client and establish a connection with the metadata nodes in the structured distributed file system. Computing device 110 receives an access request from client 210 (S201) and determines the access path. Computing device 110 sends the access path to server 120 (S202). Server 120 calls the interface to access the metadata node according to the access path (S203). Server 120 generates an access statement (S204). The access statement can indicate the creation, deletion, or renaming of metadata. Server 120 uses the access statement to access database server 310 (S205) to complete the metadata access.

[0065] However, the above method requires the client to access an external metadata database, which increases the addressing cost and leads to a decrease in metadata concurrency performance.

[0066] In other examples, metadata is managed through router-based federation (RBF) management software. This involves using a gateway agent to manage multiple independent sub-clusters, forming a unified directory for metadata scaling and management. Figure 6 A schematic diagram of a gateway federated distributed file system architecture provided in an embodiment of this application is shown. Figure 6As shown in the diagram 400, the gateway federated distributed file system architecture includes N sub-clusters, where N is a positive integer greater than 1. The first sub-cluster includes a routing service, which reports metadata information from metadata nodes to the state storage service and maintains routing information. Metadata nodes store metadata; data nodes store the data corresponding to the metadata. The routing service and metadata nodes provide interfaces for client applications to perform access operations. Figure 7 This illustration shows a schematic diagram of a gateway federated distributed file system access process provided in an embodiment of this application. Figure 7 As shown, client 210 runs on computing device 110 and is used for access operations. The gateway federated distributed file system runs on server 120. Before computing device 110 accesses metadata nodes through client 210, relevant technical personnel configure static routing rules and mount multiple independent metadata nodes through the routing service. Computing device 110 receives an access request through client 210 (S301) and determines the access path. Computing device 110 sends the access path to server 120 (S302); simultaneously, it establishes a connection with the routing service based on the access path. Server 120 determines the target routing service based on the access path (S303). Server 120 receives the access request through the target routing service (S304), sends the access request to the target metadata node through the target routing service (S305), and completes the access operation through the target metadata node.

[0067] However, the above methods lack linear horizontal scalability, with a maximum of 100 million data points managed by a single sub-cluster. Furthermore, the number of metadata nodes managed using RBF is limited, failing to meet the linear growth requirements of metadata management capacity. Additionally, the path for users to access metadata through clients is relatively long, requiring multiple steps... Figure 7 The two transmissions of S304 and S305 cause a loss of more than 10% in metadata access performance, resulting in a decrease in metadata concurrency performance.

[0068] As the methods described above demonstrate, the issues of storage capacity and concurrent access remain unresolved when faced with storage demands for massive metadata volumes. Therefore, a robust research direction is needed for managing large-capacity file metadata that allows for linear growth in metadata volume without performance degradation during concurrent access.

[0069] In view of this, this application provides a metadata management method. This method is applied to a management node in a distributed storage system, which also includes metadata nodes for storing metadata. The method includes the management node obtaining path feature information from a data warehouse metadata service and generating routing node information based on the path feature information. The routing node information is used by the client to determine the target metadata node corresponding to an access request, and the access request is used to perform access operations on the target metadata node. The path feature information is created by the data warehouse metadata service and is used to represent the directory hierarchy followed by the metadata node when storing metadata. The management node sends the routing node information to the client, which obtains the access request; the access request includes access path information. The client determines the target metadata node based on the access path information and the routing node information and performs access operations on the target metadata node. The above method determines routing node information through path feature information in the data warehouse metadata service and sends the routing node information to the client. When the client obtains an access request, it determines the target metadata node to be accessed based on the routing node information and performs access operations on the target metadata node. This not only increases the capacity of metadata management but also ensures the efficiency of client metadata access and improves the performance of concurrent access. This improves the efficiency of metadata management.

[0070] Secondly, the system architecture of the embodiments of this application will be described by way of example.

[0071] The embodiments of this application can be applied to distributed file systems. Figure 8 A schematic diagram of a distributed file architecture system provided in an embodiment of this application is shown. Figure 8 As shown, the distributed file system architecture 500 includes a computing device 110 and a server 120. The computing device 110 and the server 120 can be connected via a network.

[0072] Specifically, server 120 can be a blade server, high-density server, rack server, or high-performance server, etc. Server 120 includes management node 510 and metadata node 520.

[0073] It should be noted that server 120 can include both management node 510 and metadata node 520, or it can include only management node 510; there is no limitation here.

[0074] Metadata node 520 is used to store metadata and can also provide services for handling file operation semantics in the metadata service, completing the addition, deletion, modification, and query of file semantic metadata. Management node 510 can provide cross-node semantic operations for the metadata service and manage metadata node 520. Management node 510 can obtain path feature information from the data warehouse metadata service and determine routing node information based on the path feature information. Management node 510 is also used to send routing node information to the client. The path feature information represents the directory hierarchy followed by the metadata node when storing metadata. The routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform access operations on the target metadata node.

[0075] A client 210 for accessing the distributed file system runs on the computing device 110; the computing device 110 can be a desktop computer, laptop computer, or smartphone, etc. The client 210 receives routing node information sent by the management node 510 and obtains an access request, which includes access path information. Based on the access path information and routing node information, the client 210 determines the target metadata node and performs access operations on the target metadata node. The management node is a node in the distributed file system, which also includes metadata nodes for storing metadata; the routing node information is generated by the management node based on path feature information; the path feature information is created by the data warehouse metadata service and is used to represent the directory hierarchy followed by the metadata node when storing metadata.

[0076] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0077] For ease of understanding, the metadata management method provided in this application is described below with reference to the accompanying drawings. This metadata management method is applicable to... Figure 8 The distributed file system architecture shown is 500.

[0078] Figure 9 This illustration shows a flowchart of a metadata management method provided in an embodiment of this application. This metadata management method can be applied to… Figure 8 The distributed file system 500 shown includes server 120. Server 120 includes a management node and a metadata node. The method is executed by the management node and includes the following steps:

[0079] S401, the management node obtains path feature information from the data warehouse metadata service.

[0080] Among them, path feature information is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata.

[0081] For example, if the distributed file system receives a request to create routing node information, the data warehouse metadata service can call the management node's interface using path feature information based on the directory hierarchy followed when storing metadata by the metadata node, and the management node can obtain the path feature information.

[0082] For example, a data warehouse metadata service could be a Hive Meta Store (HMS). HMS is a service used to maintain metadata within the data warehouse tool Hive. HMS is responsible for managing and maintaining the relationships between databases, tables, and partitions in the metadata. If the distributed file system receives a request to create a routing node, HMS can use the directory hierarchy followed by the metadata node when storing the metadata to call the management node interface using the storage location information of the created metadata, such as ` / dws / test1.db / table1`. Therefore, the management node can obtain path characteristic information from HMS.

[0083] S402, the management node determines the routing node information based on path feature information.

[0084] In this embodiment of the application, the management node can continuously acquire path feature information multiple times.

[0085] In one possible implementation, path feature information includes a storage path used to represent the metadata when it is stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node;

[0086] Where N and M are positive integers greater than 1, and N is greater than or equal to M.

[0087] For example, at least N levels of sub-path nodes include a first sub-path node under the root path node and a second sub-path node under the first sub-path node; the root path node is used to represent the database corresponding to the metadata node; the first sub-path node is used to represent a partition in the database; the first sub-path node is used to represent a table in the partition.

[0088] For example, if the management node obtains a path feature information P from the data warehouse metadata service as dws / test1.db / table1, then dws is the root path node in path feature information P, representing the database corresponding to the metadata node; test1 is the first child path node in path feature information P, representing a partition in the database; and table1 is the second child path node in path feature information P, representing a table in the database. Based on the path feature information P, the management node determines the routing node information as dws / test1.db / table1. If the management node obtains a path feature information Q from the data warehouse metadata as dws / test2.db / table1, then the management node can determine the routing node information as dws / test2.db / table1.

[0089] It should be noted that since a single routing node information entry can include the root path node and at least M levels of child path nodes under the root path node in a stored path, the routing node information can be dynamically determined based on the metadata capacity. If the metadata capacity is small, then M is less than N; if the metadata capacity is large, then M equals N. This dynamic determination method ensures the accuracy of the routing node information.

[0090] S403, the management node sends routing node information to the client.

[0091] The routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform access operations on the target metadata node.

[0092] In one possible implementation, the management node performs structured processing on the routing node information to obtain a prefix tree. The management node then performs serialization processing on the prefix tree to obtain the corresponding serialized result.

[0093] In this system, the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M levels of child path nodes. The serialization result supports transmission over a network.

[0094] The management node sends routing node information to the client, including sending the serialization result to the client.

[0095] For example, the management node obtains multiple path feature information from the data warehouse metadata service. Based on this information, it determines multiple routing node information. The management node then performs structured processing on this routing node information to generate a prefix tree. The root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node corresponds to at least M levels of child path nodes. The management node then performs serialization processing on the prefix tree to obtain a serialized result. Since the serialized result can be transmitted over the network, the prefix tree can also be transmitted over the network.

[0096] For example, based on the characteristics of multiple paths, the management node determines the information of multiple routing nodes as follows: / dws / test2.db / table1, / dws / test1.db / table2, / dws / test2.db / table3, and / dws / test3.db. Figure 10 A schematic diagram of a prefix tree 600 provided in an embodiment of this application is shown. For example... Figure 10 As shown, the root node of the prefix tree 600 corresponds to the root path node / in the routing node information, the first leaf node of the prefix tree 600 corresponds to the first sub-path information dws in the routing node information, the second leaf node of the prefix tree 600 corresponds to the second sub-path information test1.db, test2.db, and test3.db in the routing node information, and the terminal leaf node of the prefix tree 600 corresponds to the routing node information table1, table2, and table3.

[0097] The management node serializes the prefix tree, converting it into a transmittable format. Figure 10 The serialization result obtained by the prefix tree serialization process shown is ( / , (dws((test1.db, (table1, table2,)), (test2.db, (table3.db))). The management node can send the serialization result to the client.

[0098] In some examples, since the management node can determine the routing node information multiple times and continuously based on the path feature information, when the management node sends the routing node to the client, it can use an asynchronous transmission method to continuously send the routing node information to the client. This allows the client to receive the routing node information only once, which can improve the performance of the client accessing metadata.

[0099] Because a prefix tree is a multi-way tree structure that enables fast retrieval, supports both search and insertion operations, and can merge nodes at the same level, it reduces storage space by utilizing common string prefixes. Therefore, it can reduce the query time for routing node information and improve query efficiency.

[0100] In one possible implementation, the management node determines the position of each routing node's information on the hash ring by performing consistent hashing on the information. Based on the position of each routing node's information on the hash ring, it determines the mapping relationship between each routing node's information and the metadata nodes. The metadata nodes satisfy the consistent hash function and are distributed on the hash ring.

[0101] In this context, the first metadata node adjacent to the routing node information on the hash ring in a clockwise direction is the metadata node corresponding to the routing node information; the mapping relationship is used to verify the client's access request for the target metadata node.

[0102] For example, in a distributed file system, metadata nodes satisfy a consistent hash function, ensuring their even distribution across the hash ring. The management node determines the position of each routing node's information on the hash ring by performing consistent hashing on the routing node information, thus establishing the mapping between routing node information and metadata nodes. Specifically, the first metadata node adjacent to the routing node information in a clockwise direction on the hash ring corresponds to that routing node. This mapping is used to validate client requests for access to the target metadata node.

[0103] for example, Figure 11 This illustration shows a mapping relationship between metadata nodes and routing nodes provided in an embodiment of this application. Figure 11 As shown, the distributed file system uses a consistent hash function to ensure that metadata node 1, metadata node 2, metadata node 3, and metadata node 4 are evenly distributed on the hash ring. Routing nodes are the M-th level sub-path nodes in the routing node information; specifically, routing node A is table1, routing node B is table2, routing node C is table3, and routing node D is table4.

[0104] The management node takes the serialized result of the prefix tree as a parameter, calls the metadata node, and uses the consistent hash function rule: mds(n) = consistent_hash_ring(router_node) to evenly distribute the routing nodes in the prefix tree on the hash ring formed by the metadata nodes. Specifically, it uses the string of the routing node to calculate the hash value and determine its position on the hash ring.

[0105] Here, mds represents metadata nodes; node represents routing nodes. That is, on the hash ring, the first metadata node adjacent to the routing node information in a clockwise direction is the metadata node corresponding to that routing node information. For example, the metadata node corresponding to routing node B is metadata node 3.

[0106] As described above, the management node utilizes path feature information from the data warehouse metadata service to create routing node information. The management node then uses this routing node information for unified directory management of metadata, employing a prefix tree for fast indexing. This improves the growth capacity of metadata and enhances the ability to access metadata concurrently, thereby increasing the efficiency of metadata management.

[0107] The following section introduces methods for metadata management from the perspective of client-side metadata access. Figure 12 This illustration shows a flowchart of a metadata management method provided in an embodiment of this application. This metadata management method can be applied to… Figure 8 The distributed file system 500 shown includes a computing device 110. Client 210 runs on computing device 110. The process includes the following steps:

[0108] S501, the client receives routing node information sent by the management node.

[0109] The management node is a node in the distributed file system. The distributed file system also includes metadata nodes. Metadata nodes are used to store metadata. Routing node information is generated by the management node based on path characteristic information. Path characteristic information is created by the data warehouse metadata service and is used to represent the directory hierarchy that the metadata nodes follow when storing metadata.

[0110] For example, path feature information includes a storage path used to represent metadata when it is stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node.

[0111] In one possible implementation, the client receives the serialized result sent by the management node; the client then deserializes the serialized result to obtain the prefix tree. Based on the prefix tree, the client retrieves the routing node information.

[0112] The serialization result is obtained by serializing the prefix tree, and the serialization result supports transmission over the network. The root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M level child path nodes.

[0113] For example, if a client receives a serialized result ( / , (dws((test1.db, (table1, table2,)), (test2.db, (table3.db)))) sent by the management node, the client can deserialize the received serialized result to restore it to its original form. Figure 10The prefix tree shown is used to obtain routing node information.

[0114] S502, the client obtains an access request.

[0115] The access request includes access path information.

[0116] For example, the access path information includes a root path node in a storage path and at least K level sub-path nodes under the root path node; where N, M, and K are positive integers greater than 1, and N is greater than or equal to M.

[0117] It should be noted that the client can generate an access request by receiving an operation request from the user; the client can also receive access requests sent by the user. The methods described above for the client to obtain access requests are for illustrative purposes only and do not constitute a limitation.

[0118] S503: The client determines the target metadata node based on the access path information and routing node information.

[0119] In this embodiment of the application, the client needs to determine the target metadata node to be accessed in order to complete the access operation.

[0120] In one possible implementation, the client matches the access path information with the information of each routing node to determine the matching result. If the matching result indicates that the target routing node information is matched according to the access path information, consistent hashing is performed on the target routing node information to obtain its position on the hash ring. Based on the position of the target routing node information on the hash ring, the target metadata node is determined.

[0121] In this system, the target metadata node is the first metadata node adjacent to the target routing node information on the hash ring in a clockwise direction. The metadata nodes in the distributed storage system satisfy a consistent hash function and are distributed across the hash ring.

[0122] For example, metadata nodes in a distributed file system satisfy a consistent hash function, ensuring that metadata nodes are evenly distributed across the hash ring. The client segments the access path information and matches the root path node and at least K levels of child path nodes in the access path information against a prefix tree composed of routing node information. If a target routing node is matched in the prefix tree, the client uses the string corresponding to the target routing node information to perform a consistent hash function to determine the target metadata node.

[0123] For example, if the client receives the access path information as / dws / test1.db / table1 / .tmp, the client will split the access path information into / , dws, test1.db, and .tmp. The client will then match each of these split path information with the prefix tree it received. If the prefix tree processed by the client after deserialization is as follows... Figure 10 As shown, it can be determined that the string corresponding to the target route node information of the access path information / dws / test1.db / table1 / .tmp is table1. The client uses table1 to perform a consistent hash function to determine the target metadata node corresponding to the string table1. The mapping logic of the consistent hash function is: mds(2) = consistent_hash_ring(router_node(table1)). That is, the target metadata node is the first metadata node adjacent to the target route node information on the hash ring in a clockwise direction. Figure 11 As shown, the metadata node corresponding to the string table1 of the routing node is metadata node 2.

[0124] In another possible implementation, if the matching result indicates that no routing node information was matched based on the access path information, the client performs consistent hashing on the root path information in the access path information to obtain the position of the root path information on the hash ring. Based on the position of the root path information on the hash ring, the target metadata node is determined.

[0125] The target metadata node is the first metadata node adjacent to the root path information on the hash ring in a clockwise direction.

[0126] For example, if the client receives the access path information as / dws / testmiss.db / table1 / .tmp, the client will split the access path information into / , dws, testmiss.db, and .tmp. The client will then use the split path information to match the received prefix tree. If the client's deserialized prefix tree is as follows... Figure 10 As shown, it can be determined that the access path information does not match any routing node information. The client can perform consistent hashing on the string corresponding to the root path node in the access path information. That is, the client uses the string " / " to perform consistent hashing to determine the target metadata node corresponding to the string " / ".

[0127] S504, the client performs access operations on the target metadata node.

[0128] In this embodiment of the application, the client's access operations may include adding, deleting, modifying, and querying metadata.

[0129] In one possible implementation, the client receives the access result sent by the management node.

[0130] The access result includes whether the access was successful or failed.

[0131] For example, if the management node detects that a client is accessing a target metadata node, the management node performs a verification operation based on the mapping relationship. This mapping relationship is determined by the management node based on the routing node information. The mapping relationship follows a clockwise direction, with the first metadata node adjacent to the routing node information on the hash ring being the metadata node corresponding to that routing node information. If the target metadata node accessed by the client matches the target metadata node determined by the management node based on the mapping relationship, the management node can determine that the access is correct and sends an access success result to the client. If the target metadata node accessed by the client does not match the target metadata node determined by the management node based on the mapping relationship, the management node can determine that the access has failed and sends an access failure result to the client. By having the management node perform a verification operation based on the mapping relationship when a client accesses a target metadata node, the accuracy of the client's access can be guaranteed.

[0132] As described above, when a client accesses a metadata node, it only needs to match the access path information with the route node information cached on the client to determine the target route node information, and then determine the target metadata node, which is consistent with the above. Figure 7 Compared to the method described above, this method eliminates the need for a routing service, thereby improving the performance of concurrent access to metadata.

[0133] In summary, this application provides a metadata management method applied to a management node in a distributed storage system. The distributed system also includes metadata nodes for storing metadata. The management node obtains path feature information from a data warehouse metadata service and generates routing node information based on this path feature information. The routing node information is used by the client to determine the target metadata node corresponding to an access request, and the access request is used to perform access operations on the target metadata node. The path feature information is created by the data warehouse metadata service and represents the directory hierarchy followed by the metadata node when storing metadata. The management node sends the routing node information to the client, which obtains the access request; the access request includes access path information. The client determines the target metadata node based on the access path information and the routing node information and performs access operations on the target metadata node. The above method determines routing node information using path feature information from the data warehouse metadata service and sends the routing node information to the client. When the client obtains an access request, it determines the target metadata node to be accessed based on the target routing node information and performs access operations on the target metadata node. It can not only increase the capacity of metadata management, but also improve the performance of concurrent access to metadata; it can balance the capacity of metadata management and the performance of concurrent access, thereby improving the efficiency of metadata management.

[0134] The foregoing primarily describes the solutions of the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the metadata management device includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0135] This application embodiment can divide the metadata management device into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0136] For example, Figure 13A schematic diagram of the structure of a metadata management device 800 provided in an embodiment of this application is shown. Figure 13 As shown, this metadata management device can be applied to a management node. The metadata management device 800 includes:

[0137] The acquisition module 801 is used to obtain path feature information from the data warehouse metadata service. The path feature information is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata.

[0138] Processing module 802 is used to determine routing node information based on path feature information.

[0139] The sending module 803 is used to send routing node information to the client; the routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform access operations on the target metadata node.

[0140] In one possible implementation, the processing module 802 is further configured to perform structured processing on the routing node information to obtain a prefix tree; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M level child path nodes; the prefix tree is serialized to obtain the serialization result corresponding to the prefix tree; the serialization result supports transmission over the network.

[0141] In one possible implementation, processing module 802 is further configured to determine the position of each routing node information on the hash ring by performing consistent hashing on the routing node information; metadata nodes satisfy the consistent hash function and are distributed on the hash ring. Based on the position of each routing node information on the hash ring, the mapping relationship between each routing node information and the metadata nodes is determined. Specifically, the first metadata node adjacent to the routing node information on the hash ring in a clockwise direction is the metadata node corresponding to the routing node information; the mapping relationship is used to verify client access requests for the target metadata node.

[0142] In one possible implementation, the sending module 803 is also used to send the serialization result to the client.

[0143] It should be noted that the aforementioned metadata management device 800 can be applied to Figure 8 The distributed file system architecture diagram shown in Figure 500 is located in server 120.

[0144] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, explanations of any of the metadata management devices 800 provided above, as well as descriptions of their beneficial effects, can be found in the above descriptions. Figure 9 The corresponding method implementations will not be described in detail.

[0145] For example, Figure 14 A schematic diagram of the structure of a metadata management device 1000 provided in an embodiment of this application is shown. Figure 14 As shown, this metadata management device can be applied to a client, and the metadata management device 1000 includes:

[0146] The receiving module 1001 is used to receive routing node information sent by the management node. The management node is a node in the distributed file system, which also includes a metadata node for storing metadata. The routing node information is generated by the management node based on path feature information. The path feature information is created by the data warehouse metadata service and is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata.

[0147] The acquisition module 1002 is used to acquire access requests; the access request includes access path information.

[0148] Processing module 1003 is used to determine the target metadata node based on access path information and routing node information.

[0149] Access module 1004 is used to access the target metadata node.

[0150] In one possible implementation, the processing module 1003 is further configured to match the access path information with the information of each routing node to determine the matching result. If the matching result indicates that the target routing node information is matched according to the access path information, the target routing node information is subjected to consistent hashing to obtain the position of the target routing node information on the hash ring. Based on the position of the target routing node information on the hash ring, the target metadata node is determined; wherein, the target metadata node is the first metadata node adjacent to the target routing node information on the hash ring in a clockwise direction; the metadata nodes in the distributed storage system satisfy the consistent hash function and are distributed on the hash ring.

[0151] In one possible implementation, the processing module 1003 is further configured to, if the matching result indicates that no routing node information is matched based on the access path information, perform consistent hashing on the root path information in the access path information to obtain the position of the root path information on the hash ring. Based on the position of the root path information on the hash ring, the target metadata node is determined; wherein, the target metadata node is the first metadata node adjacent to the root path information on the hash ring in a clockwise direction.

[0152] In one possible implementation, the receiving module 1001 is further configured to receive the serialization result sent by the management node. The serialization result is obtained by serializing the prefix tree, and the serialization result supports transmission over the network; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to at least M levels of child path nodes. The serialization result is deserialized to obtain the prefix tree. Based on the prefix tree, the routing node information is obtained.

[0153] It should be noted that the aforementioned metadata management device 1000 can be applied to... Figure 8 The distributed file system architecture diagram 500 shows computing device 110.

[0154] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, explanations of any of the metadata management devices 800 provided above, as well as descriptions of their beneficial effects, can be found in the above descriptions. Figure 12 The corresponding method implementations will not be described in detail.

[0155] Figure 15 A schematic diagram of a computing device according to an embodiment of this application is shown. Figure 15 As shown, the computing device 110 includes a bus 901, a processor 902, a memory 903, and a communication interface 904. The processor 902, the memory 903, and the communication interface 904 communicate with each other via the bus 901. The computing device 110 can be a terminal device such as a server, desktop computer, laptop computer, or smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 110.

[0156] The 901 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 15 The bus 901 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 901 may include a path for transmitting information between various components of the computing device 110 (e.g., processor 902, memory 903, communication interface 904).

[0157] Processor 902 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0158] Memory 903 may include volatile memory, such as random access memory (RAM). Processor 902 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0159] The memory 903 stores executable program code, and the processor 902 executes the executable program code to implement the following respectively. Figure 13 or Figure 14 The module shown demonstrates its functionality, thereby enabling metadata management. In other words, the memory 903 stores instructions for executing the metadata management method.

[0160] Alternatively, the memory 903 stores executable code, which the processor 902 executes to implement the functions of the aforementioned metadata management device 800 or metadata management device 1000, thereby implementing the metadata management method. That is, the memory 903 stores instructions for executing the metadata management method.

[0161] The communication interface 904 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 110 and other devices or communication networks.

[0162] It should be noted that computing devices running management nodes and metadata nodes are different from computing devices running clients.

[0163] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, specifically a central server, an edge server, or a local server in a local data center.

[0164] Figure 16 A schematic diagram of a computing device cluster provided in an embodiment of this application is shown. Figure 16As shown, the computing device cluster includes at least one computing device 110. The memory 903 of one or more computing devices in the computing device cluster may store the same instructions for performing methods of metadata management.

[0165] In some possible implementations, the memory 903 of one or more computing devices in the computing device cluster may also store partial instructions for executing metadata management methods. In other words, a combination of one or more computing devices 110 can jointly execute instructions for executing metadata management methods.

[0166] It should be noted that the memory 903 in different computing devices 110 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the metadata management device. That is, the instructions stored in the memory 903 of different computing devices 110 can implement... Figure 13 The functionality of one or more modules within it.

[0167] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 17 This illustrates one possible cluster implementation method provided by an embodiment of this application. For example... Figure 17 As shown, the two computing devices 110A and 110B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 903 in computing device 110A stores instructions for performing the functions of the sending module 803. Meanwhile, the memory 903 in computing device 110B stores instructions for performing... Figure 13 Instructions for obtaining the functions of module 801 and processing module 802.

[0168] Figure 17 The connection method between the computing device clusters shown can be considered in light of the metadata management method provided in this application (e.g., large-scale data storage), therefore considering... Figure 13 The functions implemented by the acquisition module 801 and the processing module 802 are handed over to the computing device 110B for execution.

[0169] It should be understood that Figure 17 The functions of the computing device 110A shown can also be performed by multiple computing devices 110. Similarly, the functions of the computing device 110B can also be performed by multiple computing devices 110.

[0170] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 16 and Figure 17The connection method of the computing device cluster shown is different in that the memory 903 of one or more computing devices 110 in this computing device cluster can store the same instructions for executing metadata management methods.

[0171] In some possible implementations, the memory 903 of one or more computing devices 110 in the computing device cluster may also store partial instructions for executing metadata management methods. In other words, a combination of one or more computing devices 110 can jointly execute instructions for executing metadata management methods.

[0172] It should be noted that the memory 903 in different computing devices 110 within the computing device cluster can store different instructions for executing some functions of the distributed file system. That is, the instructions stored in the memory 903 of different computing devices 110 can implement the functions of a metadata management device.

[0173] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform... Figures 9-11 The operation of any corresponding implementation scheme and its various feasible implementation methods.

[0174] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform... Figure 12 The corresponding implementation plan and the operation of its various feasible implementation methods.

[0175] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform... Figures 9-11 The operation of any corresponding implementation scheme and its various feasible implementation methods.

[0176] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform... Figure 12 The corresponding implementation plan and the operation of its various feasible implementation methods.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions, which, when loaded and executed on a computer, generate all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one network site, computer, server, or data center to another network site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or it can be a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape, etc.), an optical medium (e.g., DVD, etc.), or a semiconductor medium (e.g., solid-state drive), etc.

[0180] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A metadata management method, characterized in that, The method is applied to the management node in a distributed file system, and the distributed storage system further includes a metadata node for storing metadata; including: Path feature information is obtained from the data warehouse metadata service. The path feature information is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata. Based on the path feature information, determine the routing node information; The routing node information is sent to the client; the routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform an access operation on the target metadata node.

2. The method according to claim 1, characterized in that, The path feature information includes a storage path used to represent the metadata when it is stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node; Where N and M are positive integers greater than 1, and N is greater than or equal to M.

3. The method according to claim 2, characterized in that, The at least N-level sub-path nodes include a first sub-path node under the root path node and a second sub-path node under the first sub-path node; the root path node is used to represent the database corresponding to the metadata node; the first sub-path node is used to represent a partition in the database; the first sub-path node is used to represent a table in the partition.

4. The method according to any one of claims 2-3, characterized in that, Before sending the routing node information to the client, the method further includes: The routing node information is structured to obtain a prefix tree; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to the at least M level of child path nodes. The prefix tree is serialized to obtain the serialized result corresponding to the prefix tree; the serialized result supports transmission over a network. Sending the routing node information to the client includes: The serialization result is sent to the client.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The position of each routing node information on the hash ring is determined by performing consistent hashing on the routing node information; the metadata nodes satisfy the consistent hash function and are distributed on the hash ring. Based on the position of each routing node information on the hash ring, determine the mapping relationship between each routing node information and the metadata node; Wherein, in a clockwise direction, the first metadata node on the hash ring adjacent to the routing node information is the metadata node corresponding to the routing node information; the mapping relationship is used to perform a verification operation on the client's access request for the target metadata node.

6. A metadata management method, characterized in that, The method is executed by the client and includes: The system receives routing node information sent by a management node; the management node is a node in a distributed file system, which also includes a metadata node for storing metadata; the routing node information is generated by the management node based on path feature information; the path feature information is created by the data warehouse metadata service and is used to represent the directory hierarchy followed by the metadata node when storing metadata; Obtain an access request; the access request includes access path information; Based on the access path information and the routing node information, the target metadata node is determined; Perform access operations on the target metadata node.

7. The method according to claim 6, characterized in that, The step of determining the target metadata node based on the access path information and the routing node information includes: The access path information is matched with the routing node information to determine the matching result; If the matching result indicates that target route node information is matched according to the access path information, the target route node information is subjected to consistent hashing to obtain the position of the target route node information on the hash ring; Based on the position of the target routing node information on the hash ring, the target metadata node is determined; wherein, the target metadata node is the first metadata node on the hash ring adjacent to the target routing node information in a clockwise direction; the metadata nodes in the distributed storage system satisfy a consistent hash function and are distributed on the hash ring.

8. The method according to claim 7, characterized in that, The step of determining the target metadata node based on the access path information and the routing node information further includes: If the matching result indicates that no routing node information is matched according to the access path information, a consistent hashing process is performed on the root path information in the access path information to obtain the position of the root path information on the hash ring. Based on the position of the root path information on the hash ring, the target metadata node is determined; wherein, the target metadata node is the first metadata node on the hash ring adjacent to the root path information in a clockwise direction.

9. The method according to any one of claims 6-8, characterized in that, The path feature information includes a storage path used to represent the metadata when it is stored in a metadata node; a storage path includes a root path node and at least N levels of sub-path nodes under the root path node; a routing node information includes a root path node in a storage path and at least M levels of sub-path nodes under the root path node; The access path information includes a root path node in a storage path and at least K level sub-path nodes under the root path node; wherein N, M, and K are positive integers greater than 1, and N is greater than or equal to M.

10. The method according to claim 6, characterized in that, The receipt of routing node information sent by the management node includes: Receive the serialization result sent by the management node; The serialization result is obtained by serializing the prefix tree, and the serialization result supports transmission over the network; the root node of the prefix tree corresponds to the root path node in the routing node information; each level of child nodes under the root node in the prefix tree corresponds to the at least M level of child path nodes. The serialization result is deserialized to obtain the prefix tree; Based on the prefix tree, obtain the routing node information.

11. A metadata management device, characterized in that, The device is applied to the management node and includes: The acquisition module is used to acquire path feature information from the data warehouse metadata service. The path feature information is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata. The processing module is used to determine the routing node information based on the path feature information; The sending module is used to send the routing node information to the client; the routing node information is used by the client to determine the target metadata node corresponding to the access request, and the access request is used to perform an access operation on the target metadata node.

12. A metadata management device, characterized in that, The device is applied to a computing device and includes: A receiving module is used to receive routing node information sent by a management node; the management node is a node in a distributed file system, which also includes a metadata node for storing metadata; the routing node information is generated by the management node based on path feature information; the path feature information is created by a data warehouse metadata service and is used to represent the directory hierarchy relationship followed by the metadata node when storing metadata; The acquisition module is used to acquire access requests; the access requests include access path information. The processing module is used to determine the target metadata node based on the access path information and the routing node information; The access module is used to perform access operations on the target metadata node.

13. A distributed file system, characterized in that, The distributed file system includes a management node and a metadata node. The management node is used to execute the metadata management method as described in any one of claims 1-5. The metadata node is used to store metadata.

14. A server, characterized in that, The server includes: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the instructions, causing the server to perform the metadata management method as described in any one of claims 1-5.

15. A computing device, characterized in that, The computing device includes: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions, causing the computing device to perform the metadata management method as described in any one of claims 6-10.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a management node, cause the management node to perform the metadata management method as described in any one of claims 1-5.

17. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the metadata management method as described in any one of claims 6-10.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions, which, when executed by a management node, enable the management node to perform the metadata management method as described in any one of claims 1-5.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed by a computing device, enable the computing device to perform the metadata management method as described in any one of claims 6-10.

Images