Systems, methods, and media for metadata routing in a distributed system

By distributing metadata databases on multiple metadata nodes and using partition mapping information and distributed cache, the problem of difficulty in finding the correct nodes by clients is solved, efficient request routing and dynamic system expansion is achieved, and system performance and capability are improved. Extensibility.

CN113508372BActive Publication Date: 2024-12-03HITACHI VANTARA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980093357.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-03-04
Publication Date
2024-12-03
Estimated Expiration
2039-03-04

AI Technical Summary

Technical Problem

In a multi-partitioned database, it is difficult for clients to determine which node is responsible for the specific metadata, resulting in too many network requests, resulting in network congestion, limiting system scalability and affecting system reconfiguration flexibility.

Method used

By distributing metadata databases on multiple metadata nodes and determining the correct metadata nodes using partition mapping information, optimize the client's ability to find partition leaders, and adopt distributed caching and near-caching mechanisms to improve efficiency and performance.

Benefits of technology

Request routing for highly available scalable distributed metadata databases is implemented, dynamically scalable and shrink partitions are expanded, ensuring that clients can efficiently find the correct nodes, reduce the number of network jumps, and improve system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113508372B_ABST
    Figure CN113508372B_ABST
Patent Text Reader

Abstract

In some examples, the first computing device may receive partition mapping information indicating partitions of the metadata database from the second computing device. The first computing device is capable of communicating with a plurality of metadata nodes, each metadata node maintaining a portion of the metadata database based on partitioning the metadata database to distribute the metadata database across a plurality of metadata nodes. The first computing device may determine to send a request to the metadata database based at least on the keyword information. The first computing device may determine a first metadata node among the plurality of metadata nodes based on the partition mapping information, the first metadata node being indicated as maintaining a partition of the metadata database corresponding to the keyword information. The first computing device may send a request to perform a database operation to the first metadata node based on the partition mapping information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data storage. Background Art

[0002] A multi-partitioned database can provide scalability by dividing metadata services among multiple computing devices (also called "nodes"). For example, metadata can be replicated across multiple nodes to provide high availability and fault tolerance of metadata. However, it may be difficult for a client seeking to access metadata to determine which node is responsible for the metadata that the client is trying to locate. In some cases, this may result in excessive network requests to find the correct node to access, which may cause network congestion, limit system scalability, affect the flexibility of system reconfiguration, etc. Summary of the invention

[0003] Some implementations include a first computing device capable of communicating with a plurality of metadata nodes, each metadata node maintaining a portion of a metadata database based on partitioning the metadata database to distribute the metadata database across a plurality of metadata nodes. In some examples, the first computing device may receive partition mapping information indicating partitions of the metadata database from a second computing device. The first computing device may determine to send a request to the metadata database based at least on the keyword information. The first computing device may determine a first metadata node among the plurality of metadata nodes based on the partition mapping information, the first metadata node being indicated as maintaining a partition of the metadata database corresponding to the keyword information. The first computing device may send a request to perform a database operation to the first metadata node based on the partition mapping information. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is set forth with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items or features.

[0005] Figure 1 An example architecture of a system capable of storing data and metadata according to some implementations is shown.

[0006] Figure 2 is a block diagram illustrating an example logical configuration of a system according to some implementations.

[0007] Figure 3 is a block diagram illustrating an example configuration of metadata node partitioning according to some implementations.

[0008] Figure 4 is a block diagram illustrating example data structures according to some implementations.

[0009] Figure 5 is a block diagram illustrating an example configuration of multiple metadata nodes according to some implementations.

[0010] Figure 6 is a block diagram illustrating an example data structure for a partition map according to some implementations.

[0011] Figure 7 is a block diagram illustrating an example configuration of a metadata node according to some implementations.

[0012] Figure 8 is a block diagram illustrating an example configuration of a system according to some implementations.

[0013] Fig. 9 is a block diagram illustrating an example configuration of a key-space-tree data structure according to some implementations.

[0014] Fig.10 is a flow diagram illustrating an example process for routing metadata requests according to some implementations.

[0015] Fig.11 Shown are selected example components of a service computing device that may be used to implement at least some of the functionality of the systems described herein. DETAILED DESCRIPTION

[0016] Some implementations herein relate to techniques and arrangements for distributed computer systems where partition information is not stored or controlled by a single component or table in the system. In some examples, partition information such as key space and connection information may be provided to client applications as partitions come online. In addition, client applications may have multiple fallback mechanisms for locating the correct node to contact, thereby preventing any single point of failure. Additionally, having client applications involved in partition information may enable optimal routing of data requests.

[0017] Some examples herein include a system capable of routing requests to a highly available, scalable, distributed metadata database. The metadata database herein can provide high availability by maintaining highly consistent copies of metadata on separate metadata nodes. Additionally, the metadata database provides scalability by partitioning the metadata and distributing the metadata across different metadata nodes. Each partition can have a single metadata node that serves as a partition leader. The partition leader can be responsible for servicing all requests for that particular partition. Additionally, the scheme herein optimizes the ability of a client application to find a partition leader for a given request.

[0018] As described above, the examples in this article make metadata highly available by storing highly consistent copies of metadata in a distributed configuration on multiple metadata nodes. In some cases, the consistency of the distributed metadata database is maintained using the Raft consensus algorithm. In the Raft algorithm, one node can be elected as a leader and can be responsible for serving all read and write operations. Other nodes are follower nodes, which receive copies of all transactions so that they can update their own metadata database information. If the leader node fails, one of the follower nodes can be elected as a leader and can take over to serve read and write transactions. The client nodes of the metadata system here are able to discover which node is the raft leader and direct requests to that node. If the leader of a partition should change, the client is automatically routed to the new leader.

[0019] In some examples, each client can maintain a near cache of partition information. In addition, the system can include a distributed cache that can publish updates to the distributed cache to the clients. In addition, the nodes here can provide hints to the clients indicating where the data is located. In addition, to maintain efficiency, the number of distributed caches may be a small number compared to the total number of nodes in the system. On the other hand, in some examples, a near cache can be included on each client in the system to improve the performance of the client.

[0020] As an example, suppose the system receives a user request to retrieve data from a user device. In this case, the user request may be routed to the near cache first. During steady state conditions, the client can usually get the correct answer in the near cache as to which node is able to serve the user request. In this case, the client can achieve optimal efficiency by being able to communicate directly with the correct node. For example, when a node is responsible for serving a partition of data, the node can notify the distributed cache of the new routing information for the data managed by the node. The distributed cache can then publish this information to the client for storage in the corresponding near cache.

[0021] In some cases, the near cache of a client receiving a user request may have a stale mapping. Therefore, when a user request is routed to a node specified in a stale near cache, the client may receive a failed response from the node. The failed response can trigger a discovery process and may also include a hint as to which node to check next to respond to the user request. If the failure continues for a threshold number of attempts, the client may fall back to querying each node individually. The performance resulting from polling all nodes may be very poor, but allows the client to determine the correct node and satisfy the user request in a worst-case scenario. Therefore, the implementation herein provides a system with a distributed metadata database that can be dynamically expanded and contracted, such as by enabling partitions to be dynamically added and removed as data grows or shrinks. The system may include a set of hierarchical caches that cache system metadata and use a reactive cache update mechanism to achieve efficiency in implementing large geographically distributed databases distributed across multiple physical and / or logical nodes.

[0022] For discussion purposes, some example implementations are described in the context of one or more service computing devices communicating with a cloud storage system for managing storage of data using a distributed metadata database. However, the implementations herein are not limited to the specific examples provided and may be extended to other types of computing system architectures, other types of storage environments, other types of user configurations, other types of data, etc., as will be apparent to those skilled in the art based on the disclosure herein.

[0023] Figure 1 An example architecture of a system 100 capable of storing data and metadata according to some implementations is shown. The system 100 includes a plurality of service computing devices 102 that can communicate or otherwise couple with at least one network storage system 104, such as via one or more networks 106. In addition, the service computing devices 102 can communicate with one or more user computing devices 108 and one or more administrator devices 110 via one or more networks 106, which can be any of a variety of types of computing devices, as discussed further below.

[0024] In some examples, the service computing device 102 may include one or more servers that may be implemented in any number of ways. For example, the programs, other functional components, and at least a portion of the data storage of the service computing device 102 may be implemented on at least one server, such as in a server cluster, server farm, data center, cloud hosting computing service, etc., although other computer architectures may additionally or alternatively be used. Fig.11 Additional details of serving computing device 102 are discussed.

[0025] The service computing device 102 may be configured to provide storage and data management services to users 112. As a few non-limiting examples, users 112 may include users performing functions for businesses, enterprises, organizations, government entities, academic entities, etc., and which in some examples may include storage of very large amounts of data. However, implementations herein are not limited to any particular use or application of the system 100 and other systems and arrangements described herein.

[0026] In some examples, one or more network storage systems 104 may be referred to as “cloud storage” or “cloud-based storage,” and in some cases may enable lower cost per megabyte / gigabyte storage solutions than local storage available at the service computing device 102. Furthermore, in some examples, one or more network storage systems 104 may include commercially available cloud storage known in the art, while in other examples, one or more network storage systems 104 may include private or enterprise storage systems, or a combination thereof, that are accessible only to entities associated with the service computing device 102.

[0027] The one or more networks 106 may include any suitable network, including: a wide area network, such as the Internet; a local area network (LAN), such as an intranet; a wireless network, such as a cellular network, a local wireless network (e.g., Wi-Fi), and / or a short-range wireless communication (e.g., ); wired networks, including Fibre Channel, optical fiber, Ethernet, or any other such network, direct wired connections, or any combination thereof. Thus, one or more networks 106 may include wired and / or wireless communication technologies. The components used for such communications may depend, at least in part, on the type of network, the selected environment, or both. Protocols for communicating over such networks are well known and will not be discussed in detail herein. Thus, the service computing device 102, one or more network storage systems 104, user devices 108, and management devices 110 are able to communicate over one or more networks 106 using wired or wireless connections and combinations thereof.

[0028] Additionally, the service computing devices 102 can communicate with each other via one or more networks 107. In some cases, the one or more networks 107 may be a LAN, a private network, etc., while in other cases, the one or more networks 107 may include any of the networks 106 discussed above.

[0029] Each user device 108 can be any suitable type of computing device, such as a desktop, laptop, tablet computing device, mobile device, smart phone, wearable device, terminal and / or any other type of computing device capable of sending data over a network. Users 112 can be associated with user devices 108, such as through corresponding user accounts, user login credentials, etc. In addition, user devices 108 can communicate with one or more service computing devices 102 through one or more networks 106, through a separate network, or through any other suitable type of communication connection. Many other variations will be clear to those skilled in the art having the benefit of the disclosure herein.

[0030] In addition, each user device 108 may include a respective instance of a user application 114 that may be executed on the user device 108, e.g., for communicating with a user web application 116 that may be executed on one or more service computing devices 102, e.g., for sending user data for storage on one or more network storage systems 104 and / or receiving stored data from one or more network storage systems 104 via data requests 118, etc. In some cases, the application 114 may include a browser or may be operated via a browser, while in other cases, the application 114 may include any other type of application having communication functionality for communicating with the user web application 116 via one or more networks 106.

[0031] In the system 100, users 112 may store data to and receive data from one or more service computing devices 102 with which their respective user devices 108 communicate. Thus, the service computing devices 102 may provide local storage for the users 112 and the respective user devices 108. During steady state operation, there may be users 108 in regular communication with the service computing devices 102.

[0032] Additionally, the administrator device 110 may be any suitable type of computing device, such as a desktop, laptop, tablet computing device, mobile device, smartphone, wearable device, terminal, and / or any other type of computing device capable of sending data over a network. The administrator 120 may be associated with the administrator device 110, such as through a corresponding administrator account, administrator login credentials, etc. Additionally, the administrator device 110 may be capable of communicating with one or more service computing devices 102 over one or more networks 106, over a separate network, or over any other suitable type of communication connection.

[0033] In addition, each administrator device 110 may include a respective instance of an administrator application 122 that may be executed on the administrator device 110, such as for communicating with a management network application 124 that may be executed on one or more service computing devices 102, such as for sending management instructions to manage the system 100, and for sending management data for storage on one or more network storage systems 104 and / or receiving stored management data from one or more network storage systems 104, such as via management requests 126, etc. In some cases, the administrator application 122 may include a browser or may be operated via a browser, while in other cases, the administrator application 122 may include any other type of application having communication functionality to enable communication with the management network application 124 via one or more networks 106.

[0034] The service computing device 102 may execute a storage program 130 that may provide a gateway to the one or more network storage systems 104, e.g., for sending data to be stored to the one or more network storage systems 104 and for retrieving requested data from the one or more network storage systems 104. In addition, the storage program 142 may manage data stored by the system 100, e.g., for managing data retention periods, data protection levels, data replication, etc.

[0035] The service computing device 102 may also include a metadata database (DB) 132, which may be divided into a plurality of metadata DB parts 134(1)-134(N) and may be distributed across a plurality of service computing devices 102. For example, the metadata DB 132 may be used to manage object data 136 stored at one or more network storage systems 104. The metadata DB 132 may include a number of metadata about the object data 136, such as information about each object, how to access each object, storage protection level of the object, storage retention period, object owner information, object size, object type, etc. In addition, a DB management program 138 may manage and maintain the metadata DB 132, such as for updating the metadata DB 132 as new objects are stored, old objects are deleted, objects are migrated, etc.

[0036] Additionally, the service computing device 102 may include a distributed cache 144 that may track the location of the metadata DB portions 134, e.g., to track the specific service computing device on which the metadata DB portions are stored, which partition each portion is included in, etc. In some cases, the DB management program 138 may update the distributed cache as the metadata DB 132 is updated, migrated to a different service computing device 102, etc.

[0037] The information in the distributed cache 144 may be used to update the near cache 146, which may be maintained on certain of the service computing devices 146 that may execute applications that provide client services (e.g., the user web application 116 and the management web application 124). For example, the near cache may be updated periodically based on updates to the distributed cache 144. Thus, for example, when the user web application 116 receives a data request 118 from a user device 108, the user web application may access the near cache 146 to determine which service computing device 102 to communicate with to access the desired metadata DB portion 134 to perform the data request 118. Thus, by using the near cache 146, the user web application is able to reduce the number of queries used to obtain the required information from the metadata DB 132 to perform the data request. Additional details of the distributed cache 144 and the near cache 146 are discussed below.

[0038] In addition, the service computing device may include an asynchronous manager 148 that may be executed to perform updates to the metadata DB 132 and / or the object data 136. In some cases, the asynchronous manager 148 may be a module of the storage program 130, while in other cases, the asynchronous manager 148 may be a separate program. The asynchronous manager 148 may maintain an update queue or other data structure for data storage and management actions to be performed asynchronously in the system 100. For example, instead of having to perform certain storage operations and report back to the user after the operation is completed, the asynchronous manager 148 may continuously save the required operations to a queue that may be configured to guarantee that the operations will be performed, and then respond to the user without delays that may otherwise include delays for certain types of operations. The asynchronous manager 148 may then asynchronously perform the operations in the update queue as background operations, etc.

[0039] In some cases, the service computing devices 102 may be arranged at the sites 150 into one or more groups, clusters, systems, etc. In some cases, multiple sites 150 may be geographically dispersed from one another, e.g., to provide data replication, disaster recovery protection, etc. Furthermore, in some cases, the service computing devices 102 at multiple different sites 150 may be configured to securely communicate with one another, e.g., to provide federation of multiple sites 150.

[0040] In some examples (not in Figure 1), some service computing devices 102 may be arranged in pairs, where a first service computing device 102 may be coupled to a second service computing device 102, such that together they form a computing node for providing storage and data management services to a plurality of user devices 108. For example, a first service computing device 102 may act as a primary computing device, while a second service computing device 102 may act as a secondary computing device, e.g., for backup, failover, etc. Furthermore, as discussed further below, in some cases, a plurality of service computing devices 102 may be configured in a raft configuration to manage partitioned data, e.g., to provide redundancy of a metadata DB at a plurality of computing devices. Many other configurations will be apparent to those skilled in the art having the benefit of the disclosure herein.

[0041] Figure 2 200 is a block diagram showing an example logical configuration of a system 200 according to some implementations. In some examples, the system 200 may correspond to any of the system 100 discussed above or various other possible computing system architectures, which will be clear to those skilled in the art who benefit from the disclosure herein. The system 200 may implement distributed object storage and may include using a network application as a front end for users and administrators. In some cases, the system 200 may store objects in buckets that may be created by users on a network storage. The system 200 may use resources distributed on locally deployed (on-premises) systems and cloud systems to implement complex management and storage of data. In the system 200, scalability may be provided by logically partitioning stored data stored on geographic regions. As discussed further below, the system 200 includes efficient processing for determining the location of desired metadata, such as for responding to user or system requests.

[0042] In addition, the system 200 may provide a distributed object store that employs a highly available, scalable, and distributed metadata DB 132 to track metadata for objects or other data stored by the system 200, for example, at one or more network storage systems 104. The system 200 includes a tiered cache to enable access to individual metadata nodes 202 of the metadata DB 132 to efficiently read and write metadata stored in the metadata DB 132. In this example, the metadata DB 132 includes metadata nodes 202(1), 202(2), 202(3), ..., where each metadata node may include one or more service computing devices 102 that maintain a metadata DB portion 134, as described above with respect to FIG. Figure 1 As discussed.

[0043] The system 200 also includes a user web application 116, an administrative web application 124 that provide administration and data access services to users and administrators, respectively. For example, the user web application 116 may include a client module 204 and the administrative web application 124 may include a client module 206, each of which is configured to interact with other nodes in the system 200. For example, the client modules 204, 206 may interact with the metadata node 202 to retrieve metadata, may interact with the distributed cache node to receive cache information to update the associated near cache 146(1) or 146(2), respectively, and may interact with the storage program 136, for example, to retrieve data 210 based on the retrieved metadata.

[0044] The system 200 may also include an asynchronous manager 148, which may also include a client module 212 that can communicate with the metadata nodes 202, distributed cache nodes 208, and / or storage program 136 while the asynchronous manager 148 is performing various asynchronous services, such as processing, maintaining, and responding to data and metadata in the system. In some cases, the client 212 can effectively connect to the metadata database 132 based on accessing the associated near cache 146(3) and read from / write to the metadata database 132.

[0045] Additionally, in some cases, the distributed cache node 208 may include a registered listener 214. As discussed further below, each near cache 146(1), 146(2), and 146(3) may have a registered listener 214 registered with the distributed cache node 208. When updates are made to the distributed cache 144, the associated registered listener 214 may send information about the updates to the corresponding near cache 146. Thus, the registered listeners enable the near cache 146 to stay up to date, thereby reducing the number of incorrect calls to the metadata node by the client modules 204, 206, 212.

[0046] Figure 3is a block diagram showing an example configuration of a metadata node partition 300 according to some implementations. The metadata node 202 here can be configured into multiple partitions 300, each partition 300 is configured to operate according to the raft consensus algorithm to provide redundant backup of the metadata DB portion 134 managed by the metadata node 202. For example, the raft algorithm ensures that each metadata node 202 agrees on the same changes to the metadata DB portion 134. The raft group reaches consensus via an elected leader, for example, the metadata node 202 in the raft group can be a leader or a follower. The leader can be responsible for copying changes to the metadata DB portion 134 to the follower nodes. The leader can periodically notify the followers of its existence by sending a heartbeat message. If the leader does not send a heartbeat message within a threshold time, the followers can elect a new leader from the followers. Therefore, in this example and the examples discussed below, each partition 300 has a single leader metadata node 202 called a "partition leader" that is responsible for servicing all requests for that particular partition. Therefore, one problem solved by the implementation of this article is to optimally determine the correct partition leader to which a given request from a client is sent.

[0047] In the example shown, metadata node 202(1) is the partition leader for partition 300, while metadata nodes 202(2) and 202(3) are followers. As the leader, metadata node 202(1) receives a client request 302 to create a record. In response, metadata node 202(1) creates a new record 304 in the metadata DB portion 134 managed by metadata node 202(1). In addition, based on the raft algorithm, metadata node 202(2) replicates the new record 304 to follower metadata nodes 202(2) and 202(3). Each of these metadata nodes 202(2) and 202(3) adds the new record 304 to the metadata DB portion 134 it manages.

[0048] Also, in this example, it is worth noting that the distinction between "leader" and "follower" is about the partition, not about the metadata node 202 itself. For example, the "leader" metadata node 202(1) in this example is the leader for that partition, but the metadata node 202(1) may also be a follower or a leader for one or more other partitions that the metadata node 202(1) may also manage.

[0049] therefore, Figure 3The configuration provides highly available metadata by making highly consistent copies of the metadata on multiple metadata nodes 202. The distributed metadata DB here uses the established raft protocol to achieve this, in which one or more follower nodes receive a copy of all transactions. If the leader node fails, one of the follower nodes can be elected as the leader node and take over servicing read and write transactions. As discussed further below, clients of the metadata DB are typically able to discover which node is the raft leader and make their requests to that node based on reference to the associated near cache. If the partition leader should change, the distributed cache and near cache are updated so that clients can be routed to the new leader.

[0050] Figure 4 408 and corresponding values ​​410.

[0051] Partitioning of metadata in the metadata DB here (sometimes also referred to as sharding) allows scaling of the metadata DB by dividing the responsibility for metadata areas across different metadata nodes in the system. In this example, partitioning of the metadata DB is achieved by partitioning the key-space (i.e., the set 408 of all possible keys) of each metadata table into key-space-ranges associated with a partition. Each partition is given a partition identifier and a set of key-space-ranges for which the node managing the DB partition is responsible.

[0052] In the example shown, the first partition 402 includes keywords a to e; the second partition 404 includes keywords e to o; and the third partition 406 includes keywords o to z. Thus, the portion of the metadata DB managed by each set of metadata nodes may correspond to the corresponding partition determined using the above techniques. In addition, the partitions may be adjusted periodically based on changes in the amount of data in each partition 402-406, for example, to balance the amount of data managed by each metadata node.

[0053] Figure 5is a block diagram illustrating an example configuration 500 of multiple metadata nodes 202 according to some implementations. In this example, a raft group is formed for each of three partitions 502, 504, and 506 on a subset of metadata nodes 202(1)-202(4). The leader of the raft group is the partition leader and is responsible for servicing all requests to other members of the partition. Each metadata node 202 can be a member of multiple raft groups and therefore a member of multiple partitions. Any metadata node 202 can become the leader of any partition it manages. In this example, metadata node 202(1) manages a first partition 502 as a follower, a second partition 504 as a follower, and a third partition 506 as a follower. Metadata node 202(2) manages the first partition 502 as a leader and the second partition 504 as a follower. Metadata node 202(3) manages the second partition 504 as a leader and the third partition 506 as a follower. Metadata node 202(4) manages the first partition as a follower and the third partition 506 as a leader.

[0054] Clients of the metadata system are able to discover which node is the partition leader for a given metadata DB access request, for example by consulting the near cache (not in Figure 5 ). If the partition leader should change, clients can be routed to the new leader, for example based on update information provided to the near cache, or by other techniques, as discussed further below.

[0055] In some cases, partitions can also be split and merged. Splitting a partition means dividing the key-space-range served by a single partition into two or more new partitions. Splitting a partition may occur when the amount of metadata contained in a partition increases and it is desired to distribute the metadata across a greater number of metadata nodes. On the other hand, merging partitions can include merging the metadata of multiple partitions together into a single partition. The metadata merged into a single partition can have continuous key-space-ranges to simplify access to the partitions subsequent to the merge. Merging partitions can occur when a set of partitions has shrunk to the point where it is more efficient to provide the corresponding metadata in a single partition.

[0056] Figure 66 is a block diagram illustrating an example data structure 600 for partition mapping according to some implementations. In this example, the data structure 600 includes a partition ID 602, a key-space-range 604 corresponding to the partition ID, a leader ID 606 of the leader of the partition, and node IDs 608 of nodes that are members of the partition. As described above, for a given request, a client can determine which partition the request belongs to. To do this, the client can map the key for the request to a key-space-range 604. The client can then determine which partition is responsible for the key-space-range 604 that contains the key. If the key-space-range and / or partitions change, the data structure may need to be updated to enable the client to find the required information, such as in the case of new partitions, merged partitions, etc.

[0057] To communicate partition information between system components, the system herein may use partition map entries as shown in data structure 600. A partition map entry describes enough information for a client to route a request to a partition leader. For example, a partition map entry associates a key-space-range with a system unique identifier of a node that is a partition leader for each key-space-range.

[0058] Figure 7 202 is a block diagram illustrating an example configuration of a metadata node 202 according to some implementations. The metadata node 202 here is an actual database node. The metadata node 202 can serve metadata requests, store metadata on a permanent storage such as a disk, and can participate in a raft group to provide consistent high availability and persistence of metadata. Each metadata node 202 can be part of multiple separate raft groups, each of which represents a different partition. The metadata nodes 202 can be elected as the leader of any raft group of which they are members, and the elected leader will serve requests for the partition at this time.

[0059] In the example shown, the metadata node 202 includes a metadata application programming interface (API) server program 702. In addition, the metadata node 202 manages a first partition 704 as a leader, a second partition 706 as a leader, and a third partition 708 as a follower. The metadata node 202 can receive a client request 710 for a keyword in the first partition 704 via the metadata API server program. In response, the metadata API server program can access the first partition 704 to obtain the requested information from the associated metadata database portion, and can return the requested information to the client that sent the request 710.

[0060] Each metadata node 202 can maintain a partition map 712 between a partition map entry and a raft group. When a metadata node 202 receives a request from a client, the partition map 712 can be consulted to determine which raft group the request belongs to. If the metadata node does not have a mapping for a particular request, an error can be returned to the client that issued the request. Otherwise, the request is forwarded to the correct raft group. If the metadata node 202 is the leader of the raft group, the request is served by the metadata node. Otherwise, for example, when the metadata node is a follower of the raft group, an error is returned to the client that issued the request, and the error contains the connection information of the leader of the raft group. Therefore, a prompt about which metadata node the request is to be redirected to is provided to the client that issued the request in the returned error message.

[0061] During a partition membership event, such as if a metadata node is elected as a leader or if, as a leader, the raft group membership changes (e.g., in the event that a metadata node is added or removed), the metadata node 202 can submit new partition map entries to two locations. First, the metadata node 202 can update the partition map entry to the internal mapping 712 of the raft group so that the metadata node 202 can service requests for the partition. Second, the metadata node 202 can update the distributed cache of the partition map entry so that the client of the system can see the new routing information. To prevent the failure of the distributed cache to lose information, the metadata node 202 can periodically retransmit its partition map entries to the cache.

[0062] Figure 8 800 according to some implementations. In some examples, system 800 can correspond to system 100 and / or system 200 discussed above, as well as other system architectures. System 800 has three main components, namely clients 802 and 804, distributed cache 144, and metadata nodes 202(1)-202(4).

[0063] The distributed cache 144 can be maintained on the distributed cache node 208 and can be used to store and distribute partition map entries to clients 802, 804 (which can correspond to any of the client modules 204, 206, 212 discussed above or other client applications that can operate in the system of this article). In some examples, the distributed cache 144 can be ephemeral and / or in memory only. The distributed cache 144 contains a collection of partition map entries submitted from the metadata node 202 regarding partition membership events. A first entry 806, a second entry 808, and a third entry 810 are shown in this example.

[0064] As mentioned above Figure 2As discussed above, clients 802, 804 and Figure 8 Other clients shown in FIG. 140 may register a listener 214 that is used to update the corresponding near cache or caches when changes occur in the distributed cache 144. For example, when a partition map entry is committed to the distributed cache 144, the distributed cache node will open a connection with each client that has a registered listener 214 and send the update to the client. In the event of a complete failure, the distributed cache 144 may be rebuilt, for example, on a different node, using the latest entries, which may be received when the metadata nodes 202 resend their current partition map entries.

[0065] Clients 802 and 804 may receive requests, for example, from a user, and may send corresponding requests to a metadata database, may receive responses, and may return the responses to the requesting user. When a client starts, the client may first connect to the distributed cache 144 and read all existing partition map entries. The client may store this information as a near cache 146. In addition, the client may register a listener 214 to receive all new partition map entries.

[0066] In the example shown, metadata nodes 202(1)-202(4) maintain three partitions, namely, a first partition 812, a second partition 814, and a third partition 816. The leader of the first partition 812 is on metadata node 202(1), the leader of the second partition 814 is on metadata node 202(3), and the leader of the third partition is on metadata node 202(2). Therefore, as shown by the solid black arrows between the partition leaders and the entries, only the leaders of the partition groups can update the corresponding entries at the distributed cache 144. In addition, as shown by the dashed arrows, updates to entries 806-810 can be propagated to clients 802 and 804 to update the recent cache 146.

[0067] Therefore, only certain metadata is stored in the distributed cache 144, such as system metadata including partition mapping. The number of distributed caches 144 in a distributed multi-site computing system may be a small number compared to the total number of nodes in the system. On the other hand, the near cache 146 may exist on each node in the system that has client functionality, but may have stale data in some cases. User requests may usually be routed to the near cache 146 first, which may result in a mapping of stale search data to partitions. If the routed request receives a failure response from the node to which it is directed, this may trigger a discovery process in the sending client. For example, the discovery process may involve consulting a distributed cache 144, which may also result in an update of the client's near cache. The near cache and shared distributed cache using partition information can make a typical client request require only one network hop. Whenever the partition membership changes, the distributed cache 144 can be updated by notification from the metadata node 202, while the near cache 146 can be updated based on the registered listener 214. In this way, after any update at the centralized distributed cache 144, all client near caches can be kept up to date with changes in all partitions. Distributed cache 144 may be distributed for high availability.

[0068] In the event that the near cache 146 is stale and the distributed cache 144 is unavailable, the client may resort to polling each individual metadata node 202 to complete the request. The system performance associated with polling all nodes 202 may be poor, but still allows the client to discover the request and cache the result itself. An example algorithm for discovery may include checking the client near cache for the partition that serves the desired metadata keyword. If a cache miss occurs, the client may check the distributed cache 144 partition that serves the metadata keyword. If a cache miss occurs or a connection failure occurs, the service computing device may attempt to perform the request on each metadata node until the correct node is found.

[0069] Fig. 9 is a block diagram illustrating an example configuration of a key-space-tree data structure 900 according to some implementations. For each partition map entry received, the client may update two data structures. The first data structure (not in Fig. 9 ) can be a simple mapping of partition IDs to partition map entries. The second data structure is a keyword-space-tree data structure 900 that maintains a mapping of keyword-space IDs to keyword-space trees. The keyword-space tree may include a tree-based mapping from the lower boundary of a keyword-space range to the partition ID for that keyword-space range. Thus, the keyword-space tree data structure 900 may be searched in efficient logarithmic time to find the correct partition ID for any given keyword.

[0070] exist Fig. 9 In the example of , the key-space tree data structure 900 indicates the upper boundary keys of four partitions, such as A, C, H, N and R, i.e., the first partition 902 with key AC, the second partition 904 with key CH, the third partition 906 with key HN and the fourth partition with key NZ. Note that Figure 4 As shown, in these examples, the upper letters are not included. In this example, assuming that the keyword is "J", then, as shown in 912, the client can locate the correct partition based on J being less than N, J being greater than C, J being greater than H, and H being the lowest keyword of J, therefore, J is in the third partition, as shown in 914.

[0071] When a request comes from a user, the client can use the mapping function provided for this request type to determine the key-space requested by the user. Once the key-space is determined, the partition ID can be found from the associated key-space-tree data structure. The partition ID can then be used to look up the partition map entry for the partition, which identifies the metadata node in the raft group for the partition. The client can then send a first request to the node identified as the leader in the partition map entry. In the typical steady state of the system here, the partition map that the client has will be up to date and the request will be sent to the correct partition leader metadata node. If the metadata node contacted by the client is no longer the partition leader for the partition, the client can determine whether a hint was provided in the response, for example, indicating that another node is now the partition leader of the raft group, as described above with respect to Figure 7 as described. If so, the client will send a request to the node indicated in the hint. If there is no hint, the client may proceed to first check all nodes with an indication of being in the same raft group. If any other metadata node in the raft group provides a hint as to which node is the leader, then the client will prioritize checking the hinted node. If the client has eliminated all raft group nodes, the client may proceed to poll all metadata nodes in the system. In the worst case, no metadata node is the leader for the partition, and the client will return an error message, which may be propagated to users and / or administrators.

[0072] Fig.10is a flow chart illustrating an example process for routing metadata requests according to some implementations. The process is illustrated in a logical flow chart as a collection of boxes that represent a series of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the case of software, the boxes may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform specific functions or implement specific data types. The order in which the boxes are described should not be construed as limiting. Many of the described boxes may be combined in any order and / or in parallel to implement a process or an alternative process, and not all boxes need to be executed. For purposes of discussion, the process is described with reference to the environments, frameworks, and systems described in the examples herein, but the process may be implemented in a variety of other environments, frameworks, and systems. In Fig.10 In the example embodiment, process 1000 may be performed at least in part by one or more service computing devices 102 .

[0073] At 1002, a metadata database may be divided into a plurality of partitions and the partitions may be distributed across a plurality of metadata nodes.

[0074] At 1004, partition map information for the plurality of partitions may be stored at a distributed cache node.

[0075] At 1006 , the client computing device may receive partition map information from the distributed cache node and may store the partition map information on a local computer-readable medium.

[0076] At 1008 , the client computing device may register at the distributed cache node a request to receive an update of the partition map information at the client computing device in response to a change of the partition map information at the distributed cache node.

[0077] At 1010 , the client computing device may receive an update to the partition map information in response to a change to the partition map information at the distributed cache node.

[0078] At 1012, the client computing device may update the partition map information stored at the client computing device based on the received update.

[0079] At 1014 , the client computing device may determine to perform a database operation on a metadata database for first metadata, wherein the first metadata is associated with keyword information.

[0080] At 1016 , the client computing device may access the partition map information to determine a first metadata node among the plurality of metadata nodes that maintains a partition of the metadata database that corresponds to the keyword information.

[0081] At 1018, the client computing device may send a request to the first metadata node to perform a database operation on the first metadata based on the partition map information.

[0082] The example processing described herein is merely an example of processing provided for discussion purposes. Many other changes will be clear to those skilled in the art based on the disclosure herein. In addition, although the disclosure herein sets forth multiple examples of suitable frameworks, architectures, and environments for performing processing, the implementations herein are not limited to the specific examples shown and discussed. In addition, the present disclosure provides various example implementations described and shown in the accompanying drawings. However, the present disclosure is not limited to the implementations described and shown herein, but can be extended to other implementations known or to become known to those skilled in the art.

[0083] Fig.11 The diagram illustrates selected example components of a service computing device 102 that can be used to implement at least some of the functions of the system described herein. The service computing device 102 may include one or more servers or other types of computing devices that can be implemented in a variety of ways. For example, in the case of a server, programs, other functional components, and data can be implemented on a single server, a server cluster, a server farm or a data center, a cloud hosting computing service, etc., but other computer architectures can be used in addition or alternatively. Multiple service computing devices 102 can be set together or separately, and are organized, for example, as virtual servers, server groups, and / or server farms. The described functions can be provided by a server of a single entity or enterprise, or can be provided by servers and / or services of multiple different entities or enterprises.

[0084] In the example shown, the service computing device 102 includes one or more processors 1102, one or more computer-readable media 1104, and one or more communication interfaces 1106, or may be associated with one or more processors 1102, one or more computer-readable media 1104, and one or more communication interfaces 1106. Each processor 1102 may be a single processing unit or multiple processing units, and may include a single or multiple computing units or multiple processing cores. One or more processors 1102 may be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. For example, one or more processors 1102 may include one or more hardware processors and / or any appropriate type of logic circuits that are specifically programmed or configured to perform the algorithms and processes described herein. One or more processors 1102 may be configured to obtain and execute computer-readable instructions stored in a computer-readable medium 1104, which may program one or more processors 1102 to perform the functions described herein.

[0085] The computer-readable medium 1104 may include volatile and non-volatile memory and / or removable and non-removable media implemented by any type of information storage technology for information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable medium 1104 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, tape, disk storage, RAID storage systems, memory arrays, network attached storage, storage area networks, cloud storage, or any other medium that can be used to store the desired information and can be accessed by a computing device. Depending on the configuration of one or more service computing devices 102, the computer-readable medium 1104 may be a tangible, non-transient medium, such that when referred to, the non-transient computer-readable medium does not include media such as energy, carrier signals, electromagnetic waves, and / or signals themselves. In some cases, the computer-readable medium 1104 may be located in the same location as the service computing device 102, while in other examples, the computer-readable medium 1104 may be partially remote from the service computing device 102. For example, in some cases, the computer-readable medium 1104 may include the above-mentioned non-transient computer-readable medium. Figure 1 A portion of the network storage 120 is discussed.

[0086] The computer-readable medium 1104 may be used to store any number of functional components that may be executed by one or more processors 1102. In many implementations, these functional components include instructions or programs that may be executed by one or more processors 1102, which, when executed, specifically program the one or more processors 1102 to perform the actions attributed herein to the service computing device 102. The functional components stored in the computer-readable medium 112 may include a user network application 116, an administrative network application 124, a storage program 130, a database management program 138, an asynchronous management program 148, and a metadata API server program 702, each of which may include one or more computer programs, applications, executable code, or portions thereof. In addition, although these programs are shown together in this example, during use, some or all of these programs may be executed on separate service computing devices 102.

[0087] In addition, the computer-readable medium 1104 can store data, data structures, and other information for performing the functions and services described herein. For example, the computer-readable medium 1104 can store metadata database 132, distributed cache 144, and one or more near caches 146. In addition, although these data structures are shown together in this example, some or all of these data structures can be stored on a separate service computing device 102 during use. The service computing device 102 may also include or maintain other functional components and data (which may include programs, drivers, etc.) and data used or generated by the functional components. In addition, the service computing device 102 may include many other logical, program, and physical components, of which the above-mentioned ones are merely examples relevant to the discussion herein.

[0088] The one or more communication interfaces 1106 may include one or more software and hardware components for enabling communication with various other devices, for example, via one or more networks 106. For example, the one or more communication interfaces 1106 may be connected via a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi) and a wired network (e.g., Fiber Channel, fiber optic, Ethernet), a direct connection, and other methods listed elsewhere herein, such as Communication is achieved by one or more of the short-range communications.

[0089] The various instructions, processes, and techniques described herein may be considered in the general context of computer executable instructions (e.g., computer programs) stored on a computer-readable medium and executed by one or more processors herein. Typically, the terms "program" and "application" may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, and the like that perform a particular task or implement a particular abstract data type. These programs, applications, and the like may be executed as native code, or may be downloaded and executed, for example, in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of programs and applications may be combined or distributed as desired in a variety of implementations. The implementations of these programs, applications, and techniques may be stored on a computer storage medium or transmitted via some form of communication medium.

[0090] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A system for metadata routing in a distributed system, comprising: A first computing device capable of communicating with a plurality of metadata nodes, each metadata node maintaining a portion of a metadata database based on partitioning the metadata database to distribute the metadata database across the plurality of metadata nodes, the first computing device being configured by executable instructions to perform operations comprising: sending, by the first computing device, a request for partition map information indicating a partition of the metadata database to a distributed cache node, the distributed cache node being used to maintain a shared distributed cache of partition map entries, updated by notification from the metadata node; The first computing device registers a listener with the distributed cache node, receives the partition mapping information from the distributed cache node, and stores the received partition mapping information on a computer-readable medium of the first computing device as a near cache, each near cache having the registered listener registered with the distributed cache node, the distributed cache node including an associated registered listener, the associated registered listener being used to send update information to the first computing device to update the corresponding near cache when the distributed cache changes, In response to a change in the partition mapping information at the distributed cache node, registering at the distributed cache node a request for an update of the partition mapping information received at the first computing device, and sending the update information to the corresponding near cache by an associated registered listener; receiving, at the first computing device, an update to the partition map information; and updating, by the first computing device, the partition map information stored on the computer-readable medium of the first computing device based on the received update; determining, by the first computing device, to perform a database operation on the metadata database for first metadata, wherein the first metadata is associated with keyword information; accessing, by the first computing device, the partition mapping information stored on a computer-readable medium of the first computing device to determine a first metadata node among the plurality of metadata nodes, the first metadata node being indicated as maintaining a partition of the metadata database corresponding to the keyword information; and The first computing device sends a request to the first metadata node to execute the database operation on the first metadata based on the partition mapping information. Each partition has a single leader node that serves as the partition leader, which is responsible for servicing all requests for that partition. The distributed cache contains a set of partition map entries submitted from a metadata node that is the leader node regarding partition membership events, When the partition map entry is committed to the distributed cache, the distributed cache node opens a connection with each first computing device having the registered listener and sends the update to the first computing device to update the corresponding near cache, Whenever partition membership changes, the distributed cache is updated via notification from the leader node and the near cache is updated based on the registered listeners. The first computing device is capable of discovering a partition leader responsible for servicing requests to perform the database operation by referring to the near cache, and being routed to a new leader based on update information provided to the near cache when the partition leader should change.

2. The system according to claim 1, wherein: Partitioning the metadata database to distribute the metadata database across the plurality of metadata nodes comprises assigning a first partition of the metadata database to the plurality of metadata nodes, wherein: the first metadata node being a leader node configured to respond to metadata database requests based on maintaining the first partition of the metadata database at the first metadata node; and At least a second metadata node maintains a replica of the first partition and is configured to receive updates to the first partition of the metadata database from the first metadata node and apply the updates to the replica of the first partition.

3. The system according to claim 2, wherein: The operation also includes: based on not receiving a response to the request sent to the first metadata node, and based on the partition mapping information indicating that the second metadata node maintains the copy of the first partition, sending a request by the first computing device to the second metadata node to perform the database operation.

4. The system according to claim 2, wherein: The second metadata node replaces the first metadata node as the leader node, and the second metadata node is configured to respond to a metadata database access request to access the first partition of the metadata database, the operation further comprising: receiving, by the first computing device, in response to a request for the first metadata associated with the keyword information, an indication that the first metadata node will not respond to the request, wherein receiving the indication further comprises receiving a notification that the second metadata node is now the leader node configured to respond to metadata database requests for the first partition; and A request for the first metadata associated with the keyword information is sent by the first computing device to the second metadata node based on the notification that the second metadata node is now the leader node.

5. The system of claim 2, wherein: The first metadata node is configured to periodically send messages to the second metadata node and at least one other metadata node that also maintains a copy of the first partition of the metadata database; and After the first metadata node fails to send the message within a threshold time period, a second metadata node replaces the first metadata node as the leader node based on a consensus from the at least one other node, wherein the second metadata node is configured to respond to metadata database access requests to access the first partition of the metadata database.

6. The system according to claim 1, wherein: The first computing device determines to perform a database operation on the metadata database for the first metadata in response to receiving at least one of the following requests: a request from a user's device via a user's web application; or Request from the administrator's device via the user web application.

7. The system according to claim 1, wherein: The determination by the first computing device to perform a database operation on the metadata database for the first metadata is an operation in response to execution of one or more asynchronous storage-related operations determined by an asynchronous hypervisor based on an update queue.

8. The system according to claim 1, wherein: The first computing device stores at least a portion of the partition mapping information as a keyword-space-tree data structure, and the operation of accessing the partition mapping information to determine the first metadata node among the multiple metadata nodes also includes traversing the keyword-space-tree data structure to determine the first partition corresponding to the keyword information.

9. The system of claim 1, further comprising: Based on receiving the response from the first metadata node, metadata information is sent to another computing device to obtain data from a remote storage system over a network.

10. A method for metadata routing in a distributed system, comprising: sending, by a first computing device, a request for partition map information indicating partitions of a metadata database to a distributed cache node, wherein the first computing device is capable of communicating with a plurality of metadata nodes, each metadata node maintaining a portion of the metadata database based on partitioning the metadata database to distribute the metadata database across the plurality of metadata nodes, the distributed cache node being configured to maintain a shared distributed cache of partition map entries that is updated by notification from the metadata node; The first computing device registers a listener, receives the partition mapping information from the distributed cache node, and stores the received partition mapping information on a computer-readable medium of the first computing device as a near cache, each near cache having the registered listener registered with the distributed cache node, the distributed cache node including an associated registered listener, the associated registered listener being used to send update information to the first computing device to update the corresponding near cache when the distributed cache changes, In response to a change in the partition mapping information at the distributed cache node, registering at the distributed cache node a request for an update of the partition mapping information received at the first computing device, and sending the update information to the corresponding near cache by an associated registered listener; receiving, at the first computing device, an update to the partition map information; as well as updating, by the first computing device, the partition map information stored on the computer-readable medium of the first computing device based on the received update; Determining, by the first computing device, based at least on the keyword information, to send a request to the metadata database; Determining, by the first computing device, a first metadata node among the plurality of metadata nodes based on the partition mapping information stored on a computer-readable medium of the first computing device, the first metadata node being indicated as maintaining a partition of the metadata database corresponding to the keyword information; as well as The first computing device sends a request to execute a database operation to the first metadata node based on the partition mapping information. Each partition has a single leader node that serves as the partition leader, which is responsible for servicing all requests for that partition. The distributed cache contains a set of partition map entries submitted from a metadata node that is the leader node regarding partition membership events, When the partition map entry is committed to the distributed cache, the distributed cache node opens a connection with each first computing device having the registered listener and sends the update to the first computing device to update the corresponding near cache, Whenever partition membership changes, the distributed cache is updated via notification from the leader node and the near cache is updated based on the registered listeners. The first computing device is capable of discovering a partition leader responsible for servicing requests to perform the database operation by referring to the near cache, and being routed to a new leader based on update information provided to the near cache when the partition leader should change.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising: sending, by a first computing device, a request for partition map information indicating partitions of a metadata database to a distributed cache node, wherein the first computing device is capable of communicating with a plurality of metadata nodes, each metadata node maintaining a portion of the metadata database based on partitioning the metadata database to distribute the metadata database across the plurality of metadata nodes, the distributed cache node being configured to maintain a shared distributed cache of partition map entries that is updated by notification from the metadata node; registering a listener by the first computing device, receiving the partition mapping information from the distributed cache node, and storing the received partition mapping information on a computer-readable medium of the first computing device as a near cache, each near cache having the registered listener registered with the distributed cache node, the distributed cache node including an associated registered listener, the associated registered listener being used to send update information to the first computing device to update the corresponding near cache when the distributed cache changes; In response to a change in the partition mapping information at the distributed cache node, registering at the distributed cache node a request for an update of the partition mapping information received at the first computing device, and sending the update information to the corresponding near cache by an associated registered listener; receiving, at the first computing device, an update to the partition map information; Based on the received update, the first computing device updates the partition map information stored on the computer-readable medium of the first computing device; Determining, by the first computing device, based at least on the keyword information, to send a request to the metadata database; Determining, by the first computing device, a first metadata node among the plurality of metadata nodes based on the partition mapping information stored on the computer-readable medium of the first computing device, the first metadata node being indicated as maintaining a partition of the metadata database corresponding to the keyword information; as well as The first computing device sends a request to execute a database operation to the first metadata node based on the partition mapping information. Each partition has a single leader node that serves as the partition leader, which is responsible for servicing all requests for that partition. The distributed cache contains a set of partition map entries submitted from a metadata node that is the leader node regarding partition membership events, When the partition map entry is committed to the distributed cache, the distributed cache node opens a connection with each first computing device having the registered listener and sends the update to the first computing device to update the corresponding near cache, Whenever partition membership changes, the distributed cache is updated via notification from the leader node and the near cache is updated based on the registered listeners. The first computing device is capable of discovering a partition leader responsible for servicing requests to perform the database operation by referring to the near cache, and being routed to a new leader based on update information provided to the near cache when the partition leader should change.