Versioned hierarchical data structure in distributed data store

Abstract

The present disclosure relates to versioned hierarchical data structures in a distributed data store. A distributed data store can maintain a versioned hierarchical data structure. Different versions of the hierarchical data structure can be maintained in consistency with a transaction log for the hierarchical data structure. When an access request is received for the hierarchical data structure, a version of the hierarchical data structure can be identified for processing the access request. For access requests with snapshot isolation, the identified version alone can be sufficient to consistently process the access request. For access requests with higher isolation requirements, such as serializable isolation, a transaction based on the access request can be committed to the transaction log so that access requests that result in committed transactions can be allowed, while access requests that result in conflicting transactions can be rejected.

Description

[0001] This application is a divisional application of the invention patent application with application number 201780024249.8, application date April 18, 2017, entitled "Versioned Hierarchical Data Structure in Distributed Data Storage Area". Technical Field

[0002] This disclosure relates to versioned hierarchical data structures in distributed data storage areas. Background Technology

[0003] Tiered storage systems have implemented many different storage schemes to efficiently and reliably maintain data. Storage schemes are typically deployed on distributed system architectures when client applications require higher availability of data permanently stored in the data storage system. Common solutions for making data available include storing one or more versions or copies of data across multiple storage nodes. However, increasing the number of versions or copies increases the complexity and operational cost of generating a consistent view of the tiered data. For example, synchronization protocols may require allowing a single component to handle operations such as writes or other modifications to the tiered data. Such synchronization protocols can reduce the availability of the distributed system in responding to client requests, as a single component can become overburdened. Therefore, there is a strong need for consistency mechanisms that provide a consistent version of tiered data without sacrificing availability. Attached Figure Description

[0004] Figure 1 This is a logical block diagram illustrating a hierarchical data storage area that provides a versioned hierarchical data structure according to some implementation schemes.

[0005] Figure 2 This is a block diagram illustrating a network of providers of directory storage services that include a hierarchical data storage area with a versioned hierarchical data structure, according to some implementation schemes.

[0006] Figure 3 This is a block diagram illustrating a hierarchical storage node according to some implementation schemes.

[0007] Figure 4A This is a block diagram illustrating a data model for a hierarchical data storage area used to provide a versioned hierarchical data structure, according to some implementation schemes.

[0008] Figure 4B and 4C This is an example table illustrating the storage format of a data model for a hierarchical data structure according to some implementation schemes.

[0009] Figure 5 This is a diagram illustrating the use of a separate transaction log store to provide consistent storage for versioned hierarchical data structures, according to some implementation schemes.

[0010] Figure 6 This is a sequence diagram illustrating the processing of access requests with serializable isolation that utilize a transaction log storage area according to some implementation schemes.

[0011] Figure 7 This is a sequence diagram illustrating the version processing of access requests with snapshot isolation using a hierarchical data structure maintained at the storage node according to some implementation schemes.

[0012] Figure 8 This is a sequence diagram illustrating the use of a transaction log storage area to process conditional access requests with serializable isolation, according to some implementation schemes.

[0013] Figure 9 This is a sequence diagram illustrating how, according to some implementation schemes, a transaction log storage area is used to process batch access requests with serializable isolation.

[0014] Figure 10 This is a sequence diagram illustrating how, according to some implementation schemes, transaction log storage areas and transaction status tokens are used to process client-specified transaction requests with serializable isolation.

[0015] Figure 11 This is a sequence diagram illustrating how, according to some implementation schemes, access requests with page number annotation results are processed using page number annotation tokens.

[0016] Figure 12 This is a high-level flowchart illustrating methods and techniques for implementing versioned hierarchical data structures to serve access requests, according to some implementation schemes.

[0017] Figure 13 This is a high-level flowchart illustrating methods and techniques for maintaining versions of hierarchical data structures at storage nodes in a manner consistent with transaction logs of hierarchical data structures, according to some implementation schemes.

[0018] Figure 14 These are example computer systems based on various implementation schemes.

[0019] While embodiments are described herein by way of example with respect to several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed descriptions thereof are not intended to limit the embodiments to the specific forms disclosed, but rather, are intended to cover all modifications, equivalents, and substitutions falling within the spirit and scope of the disclosure. The headings used herein are for organizational purposes only and are not intended to limit the scope of the specification or claims. As used throughout this application, the word “may” is used in a permissible sense (i.e., implying potential) rather than a mandatory sense (i.e., implying necessity). Words such as “comprising” indicate an open-ended relationship and therefore mean including but not limited to. Similarly, the word “having” also indicates an open-ended relationship and therefore means having but not limited to. The terms “first,” “second,” “third,” etc., used herein serve as labels for the nouns preceding them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless such ordering is explicitly stated otherwise.

[0020] Various components can be described as being "configured" to perform one or more tasks. In this context, "configured" is a broad description, generally meaning "having a structure that performs one or more tasks during operation." Thus, a component can be configured to perform a task even when it is not currently performing that task (e.g., a computer system can be configured to perform an operation even when no operation is currently being performed). In some contexts, "configured" can be a broad description of a structure, generally meaning "having any circuitry that performs tasks during operation." Therefore, a component can be configured to perform a task even when it is not currently powered on. Typically, the circuitry forming the structure corresponding to "configured" can include hardware circuitry.

[0021] For ease of description, various components may be described as performing one or more tasks. Such descriptions should be interpreted as including the phrase "configured to". The description of a component configured to perform one or more tasks is expressly intended not to be interpreted in accordance with 35 U.S.SC §112(f).

[0022] "Based on." As used herein, this term describes one or more factors that influence a determination. The term does not exclude other factors that may influence the determination. That is, a determination may be based solely on those factors or at least in part on those factors. Consider the phrase "A is determined based on B." While B may be a factor influencing the determination of A, such a phrase does not exclude the possibility that the determination of A is also based on C. In other cases, A may be determined solely on B.

[0023] The scope of this disclosure includes any feature or combination of features disclosed herein (express or implicit), or any generalization thereof, whether or not it alleviates any or all the problems addressed herein. Therefore, new claims may be formulated during the examination of any such combination of features in this application (or an application claiming priority thereto). Specifically, with reference to the appended claims, features of dependent claims may be combined with features of independent claims, and features of corresponding independent claims may be combined in any suitable manner, not just the specific combinations listed in the appended claims. Detailed Implementation

[0024] This paper describes various implementations of versioned hierarchical data structures in distributed data stores. Typically, hierarchical data stores, such as directory data stores, restrict access to the consistency and current version of the hierarchical data structure (e.g., trees, graphs, or other hierarchical data structures). This allows the hierarchical data store to provide consistency in the face of concurrent access requests. Such implementations force all access requests to be executed using similar processing paths, thus providing similar levels of consistency and isolation, which offers little flexibility for clients who may not always want consistency or isolation at the expense of speed or other performance considerations when accessing hierarchical data structures. Providing versioned hierarchical data structures allows for the implementation of multiple types of consistency models, enabling different requests to specify the desired isolation and / or consistency levels, and thus control over the execution of requests regarding access to the hierarchical data structure.

[0025] Figure 1 This is a logical block diagram illustrating a hierarchical data storage area providing a versioned hierarchical data structure according to some implementation schemes. Hierarchical data storage area 120 can provide multiple versions of hierarchical data structure 132 over time, enabling various types of consistency and isolation levels to be implemented when serving access request 112. For example, each version, such as versions 132a, 132b, and 132c, can maintain a hierarchical data structure so that historical queries (or possible queries that do not care about the existence of stale data) can utilize a version of the hierarchical data structure that is readily available (e.g., at the storage node where the hierarchical data structure is maintained in memory). Other access requests 112 that expect or require a strongly consistent version of the hierarchical data structure can invoke operations that provide serializable isolation for the access request, for example, by evaluating requests regarding the transaction log using optimistic parallelism as described below. Access request 112 can also invoke path-based traversal to determine various information stored as part of hierarchical data structure 132.

[0026] Please note that the following is provided: Figure 1This is a logical diagram of a hierarchical data storage area that provides a versioned hierarchical data structure, and is not intended to limit the physical arrangement, size, or number of components, modules, or devices that implement a distributed data storage area.

[0027] According to various implementations, the specification first describes examples of distributed data storage areas as directory storage services. In various implementations, the example directory storage service can store hierarchical data structures for many different clients. The description of the exemplary web-based database service includes relevant aspects of the example web-based directory storage service and various interactions between the directory storage service and clients. The specification then describes flowcharts of various implementations of methods for versioned hierarchical data structures in a distributed data storage area. Next, the specification describes example systems in which the disclosed techniques can be implemented. Various examples are provided throughout the specification.

[0028] Figure 2 This is a block diagram illustrating a provider network for directory storage services that provides a versioned, hierarchical data storage area with a hierarchical data structure, according to some implementation schemes. Provider network 200 may be established by an entity such as a company or public sector organization to enable access to one or more services (e.g., various types of cloud-based computing or storage) of client 210 via the Internet and / or other networks. Provider network 200 may include numerous data centers hosting various resource pools, such as physical and / or virtualized computer servers, storage devices, network devices, etc. (e.g., referred to below) that require the implementation and distribution of infrastructure and services provided by provider network 200. Figure 14 The described computing system 200 is a collection of computing systems. In some implementations, the provider network 200 may implement the described directory storage service 220, providing other computing resources or services, such as virtual computing services and storage services, such as object storage services, block-based storage services, data warehouse storage services, archive storage services 270, and / or any other type of network-based service 280 (which may include various other types of storage, processing, analysis, communication, event handling, visualization, and security services). Clients 210 may access these various services provided by the provider network 200 via network 260. Similarly, the network-based services themselves may communicate with each other and / or utilize each other to provide different services. For example, the various other services 280 may store, access, and / or depend on hierarchical data structures stored in the directory storage service 220.

[0029] In various implementation schemes, Figure 2 The components shown can be implemented directly within computer hardware, as instructions that can be executed directly or indirectly by computer hardware (e.g., a microprocessor or computer system), or a combination of these technologies. For example, Figure 2The components can be implemented by a system comprising multiple compute nodes (or simply nodes), each compute node being similar to Figure 14 The computer system implementations shown and described below. In various implementations, the functionality of a given service system component (e.g., a component of a database service or a component of a storage service) may be implemented by a specific node or may be distributed across several nodes. In some implementations, a given node may implement the functionality of more than one service system component (e.g., more than one database service system component).

[0030] Directory storage service 220 can store, manage, and maintain hierarchical data structures, such as those described below. Figure 4A The directory structure of the corresponding nodes (in a single-tenant or multi-tenant manner) stored in the hierarchical storage node 240 is discussed. Clients of the directory storage service 220 can operate on any subset or part of the hierarchical data structure with transaction semantics and / or can perform path-based traversal of the hierarchical data structure. These capabilities allow clients to access the hierarchical data structure in a variety of ways. For example, a client can leverage transactional access requests to perform multiple operations concurrently, thereby affecting different parts of the hierarchical directory structure (e.g., nodes) (e.g., reading parts of the hierarchical directory structure, adding nodes, and indexing some attributes of nodes), while enforcing that the final updates of the operations within the transaction are isolated, consistent, atomic, and permanently stored.

[0031] In various implementations, the directory storage service 220 may implement a routing layer 232 to direct access requests from internal or external clients to the appropriate tiered storage node 240. For example, the routing layer 232 may implement a set of routing nodes that maintain mapping information identifying the location of tiered data structures on the tiered storage host 240. When an access request is received, the routing layer nodes can determine which of the tiered storage nodes carrying the tiered data structure is identified as the one sending the access request. Consider scenarios where the tiered data structure can be replicated as part of a replica group across multiple different tiered storage nodes 240, for example… Figure 5 As shown in the diagram. Route 232 can implement various load balancing schemes to route requests from different clients to different tiered storage nodes within the replica group, ensuring that no single tiered storage node becomes overloaded. Furthermore, because tiered storage nodes 240 can utilize tokens to maintain state between different access requests sent by clients, allowing different tiered storage nodes 240 to handle each request from a client, route 232 does not need to track which tiered storage node is communicating with which client (because sticky sessions can be managed through the following...). Figure 10 and 11 The discussion addresses token state management techniques to avoid this.

[0032] Control plane 234 can implement various control functions to manage the tiered storage node 240 and other components of the directory storage service 220 that provide storage for tiered data structures, such as guiding the creation and placement of new tiered data structures on tiered storage node 240, storage scaling, hot management, node repair, and / or replacement. For example, various placement schemes can utilize techniques such as consistent hashing (e.g., hashing based on the identifier of the corresponding tiered data structure) to identify tiered storage nodes storing versions of tiered data structures, or randomly map tiered data structures to several tiered storage nodes 240 forming a replica set. To provide hot management, control plane 234 can, for example, collect metrics published by each tiered storage host 240. Each host may have various performance characteristic thresholds, such as memory utilization, CPU utilization, disk utilization, and request rate capacity. When a tiered storage node reports metrics exceeding a threshold (or multiple thresholds), control plane 234 can guide the migration of one or more tiered data structures to different tiered storage nodes. Similarly, control plane 234 can detect when certain tiered storage nodes cannot keep up with access requests for a specific replica group of a tiered data structure, and can provide additional tiered storage nodes to horizontally scale the replica group to better meet access request demands.

[0033] Tiered storage node 240 can maintain and manage access to tiered storage nodes in directory storage service 220. Figure 3 This is a block diagram illustrating a tiered storage node according to some implementation schemes. Tiered storage node 300 may implement request handler 310 to process access requests and pass appropriate instructions or requests to other components, such as storage engine 340, transaction log interface 350, or archive interface 360. For example, access request handler 310 may interpret various requests according to a program interface format, such as an application programming interface (API), as described below. Figures 6 to 11 The interface discussed is 600. Access requests can include the various requests described in the aforementioned diagram, as well as other types of requests (which may include or be embedded in the following). Figures 6 to 11 The requests or operations described in the document include various access requests. This includes creating, updating, attaching, detaching, deleting, and querying nodes in a hierarchical data structure, as well as access requests to define, populate, discover, and query local indexes (which may be strongly consistent and maintained as part of the hierarchical data structure or separately) for node attributes in the hierarchical data structure.

[0034] In various implementations, storage engine 340 may be a storage engine configured to interact with the structure or format of the data when data is stored in the current hierarchical data structure storage area 320 and the historical hierarchical data structure storage area 330 (e.g., a key-value storage engine for data maintained in a key-value storage format, a relational data storage engine for data maintained in a relational storage format, etc.), the data according to the following description... Figure 4A The discussed model is used for maintenance. In some implementations, the current hierarchical data structure storage area 320 may be implemented partially or entirely in memory or other fast access storage devices (e.g., random access memory devices (RAM)), and block-based permanent storage devices, including disks or solid-state drives, may be used to store historical hierarchical data structures 330. In some implementations, caching techniques may be implemented such that frequently accessed data portions (e.g., portions frequently accessed in the current hierarchical data structure) are maintained in memory components, while other portions are maintained in block-based permanent storage components. The hierarchical storage node 300 can operate multi-tenant storage for hierarchical data structures, allowing different hierarchical data structures maintained on behalf of different clients, accounts, customers, etc., to be maintained in both the current hierarchical data structure storage area 320 and the current hierarchical data structure storage area 330. For example, for different hierarchical data structures stored at the hierarchical storage node 300, the hierarchical storage node 300 may participate in different replica groups with different hierarchical storage nodes.

[0035] Based on the following text Figures 5 to 11 The various techniques discussed herein allow the transaction log interface 350 to provide the ability to interact with logs corresponding to the hierarchical data structures stored in the transaction log store 250 for hierarchical data structures (e.g., to verify transactions). Similarly, the archive interface 360 ​​can be implemented to retrieve archived transactions or snapshots to serve access requests for historical changes to the hierarchical data structure, historical queries, or other access requests requiring older versions of the hierarchical data structure than those maintained in the historical hierarchical data structure store.

[0036] Back Figure 2 The transaction log store 250 provides fault-tolerant, high-performance, and persistent log publishing services. The transaction log store 250 can be used as a commit log for substantially consistent distributed applications (such as databases, key-value stores, and lock managers), and as... Figure 2 As shown, the directory storage service 220 provides a hierarchical data storage area. The transaction log storage area 250 can provide strong consistency guarantees and support constraints between committed records to implement features such as deduplication, sorting, and read-write conflict detection. For example, in the following... Figure 5 , 6In the various requests shown in 8, 9, and 11, the transaction log store 250 can determine whether to commit changes to the hierarchical data structure (e.g., write requests and other modifications) by checking whether the proposed transaction conflicts with other committed transactions. This feature can provide a fine-grained locking model on the hierarchical data structure (e.g., only those parts of the hierarchical data structure affected by conflicts between transactions can be locked). The transaction log store can maintain a separate log record or chain of log records for each hierarchical data structure, serving as an authoritative definition of the changes in the state of the hierarchical data structure over time. Transactions can be ordered according to their transaction sequence numbers, which can monotonically increment to refer to the state of the hierarchical data structure at the corresponding point in time. Note that in some implementations, the transaction log store 250 may be a standalone network-based storage service implemented as part of a provider network 250 external to the directory storage service 220.

[0037] Archive management 236 can utilize transactions stored in different hierarchical data structures within the corresponding transaction logs stored in transaction log storage area 250 to generate and store snapshots of the hierarchical data structures at different points in time within the archive storage service 270. For example, archive management can determine when to capture snapshots of the hierarchical data structures, provide appropriate storage locations within the archive storage service 270, and instruct archive worker nodes (not shown) to perform read, write, and other operations to generate and place snapshots within the archive storage service 270. Similarly, archive management 236 can instruct the replication and storage of individual log records / transactions and / or groups of log records and transactions as part of the archive transaction logs of the hierarchical data structures within the archive storage service 270.

[0038] Generally, client 210 can encompass any type of client configurable to submit web-based service requests to provider network 200 via network 260, including requests for directory services (e.g., requests to create or modify hierarchical data structures for storage in directory storage service 220, etc.). For example, a given client 210 may include a suitable version of a web browser, or may include plug-in modules or other types of code modules configured to act as an extension of or within the execution environment provided by a web browser. Alternatively, client 210 can encompass, for example, database applications (or their user interfaces), media applications, office applications, or any other applications that can utilize persistent storage resources to store and / or access one or more hierarchical data structures to perform technologies such as organization management, identity management, or permissions / authorization management. In some implementations, such applications may include sufficient protocol support (e.g., for a suitable version of Hypertext Transfer Protocol (HTTP)) for generating and processing web-based service requests without necessarily implementing full browser support for all types of web-based data. That is, client 210 can be an application configured to interact directly with web-based service platform 200. In some implementations, client 210 may be configured to generate web-based service requests based on a representative state transition (REST) ​​style web-based service architecture, a document- or message-based web-based service architecture, or other suitable web-based service architecture.

[0039] In some implementations, client 210 can be configured to provide access to network-based services to other applications in a manner transparent to those applications. For example, client 210 can be configured to integrate with an operating system or file system to provide storage according to a suitable variant of the storage model described herein. However, the operating system or file system may provide different storage interfaces to applications, such as a traditional file system hierarchy of files, directories, and / or folders. In such implementations, it may not be necessary to modify the application to utilize the storage system service model. Instead, the details of the interface with provider network 200 can be coordinated by client 210 and the operating system or file system on behalf of the application executing within the operating system environment.

[0040] Client 210 can transmit network-based service requests (e.g., access requests for hierarchical data structures in directory storage service 220) to and receive responses from network-based service platform 200 via network 260. In various embodiments, network 260 can include any suitable combination of network hardware and protocols necessary to establish network-based communication between client 210 and platform 200. For example, network 260 can typically include various telecommunications networks and service providers that jointly implement the Internet. Network 260 can also include private networks, such as local area networks (LANs) or wide area networks (WANs), and public or private wireless networks. For example, given client 210 and network-based service platform 200, they may each be provided within an enterprise with their own internal networks. In such embodiments, network 260 can include hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, billing software, firewall / security software, etc.). Networking links are established between given client 210 and the Internet, and between the Internet and network-based service platform 200. Note that in some implementations, client 210 may use a private network instead of the public Internet to communicate with network-based service platform 200.

[0041] Different types of hierarchical data structures can be stored, managed, and / or represented in different ways. Figure 4A This is a block diagram illustrating an example of a hierarchical data storage area for providing a versioned hierarchical data structure, according to some implementation schemes. Nodes can be basic elements of a hierarchical data structure, such as directory structures 410a or 410n, and can be... Figure 4A The diagrams shown use circles or squares to represent nodes (e.g., nodes 400, 401, 402, 403, 404, 405, 406, 407, and 421). Nodes can have a globally unique identifier (GUID), zero or more attributes (key-value pairs), and zero or more links to other nodes. In some implementations, a directory can be a type of node with zero or more child links to other nodes (directories or resources). In some implementations, a directory node can have zero or one parent directory node, meaning that directory nodes and links define a tree structure. Figure 4AIn Figure 4, node 401 is an example of a directory node. Node 400 can be the root node, which is the logical root of the multi-directory structure 410 and can be invisible to clients of the directory storage service 220. Resource nodes (represented by squares, such as resource nodes 404, 405, 406, and 407) can be leaf nodes in the directory structure 410. Resource nodes can have a unique external ID (e.g., a specified client) and client-defined attributes. Resource nodes can have multiple parent nodes (this allows some hierarchical data structures to be configured as directed acyclic graphs (DAGs). Node 405 in Figure 4 is an example of a resource node, and it has two parent nodes (nodes 402 and 403).

[0042] In some implementations, multiple types of resource nodes can be implemented. For example, in some implementations, a policy node can be a resource node type with two user-defined attributes: a policy type and a policy document (e.g., describing the policy applied to the applicable node). Figure 4A Resource node 406 in the example can be a strategy resource node. Another type of resource node can be an index resource node. For example, an index resource node is an index of the various attribute values ​​of nodes in its child nodes, as well as other descendant nodes of the directory node to which the index node is attached. For example, if resource node 407 is an index node, then index node 407 can provide indexes for the attributes of child nodes 402 and 403, as well as descendant nodes 404, 405, and 406.

[0043] In some implementations, a link can be a directed edge between two nodes that defines a relationship between them. Many types of links may exist, such as client-visible link types and another type of link used for internal operational implementations. In some implementations, a sub-link type can create a parent-child relationship between the nodes it connects to. For example, sub-link 'bb' connects nodes 401 and 403. Sub-links can define the hierarchical structure of directory structure 410. Sub-links can be named to define the path to the node to which the link targets. Another type of client-visible link can be an attachment link. An attachment link can apply a resource node (e.g., a policy resource node or an index resource node) to another resource node or directory node. Attachment links may not define the hierarchical structure of directory structure 410. For example, attachment link 'xx' applies policy attributes stored in policy resource node 406 to directory node 402. A node can have multiple attachments. In some implementations, attachment constraints can be implemented, such as restricting no more than one policy resource node of any given policy type to be attached to the same node. In some implementations, non-client-visible link types or implicit link types, i.e., reverse links, can also be implemented. Reverse links can be used to optimize the traversal of directory structure 410 for common operations such as resource node lookup (e.g., policy lookup). Directory storage service 220 can maintain reverse links in the opposite direction of sub-links and attached links.

[0044] In various implementations, nodes in directory structure 410 can be identified and located by describing how to reach a node from logical root node 400 via pathnames, starting with a link marked " / " and following sub-links separated by sub-path separators " / " until the desired node is reached. For example, node 405 can be identified using the path: " / directoryA / aa / dd". Since some nodes can be children of multiple directory nodes, multiple paths can be identified. For example, the following path can also be used to identify node 405: " / directoryA / bb / ee". Because directory structure 410 can be a set of nodes, its boundaries are defined by the hierarchy of those nodes in the set (e.g., the resulting hierarchical data structure, such as a tree or DAG created by links between nodes). In this way, directory structure 410 can represent a single, independent, or partially independent organization.

[0045] To store the illustrated directory structure in the current hierarchical data structure store 320 and the historical hierarchical data structure store 330, in some implementations, the described nodes, link attributes, etc., can be modeled after the Resource Description Framework (RDF) data. To maintain multiple versions of the hierarchical data structure, version control information can also be included to indicate how the data changes over time. RDF data can be constructed as (subject, predicate, object) tuples. When additional version information is included, this structure may become: (subject, predicate, object, version, previous version). To represent an RDF-based hierarchical data structure, multiple types of RDF predicates may exist. In some implementations, one type of RDF predicate can represent a link in the hierarchical data structure, while another type of RDF predicate can represent an attribute. Different types of predicates can represent different hierarchical data structures. Link predicates can be between two nodes, while attribute predicates can be between a node and a value. Since a single node may participate in multiple predicates of the same type but with different values, predicates can begin with a common prefix and end with some other type or naming information to aid in lookup. For example, the version entries in the predicate tuple could be logical timestamps of the creation of a link or attribute (e.g., a transaction sequence number), because all changes to the hierarchical data structure can utilize the transaction resolution process provided by the transaction. Log store 250 can be assigned ordered logical timestamps by transaction log store 250.

[0046] As mentioned above Figure 3 As indicated, hierarchical storage nodes can maintain both the current version and past versions of the hierarchical data structure. In at least some implementations, different corresponding tables can be maintained for each hierarchical data structure, with one table storing the data of the current version, for example... Figure 4B The current version table is 450, and another table stores immutable records of the previous version, for example... Figure 4C The previous version, Table 460, was used. Figure 4AIn the example directory structure 410a, Table 450 and Table 460 may show the contents of the current version table and the previous version table. For ease of reading, the tables show GUID_401, GUID_402, etc., instead of the actual GUID values. Although predicate names may be displayed as character strings, the actual representation in the storage may use a more compact binary representation than the string representation. The current version table 450 may store the latest version data of each row in the table, and the values in the previous version column may provide indexes into the previous version table 460 to locate the previous values of said rows. In the previous version table 460, the current version may be added to the predicate so that it can be filtered. The previous version column in the previous version table 460 may allow the storage engine to locate older versions, further going back in logical time to perform various access requests, such as access requests that operate on the hierarchical data structure at a specified point in time. Note that although the structures of tables 450 and 460 may be shown as relational tables, such description is not restrictive. For example, as described above, the current version hierarchical data structure storage 320 and the historical hierarchical data structure storage 330 may be non-relational key-value storages, where the key is formed by the concatenation of the subject and predicate values, and the value may be formed by the concatenation of the remaining column values shown. For example, the first row in table 450 may be logically represented as the following <key, value> pair: <GUID_401+link.child.aa, GUID_402+Ver_1+None>.

[0047] In various embodiments, one or both of tables 450 and 460 may be accessed to perform various operations. For example, an access request may specify a query: "Find all child nodes of the node with ID GUID_401, select GUID_401.child* from CurrentVersion" or a query: "Find all policies of the resource node with ID GUID_405 along all paths to the root. To serve such queries, a depth-first traversal may be performed along the parent links. At each node along the path to the root, the following internal queries may be executed: Internal Query 1: "Check if the node has a policy: select GUID_405.link.HasPolicy* from CurrentVersion"; Internal Query 2: "If the node has a policy returned in Internal Query 1, use the value in the link to get the policy document value from the policy node: select GUID_406.link.PolicyDoc from CurrentVersion"; Internal Query 3: "Find all parent nodes of the current node and execute Internal Queries 1 to 3 for each parent node until the root directory of the directory structure is reached. It should be noted that the foregoing examples are not intended to limit the format, structure, syntax or other manners in which tables 450 and 460 may express or process queries.

[0048] Figure 5 This is a block diagram illustrating the use of separate transaction log stores to provide consistent storage for versioned hierarchical data structures according to some implementation schemes. For example, multiple clients 510a, 510b, and 510c can simultaneously execute various access requests to the hierarchical data structure, such as various write requests 512a, 512b, and 512c. In at least some implementations, replica group 520 may include multiple storage nodes, such as hierarchical storage nodes 522a, 522b, and 522c, which maintain versions of the hierarchical data structure available to serve various access requests from client 510. For example, client 510 may submit different write requests 512 to hierarchical storage node 522 according to a routing pattern that can route access requests from each client to different storage nodes in replica group 520 according to a load balancing scheme. Upon receiving a request, each hierarchical storage node 522 can perform various operations on the current version of the hierarchical data structure at the storage node, and then provide write 524 to the transaction log store 210 for committing the directory structure log 530, including various information that affects or accesses the data, such as by executing write requests, the write requests themselves, and other indications of the time point in time that identify the current version of the hierarchical data structure at storage node 522. Commit 526 or conflict indications can be provided to the corresponding storage node 522. For those committed writes, the directory structure log can be read (as described below). Figure 13 (As discussed), and will submit the write application 532 to the corresponding version of the hierarchical data structure maintained at storage node 522.

[0049] In some implementations, archive management 236 can also read directory structure log 530 to retrieve transactions or snapshots written to 534 for archiving. Archive management 236 can then periodically or non-periodically update 542 the archive log 540 in the archive storage service 270 and generate and simultaneously send new snapshots 552 as part of the archive snapshot 550 for maintenance. In this way, the hierarchical data structure can be recreated at any point in time, for example, by loading a snapshot onto the storage node and applying the transaction sequence number from the archive log 540, ensuring that the version of the hierarchical data structure at the storage number is consistent with a specified point in time.

[0050] Figure 6 This is a sequence diagram illustrating the processing of access requests with serializable isolation utilizing a transaction log storage area according to some implementation schemes. Tiered storage node 620 and Figures 7 to 11Other storage nodes discussed herein may implement interface 600 to handle requests from clients. In various implementations, interface 600 may be a programming interface (API) that can be invoked by a command line, a graphical user interface, or other interface control / generation components at client 610. Serializable access request 630 may be a request, such as an access request to change data in the hierarchical data store, or a request to read data from a consistent version across the entire distributed data store of the hierarchical data store. Request 630 may include a specified operation to be performed (e.g., a read or write operation). Once received, hierarchical storage node 620 can access an identified version of the hierarchical data structure (e.g., the current version) and determine the exact changes to be made to the hierarchical data structure to perform the operation, including the accessed (read) and / or affected (written) data, such as the nodes, links, attributes, etc., to be read or changed. Hierarchical storage node 620 may then construct a conflict verification request 640 to send to transaction log store 250 for evaluation, including the accessed / affected data, the data of the hierarchical data structure at hierarchical storage node 620, the specified operation, and the last applied transaction number.

[0051] Transaction log store 250 can perform conflict analysis to determine whether the expected transaction in conflict verification request 640 can be executed without conflicting with committed transactions in the transaction log. Conflicting transactions can occur when another transaction alters the assumed state of a transaction (e.g., reads and uses data for executing the transaction) before the transaction completes and commits (e.g., optimistic parallelism). Conflict verification indication 650 can be provided by transaction log store 210 to hierarchical storage node 620 to indicate whether the transaction has been committed (or whether it conflicted and was therefore rejected). Hierarchical storage node 620 can deliver a commit confirmation (along with a transaction number token indicating the transaction's transaction sequence number) or a requested exception condition or rejection 660.

[0052] Figure 7 This is a sequence diagram illustrating how, according to some embodiments, versions of a hierarchical data structure maintained at a storage node are used to process access requests with snapshot isolation. For a snapshot read request, no access to the transaction log storage area is performed. Instead, the latest or current version of the hierarchical data structure is accessed to serve the read request. Client 710 may send a snapshot read request 730 to hierarchical storage node 720. Hierarchical storage node 720 may recognize that the request is a snapshot read and continue to process the request using the current version of the hierarchical data structure. Then, hierarchical storage node 720 may send a read response 740 indicating the result of the read request.

[0053] Figure 8This is a sequence diagram illustrating the processing of conditional access requests with serializable isolation using a transaction log store according to some implementation schemes. In some cases, access requests can be predicted when conditions in the hierarchical data store are met. For example, the anticipated condition can identify a specific point in time (logical time) or point in time when the hierarchical data structure on the storage node has progressed (e.g., if the last applied sequence number is greater than X). In another example, the anticipated condition can be based on the anticipated data values ​​of nodes, attributes, etc., of the hierarchical data structure. The anticipated condition can be included as part of a conditional access request 830 sent to hierarchical storage node 820 along with the specified operation (e.g., read or write). Hierarchical storage node 820 can evaluate the condition and, if met, continue processing access requests similar to serializable access requests (as described above) by sending a conflict verification request 840 to transaction log store 250 (including the access data of the hierarchical data structure at hierarchical storage node 820, the specified operation, and the last applied transaction number). Figure 6 The transaction log store 250 can then commit or reject conflicting transactions in the conflict verification indication 850. The hierarchical storage node 820 can send an access request response 860, indicating confirmation or rejection of the request and a transaction number token. It can also send a response 860 indicating that the condition failed (e.g., the hierarchical data structure does not meet the condition).

[0054] Figure 9 This is a sequence diagram illustrating the use of a transaction log store to process batch access requests with serializable isolation, according to some implementation schemes. In some scenarios, multiple operations of the same or different types (e.g., read operations, write operations, or read and write operations) can be performed to make large-scale changes to a hierarchical data structure (e.g., moving to rebalance or rearrange subtrees within the hierarchical data structure). Figure 9As shown, client 910 can send a batch access request 930 with specified operations to be submitted or fail together. For example, batch access request 930 may include multiple requests to create different nodes in a hierarchical data structure. In at least some embodiments, the execution of an operation in batch access request 930 may depend on the successful execution of another operation in the batch access request in order to be executed. For example, batch access request 930 may include operations to create three nodes and then connect or link these three nodes together. Additional operations may depend on the success of the creation operation (if it fails, the additional operation will also fail). In another example, data generated as a result of one or more operations (e.g., identifiers or other metadata about the creation of nodes) may be used as input to other operations. Various other operations that depend on the results of other operations in the batch request may be included, so the preceding examples are not intended to be limiting. Hierarchical storage node 920 may submit a conflict verification request 940, which includes multiple operations, the affected / accessed data (e.g., resource nodes, directory nodes, links, etc.), and the transaction number of the last application of the current version of the hierarchical data structure at hierarchical data node 920. Transaction log storage 250 can perform conflict verification on the entire set of operations and commit or reject the entire set of operations in conflict verification indication 950. Tiered storage node 920 can then provide batch response 960 to client 910 to indicate whether the batch operation has been confirmed as committed (including transaction number token) or rejected.

[0055] Figure 10 This is a sequence diagram illustrating, according to some implementations, the processing of client-specified transaction requests with serializable isolation using a transaction log store and transaction state tokens. Instead of grouping similar operations into a single batch, client-specified transactions can be executed, allowing a client to execute multiple access requests as part of the same transaction without having to commit the requests to completion. In this way, various possible or conditional scenarios can be considered, such that if not all access requests are executed as expected, the transaction can be aborted (not shown). Furthermore, transaction state can be delivered from the client to the tiered storage nodes, so that no single storage node must handle the client-specified transaction (e.g., no sticky sessions), but work can be propagated across multiple tiered storage nodes. For example, client 1010 can send request 1040 to tiered storage node 1020 to begin a transaction including multiple initial operations. Tiered storage node 1020 can execute the initial operations and return the results in the form of transaction phase 1050.

[0056] Next, client 1010 can evaluate transaction state 1050 to determine subsequent steps. For example, transaction state 1050 may include data read from the hierarchical data structure, which may or may not be the expected or desired state of client 1010. If so, client 1060 can proceed to commit the transaction and may include an additional operation transaction request 1060 in the commit, which includes a transaction state token (generated from transaction state information 1050) and additional operations (if needed). Because the transaction state token encompasses all the necessary information from another storage node to execute the transaction, load balancing techniques for evenly distributed requests can be implemented, and the commit transaction request 1060 can be routed to hierarchical storage node 1030. Hierarchical storage node 1030 can then generate a conflict verification request 1070 and send it to transaction log storage 250 for conflict assessment. The transaction state token may include the transaction sequence number of the hierarchical data structure at the time the transaction began at hierarchical storage node 1020, and may include the transaction start number (e.g., transaction sequence number) in the conflict verification request. Next, the transaction storage log 210 can send a conflict verification indication 1080 to the tiered storage node 1030, which can then confirm whether an exception condition was triggered in the committed transaction or response 1090. In some implementations, a transaction number token may be included to indicate the sequence number of the committed transaction.

[0057] Figure 11 This is a sequence diagram illustrating the use of page-marking tokens to process access requests with page-marked result delivery according to some implementation schemes. In some cases, the results generated by queries or other access requests are too large to be sent in a single response message. Instead of binding a single hierarchical storage node to certain clients to maintain state at the storage node while the client retrieves data, page-marking technology can be used to allow clients to retrieve results from any storage node without requiring a sticky session with a single storage node. For example, such as Figure 11As shown, client 110 sends a read request 1130 via interface 600, resulting in a large dataset (e.g., a query to retrieve all male user nodes). Instead of providing a read response stream that requires hierarchical storage node 1120 to be idle or maintain a result processing state, read response 1140 may include initial results and result page number tag tokens. Result page number tag tokens may indicate the remaining results to be retrieved and how to retrieve them. In some implementations, the client may modify the result page number tag tokens to change the consistency and / or isolation level used to perform the read operation, such that the remaining results can be modified based on the changed version accessed due to the change in consistency and / or isolation level. Client 1110 may then send another read request 1150 with result page number tag tokens to hierarchical storage node 1120 or another hierarchical storage node. Hierarchical storage node 1120 may then identify additional results based on the information provided in the result page number tag tokens to provide and include them in a subsequent response 1160. This technique may be performed iteratively many times until the entire result set is provided to client 110.

[0058] Figures 2 to 11 The directory storage services, access requests, and other technologies discussed herein provide examples of distributed data storage areas that store hierarchical data structures for clients and provide access to multiple versions of these hierarchical data structures in different scenarios. However, various other types of distributed storage systems can implement multiple versions of hierarchical data structures, utilizing a number of other component types that can provide distributed data storage areas. Figure 12 This is a high-level flowchart illustrating methods and techniques for implementing versioned hierarchical data structures to serve access requests, according to some implementation schemes. Various distributed data storage areas, including the above implementation schemes, can implement the techniques described below.

[0059] As shown in 1210, a version of the hierarchical data structure can be maintained that is consistent with the transaction log of the hierarchical data structure in the distributed data storage area. For example, different numbers of storage nodes can independently maintain local copies or replicas of the hierarchical data structure as part of a replica or protection group. Each storage node can maintain the same or different versions of the hierarchical data structure. In some implementations, storage nodes can implement an eventually consistent consistency model that guarantees each storage node will eventually be consistent with updates to the hierarchical data structure performed on any other storage node. For example, the transaction log of the hierarchical data structure can act as a consistency mechanism, ordering and isolating transactions to prevent conflicting transactions (e.g., using optimistic parallelism). In some implementations, transactions indicated as non-conflicting are committed to the transaction log and reported as the commit point of the committing storage node. In some implementations, the storage node can then apply the transactions, while in other implementations, the storage node can apply the transactions at a later time, as described below regarding... Figure 13 The discussion.

[0060] Over time, storage nodes can obtain many different versions of the transaction log corresponding to different points in time (e.g., logical time). Therefore, when an access request for a hierarchical data structure is received, as indicated in 1220, it may be necessary to select one of the versions to process the access request. Many different types of access requests can be received. For example, some access requests may be queries or other read requests to obtain certain data, attributes, values, or other information stored or represented by the hierarchical data structure (e.g., queries that can be performed using index attributes or queries used to determine the path to a specific node in the directory (e.g., a resource)). As mentioned above... Figures 6 to 11 As shown, access requests can be conditional or batch-based, or a specific isolation level (e.g., serializable or snapshot) can be specified when processing the access request. Access requests can explicitly (e.g., by including a transaction sequence number or token) or implicitly identify the version of the hierarchical data structure to be utilized based on the type of access request (e.g., snapshot reads utilize the current version at the storage node). Therefore, as indicated in 1230, a version of the hierarchical data structure used to serve the request can be identified.

[0061] The request may indicate the isolation level and / or consistency level to be performed when processing the request, as shown in 1240. For example, the type of request may indicate snapshot isolation (e.g., by requesting a specific kind of query or lookup operation). If snapshot isolation is indicated, as shown in the positive exit from 1270, the access request can be served using the identified version of the hierarchical data structure without invoking the transaction log. In this way, some access requests with a low probability of tolerating outdated (but still consistent) data may be able to take advantage of the high speed of performing local operations without further network communication. In at least some embodiments, the identified version for the snapshot isolation request may be maintained in memory for high-speed processing of the request. In some embodiments, the request may indicate that dirty or inconsistent reads of the corresponding portion of the hierarchical data structure are acceptable, thereby indicating an isolation level lower than snapshot isolation. In some embodiments, the request may indicate a specific point in time that provides an isolation level different from snapshot isolation (e.g., by including a transaction sequence number). Various combinations of isolation and consistency, and consistency levels, can be specified by the client, and therefore the preceding examples are not intended to be limiting.

[0062] As indicated by a negative exit from 1240, if snapshot isolation is not specified, the transaction log can be used to enforce serializable isolation. Note that in some implementations, the request may require an explicit (or implicit) invocation of the appropriate isolation level, and therefore may not be (as...) Figure 13 (As shown) Access requests that do not indicate snapshot isolation are always handled with serializable isolation. As indicated in 1210, a new transaction based on the access request can be committed to the transaction log. For example, the affected nodes, objects, and / or other information describing the changes in the transaction can be provided so that conflict detection can be performed. The transaction log can perform conflict detection analysis to determine whether the transaction conflicts with an already committed transaction. If so, as indicated by a positive exit in 1260, an exception condition, rejection, or other error indication can be returned to the client that sent the access request, as shown in 1280. In some implementations, the exception condition can indicate a conflict (including a specific conflicting operation, such as a change to a specific resource node). However, if the transaction does not conflict from the negative exit indication in 1260, an indication that the new transaction has been committed is provided. The commit confirmation can include a sequence number indicating the transaction's position in the transaction log. As part of the service request, the access request can utilize the identified version, as indicated in 1270. For read operations, this can include sending the data found in the identified version. For write operations, serving the access request can include providing an acknowledgment that the write operation has been committed.

[0063] Figure 13This is a high-level flowchart illustrating methods and techniques for maintaining a version of a hierarchical data structure at the storage node in a manner consistent with the transaction log of the hierarchical data structure, according to some implementation schemes. As shown at 1310, in various implementations, the transaction log of the hierarchical data structure can be read. For example, an access request to the transaction log can be executed starting from a transaction sequence number that identifies the latest transaction applied to the current version of the hierarchical data structure at the storage node. As shown at 1320, unapplied transactions of the hierarchical data structure can be identified. For example, the most recently committed transaction (e.g., one with a higher transaction sequence number) can be identified and obtained from the transaction log.

[0064] As indicated at 1330, the identified transaction can then be applied to the current version of the hierarchical data structure at the storage node. For example, a transaction can describe changes performed as part of a transaction (e.g., adding, removing, moving, or modifying resources, directories, attributes, links, or other parts of the hierarchical data structure). The key value or corresponding entries in other storage areas that maintain / describe the changed object can be updated to reflect the change (e.g., by adding additional attributes to the object, changing attribute values, adding new attributes, etc.). Once the identified change is complete, in some implementations, the transaction sequence number corresponding to the applied transaction can then be recorded in metadata or other information indicating the version of the hierarchical data structure (e.g., incrementing the stored sequence number to a higher transaction sequence number).

[0065] As indicated in 1340, in some implementations, historical version information of the hierarchical data structure can be updated to describe changes applied as a result of hierarchical transactions. In this way, previous versions of the hierarchical data structure (before the application of transactions) can be generated, accessed, derived, reconstructed, etc. Figure 3 As discussed in section 4, historical data storage for a hierarchical data structure can be represented as a key-value store that stores the previous version of an object in the current version of the hierarchical data structure. The previous version object can be a copy of an object modified by a transaction, along with a version number to identify the previous object version relative to the current object version. Indexes, links, or index values ​​of previous object versions can be included in the current object version. Similarly, indexes, links, or indices can be traced to object versions earlier than the previous object version (creating a series of object versions) to reach the original version.

[0066] As shown by the backward loop arrow from 1340 (and the negative exit from 1320), the transaction log can be evaluated multiple times to ensure that the version of the hierarchical data structure maintained on the storage node eventually matches the transaction log. For example, transaction log reads can be performed periodically or when an event is triggered (e.g., processing multiple access requests at the storage node).

[0067] In various implementations, the methods described herein can be implemented using any combination of hardware and software. For example, in one implementation, the method can be implemented by a computer system (e.g., such as...). Figure 14 The computer system described herein includes one or more processors that execute program instructions stored on a computer-readable storage medium coupled to the processor. The program instructions can be configured to implement the functions described herein (e.g., the functions of various servers and other components implementing the directory storage service and / or storage service / system described herein). The various methods illustrated in the accompanying drawings and described herein represent example embodiments of the methods. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0068] Furthermore, embodiments of this disclosure may be described in light of the following provisions:

[0069] 1. A system comprising:

[0070] Multiple storage nodes, each including a processor and a memory, store a corresponding version of the hierarchical data structure consistent with the transaction log of the hierarchical data structure;

[0071] A transaction log storage area, which maintains a hierarchical data structure for transaction logs;

[0072] The corresponding storage node is configured as follows:

[0073] Receive a read request from the client for specified data stored as part of a hierarchical data structure;

[0074] Identify one of the corresponding versions of the hierarchical data structure maintained at the storage node in order to serve read requests;

[0075] Access one or more locations at a storage node, said locations storing specified data including those in the identified version to obtain the specified data; and

[0076] The specified data included in the identified version is returned to the client.

[0077] 2. The system according to Clause 1, wherein the respective storage node in the storage node is further configured to:

[0078] Receive write requests for hierarchical data structures from another client;

[0079] Identify one of the corresponding versions of the hierarchical data structure maintained at the storage node in order to serve read requests;

[0080] Execute a write request for the identified version of the hierarchical data structure at the storage node;

[0081] Transactions are sent to the transaction log store based on write requests for use with hierarchical data structures, where the transaction indicates the version of the identified hierarchical data structure; and

[0082] In response to receiving an indication from the transaction log store that a transaction has been committed to the transaction log of the hierarchical data structure, an acknowledgment of the write request is sent to another client.

[0083] 3. The system as described in Clause 2, wherein the respective storage node in the storage node is further configured to:

[0084] Retrieve the transaction from the transaction log store;

[0085] Apply the transaction to the current version of the hierarchical data structure maintained at the storage node; and

[0086] Update historical version data of the hierarchical data structure maintained at the storage node to describe changes applied to the hierarchical data structure due to transactions.

[0087] 4. The system as described in Clause 1, wherein the system is a network-based directory service, wherein the client is a client of the network-based directory service, wherein the storage node is a multi-tenant storing one or more other hierarchical data structures, representing one or more other clients of the network-based directory service, and wherein an identified version of the hierarchical data structure is maintained in the storage node's memory.

[0088] 5. A method, the method comprising:

[0089] Executed by one or more computing devices:

[0090] Receive access requests for a hierarchical data structure maintained as part of a distributed data storage area from the client;

[0091] Identify one of multiple versions of a hierarchical data structure used to serve access requests, wherein multiple versions of the hierarchical data structure, consistent with the transaction log of the hierarchical data structure, are maintained in a distributed data storage area, the hierarchical data structure describing the hierarchical data structure; and

[0092] The identified version of the hierarchical data structure is used to serve access requests.

[0093] 6. As described in Clause 5,

[0094] The reception, identification, and service of access requests are performed by one of multiple storage nodes that store versions of the hierarchical data structure in the distributed data storage area.

[0095] The method further includes:

[0096] Before receiving the access request:

[0097] The storage node reads one or more transactions from the transaction log, wherein the one or more transactions in the transaction log are submitted to the transaction log by a different one of the plurality of storage nodes;

[0098] The storage node applies the one or more transactions to the current version of the hierarchical data structure maintained at the storage node; and

[0099] The identified versions of the hierarchical data structure include one or more transactions.

[0100] 7. As described in Clause 5,

[0101] The method further includes sending a transaction for the hierarchical data structure to the transaction log based on the access request, wherein the transaction indicates an identified version of the hierarchical data structure;

[0102] In response to receiving an instruction from the transaction log to commit a transaction for a hierarchical data structure, the service is executed to handle the access request; and

[0103] The service for access requests includes sending confirmation of the access request to the client.

[0104] 8. The method as described in Clause 7, wherein the access request instructs the execution of multiple operations with respect to different corresponding nodes of the hierarchical data structure, wherein the transaction sent to the transaction log includes multiple operations, and wherein instructing to commit the transaction indicates that multiple operations have been committed.

[0105] 9. The method as described in Clause 8, wherein at least one of the plurality of operations depends on the success of another of the plurality of operations in the access request to be performed.

[0106] 10. The method as described in Clause 5, wherein the access request includes at least one of a specified isolation level or a specified consistency level for processing the access request, wherein the service of the access request or the specified consistency level is performed according to the at least one specified isolation level.

[0107] 11. As described in Clause 5,

[0108] The reception, identification, and service of access requests are performed by one of multiple storage nodes that store versions of the hierarchical data structure in the distributed data storage area.

[0109] The access request is a read request;

[0110] The service for the access request includes sending a portion of the result generated for the service read request from a storage node to the client, along with a page number marker token, where the page number marker token indicates the remaining result to be sent to the client; and

[0111] The method further includes:

[0112] Receive another read request from the client on another storage node, wherein the other read request includes a page number tag token;

[0113] In response to receiving another read request at a different storage node, the remaining results of the other read request are sent to the client, as indicated by the page number marker token.

[0114] 12. As described in Clause 5,

[0115] The access request specifies the snapshot isolation used to handle the access request;

[0116] The method further includes...

[0117] Determine whether to handle access requests based on snapshot isolation; and

[0118] In response, the service for the access request is executed based on the snapshot isolation process.

[0119] 13. As described in Clause 5,

[0120] The reception, identification, and service of access requests are performed by one of multiple storage nodes that store versions of the hierarchical data structure in the distributed data storage area.

[0121] The access request indicates the start of a transaction and one or more operations;

[0122] The service for access requests includes sending one or more operations and a transaction status token from a storage node to the client, where the transaction status token indicates a change in the version of the identified hierarchical data structure due to the following: one or more operations; and

[0123] The method further includes:

[0124] Receive another access request at another storage node, the request including one or more additional operations performed as part of a transaction from the client, wherein the other access request includes a transaction status token, and wherein the other access request indicates that the transaction is to be committed;

[0125] In response to receiving another access request at a different storage node:

[0126] Based on the changes indicated in the transaction status token, modify the identification version of the hierarchical data structure at different storage nodes; and

[0127] Perform one or more additional operations related to the modified version of the hierarchical data structure;

[0128] Send transactions to the transaction log to obtain the hierarchical data structure;

[0129] Receive instructions from the transaction log to commit transactions for the hierarchical data structure; and

[0130] In response to receiving an instruction to commit the transaction from the transaction log, an acknowledgment of the access request is sent to the client.

[0131] 14. A non-transitory computer-readable storage medium storing program instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform:

[0132] A corresponding version of the hierarchical data structure, consistent with the transaction log of the hierarchical data structure, is maintained at multiple storage nodes in the distributed data storage area. The hierarchical data structure describes the changes made to the commit of the hierarchical data structure.

[0133] Receive access requests for hierarchical data structures at a storage node;

[0134] Identify one of the corresponding versions of the hierarchical data structure maintained at a storage node to serve access requests; and

[0135] The identified version of the hierarchical data structure is used to serve access requests.

[0136] 15. The non-transitory computer-readable storage medium as described in Clause 14,

[0137] The program instructions cause the one or more computing devices to further implement sending a transaction from the storage node to the transaction log of the hierarchical data structure based on the access request, wherein the transaction indicates the identified version of the hierarchical data structure;

[0138] In response to receiving an instruction from the transaction log to commit a transaction for a hierarchical data structure, the service is executed to handle the access request; and

[0139] When serving an access request, the program instructions cause one or more computing devices to confirm the sending of the access request.

[0140] 16. The non-transitory computer-readable storage medium as described in Clause 15, wherein the program instructions cause the one or more computing devices to further implement:

[0141] Transactions are acquired at multiple storage nodes.

[0142] Apply the transaction to the corresponding current version of the hierarchical data structure maintained at the storage node; and

[0143] Update the corresponding historical version data of the hierarchical data structure maintained at the storage node to describe the changes applied to the hierarchical data structure as a result of the transaction.

[0144] 17. A non-transitory computer-readable storage medium as described in Clause 14, wherein the program instructions cause the one or more computing devices to further implement:

[0145] Copy the transaction log to an archived version stored in a remote data store; and

[0146] Apply one or more transactions from the remote transaction log to a version of the hierarchical data structure at one of multiple storage nodes to generate a version of the hierarchical data structure at a specified point in time.

[0147] 18. Non-transitory computer-readable storage media as described in Clause 15,

[0148] The access request includes the expected conditions of the hierarchical data structure;

[0149] The program instructions cause one or more computing devices to further perform a version evaluation of the identified hierarchical data structure to determine whether the expected conditions are met.

[0150] In response to determining that the expected conditions are met, the transaction is sent to the transaction log, where access requests that do not include those for which the expected conditions are not determined to be met are rejected.

[0151] 19. The non-transitory computer-readable storage medium as described in Clause 14, wherein the access request includes a read request, wherein the identified version is not the current version of the hierarchical data structure, and wherein, in serving the access request, the program instructions cause one or more computing devices to perform access to historical version data of the hierarchical data structure to read a previous version of one or more nodes of the hierarchical data structure.

[0152] 20. The non-transitory computer-readable storage medium as described in Clause 14, wherein the distributed data storage area is a network-based storage service that provides multiple different service level protocols for different access requests to the network-based storage service, and wherein the program instructions cause the one or more computing devices to further implement routing the access request to the storage node according to one of the different service level protocols identified for processing the access request, wherein the storage node is configured to process the access request according to the determined service level protocol.

[0153] Figure 14 This is a block diagram illustrating a computer system configured to implement a distributed data storage area providing a versioned hierarchical data structure, according to various embodiments, as well as the various other systems, components, services, or devices described above. For example, in different embodiments, computer system 2000 may be configured to implement hierarchical storage nodes that maintain versions of hierarchical data structures or transaction log storage areas that maintain transaction logs of hierarchical data structures. Computer system 2000 can be any type of device, including but not limited to personal computer systems, desktop computers, laptop or notebook computers, mainframe computer systems, handheld computers, workstations, network computers, consumer devices, application servers, storage devices, telephones, mobile phones, or generally any type of computing device.

[0154] Computer system 2000 includes one or more processors 2010 (any of which may include multiple cores and may be single-threaded or multi-threaded) connected to system memory 2020 via input / output (I / O) interface 2030. Computer system 2000 also includes a network interface 2040 connected to I / O interface 2030. In various embodiments, computer system 2000 may be a single-processor system including one processor 2010, or a multiprocessor system including several processors 2010 (e.g., two, four, eight, or other suitable numbers). Processor 2010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of the processors 2010 may, but is not required to, implement the same ISA. Computer system 2000 also includes one or more network communication devices (e.g., network interface 2040) for communicating with other systems and / or components via a communication network (e.g., the Internet, LAN, etc.). For example, a client application executing on system 2000 can use network interface 2040 to communicate with a server application executing on a single server or on a cluster of servers implementing one or more components of the directory storage system described herein. In another example, an instance of a server application executing on computer system 2000 can use network interface 2040 to communicate with other instances of a server application (or another server application) that may be implemented on other computer systems (e.g., computer system 2090).

[0155] In the illustrated embodiments, the computer system 2000 further includes one or more persistent storage devices 2060 and / or one or more I / O devices 2080. In various embodiments, the persistent storage device 2060 may correspond to a disk drive, tape drive, solid-state memory, other mass storage device, or any other persistent storage device. The computer system 2000 (or a distributed application or operating system running thereon) can store instructions and / or data in the persistent storage device 2060 as needed and can retrieve the stored instructions and / or data as needed. For example, in some embodiments, the computer system 2000 may host a storage system server node, and the persistent storage 2060 may include an SSD attached to said server node.

[0156] Computer system 2000 includes one or more system memories 2020 configured to store instructions and data accessible by processor 2010. In various embodiments, system memory 2020 may be implemented using any suitable memory technology (e.g., cache, static random access memory (SRAM), DRAM, RDRAM, EDO RAM, DDR 10 RAM, synchronous dynamic RAM (SDRAM), Rambus RAM, EEPROM, non-volatile / flash memory, or one or more of any other type of memory). System memory 2020 may encompass program instructions 2025 executable by processor 2010 to implement the methods and techniques described herein. In various embodiments, program instructions 2025 may be encoded in platform native binary, any interpreted language (e.g., Java™ bytecode), or any other language (e.g., C / C++, Java™, etc.) or any combination thereof. For example, in the illustrated embodiment, program instructions 2025 include program instructions executable to implement the functionality of a hierarchical storage node that maintains a version of a hierarchical data structure or a component of a transaction log storage area that maintains a transaction log of the hierarchical data structure. In some implementations, Program Directive 2025 can be implemented as multiple separate client, server nodes and / or other components.

[0157] In some embodiments, program instructions 2025 may include instructions executable to implement an operating system (not shown), which may be any of a variety of operating systems, such as UNIX, LINUX, Solaris™, MacOSTM, Windows™, etc. Any or all of the program instructions 2025 may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having instructions stored thereon that can be used to program a computer system (or other electronic device) to perform methods according to various embodiments. The non-transitory computer-readable storage medium may include any means for storing information in a machine-readable (e.g., computer) form (e.g., software, processing applications). Generally, non-transitory computer-accessible media may include computer-readable storage media or storage media such as magnetic or optical media, such as a disk or DVD / catalog storage service 220-ROM connected to the computer system 2000 via an I / O interface. Some non-transitory computer-readable storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., which may be included as system memory 2020 or another type of memory in some embodiments of computer system 2000. In other embodiments, program instructions may be transmitted using optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.) transmitted via communication media such as networks and / or wireless links, for example, through network interface 2040.

[0158] In some implementations, system memory 2020 may include data storage 2045, which may be configured as described herein. For example, information stored by the tiered storage nodes or transaction log storage areas described herein may be stored at different times and in various implementations in data storage 2045 or on another portion of system memory 2020 on one or more nodes, persistent storage 2060, and / or one or more remote storage devices 2070. Typically, system memory 2020 (e.g., data storage 2045 within system memory 2020), persistent storage 2060, and / or remote storage 2070 may store data blocks, copies of data blocks, metadata associated with data blocks and / or their state, database configuration information, and / or any other information that may be used to implement the methods and techniques described herein.

[0159] In one embodiment, I / O interface 2030 may be configured to coordinate I / O traffic between processor 2010, system memory 2020, and any peripheral devices in the system, including via network interface 2040 or other peripheral interfaces. In some embodiments, I / O interface 2030 may perform any necessary protocols, timing, or other data transformations to convert data signals from one component (e.g., system memory 2020) into a format suitable for use by another component (e.g., processor 2010). In some embodiments, I / O interface 2030 may include support for devices connected via various types of peripheral buses, such as the Peripheral Component Interconnect (PCI) bus standard or variants of the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I / O interface 2030 may be divided into two or more separate components, such as a northbridge and a southbridge. Furthermore, in some embodiments, some or all of the functions of I / O interface 2030, such as the interface to system memory 2020, may be directly integrated into processor 2010.

[0160] Network interface 2040 can be configured to allow data exchange between computer system 2000 and other devices connected to the network, such as other computer systems 2090 (which may implement the embodiments described herein). Additionally, network interface 2040 can be configured to allow communication between computer system 2000 and various I / O devices 2050 and / or remote storage 2070. In some embodiments, input / output devices 2050 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or retrieving data by one or more computer systems 2000. Multiple input / output devices 2050 may reside within computer system 2000 or may be distributed across various computer systems 2000 nodes in a distributed system including computer system 2000. In some embodiments, similar input / output devices may be decoupled from computer system 2000 and may interact with one or more nodes in a distributed system including computer system 2000 via wired or wireless connections. Network interface 2040 may collectively support one or more wireless network protocols (e.g., Wi-Fi / IEEE 802.11 or another wireless network standard). However, in various embodiments, network interface 2040 may support communication via any suitable wired or wireless general data network, such as other types of Ethernet networks. Additionally, network interface 2040 may support communication via telecommunications / telephone networks, such as analog voice networks or digital fiber optic communication networks, via storage area networks, such as Fibre Channel SANs, or via any other suitable type of network and / or protocol. In various embodiments, computer system 2000 may include... Figure 14The components shown may include more, fewer, or different components (e.g., display, video card, audio card, peripheral devices, other network interfaces such as ATM interface, Ethernet interface, Frame Relay interface, etc.).

[0161] Note that any distributed system implementation described herein, or any component thereof, can be implemented as one or more network-based services. For example, a database engine head node within the database layer of a database system can present database services and / or other types of data storage services, using the distributed storage system described herein as a client as a network-based service. In some implementations, the network-based service can be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interactions on a network. The network-based service can have an interface described in a machine-processable format, such as Web Services Description Language (WSDL). Other systems can interact with the network-based service in the manner specified by the description of its interface. For example, the network-based service can define various operations that other systems can invoke, and can define specific application programming interfaces (APIs) that other systems can be expected to follow when requesting various operations.

[0162] In various implementations, a web-based service can be requested or invoked using a message that includes parameters and / or data associated with the web-based service request. Such messages can be formatted according to a specific markup language, such as Extensible Markup Language (XML), and / or encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To execute a web-based service request, a web-based service client can assemble a message including the request and transmit the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web-based service, using an Internet-based application-layer transport protocol, such as Hypertext Transfer Protocol (HTTP).

[0163] In some implementations, web-based services can be implemented using representative state transfer (“RESTful”) techniques instead of message-based techniques. For example, a web-based service implemented according to RESTful techniques can be invoked via parameters included in HTTP methods (such as PUT, GET, or DELETE) instead of being encapsulated in a SOAP message.

[0164] The various methods illustrated in the accompanying figures and described herein represent example implementations of the methods. These methods can be implemented manually, in software, in hardware, or in a combination thereof. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0165] Although the above embodiments have been described in considerable detail, many variations and modifications will be apparent to those skilled in the art once the foregoing disclosure is fully understood. The following claims are intended to cover all such modifications and variations, and therefore the above description is to be considered illustrative rather than restrictive.

Claims

1. A system comprising: Multiple storage nodes, including processors and memory, implement hierarchical data storage areas and store corresponding versions of the hierarchical data structure that are consistent with the transaction log of the hierarchical data structure; A transaction log storage area, wherein the transaction log storage area maintains the transaction log of the hierarchical data structure; One of the storage nodes is configured as follows: Receive a request from the client of the hierarchical data storage area to commit a transaction for the hierarchical data structure, wherein the request includes: A transaction state sent to the client by another storage node among the storage nodes, wherein the transaction state is generated by the other storage node among the storage nodes based on one or more previous operations performed by the other storage node among the storage nodes in response to an earlier request sent from the client to the other storage node among the storage nodes to perform one or more previous operations; and One or more additional operations to be performed as part of a transaction initiated in response to an earlier request for another of the plurality of storage nodes; Request conflict verification of a transaction including the one or more previous operations and the one or more additional operations from the transaction log storage area; and In response to determining that the transaction does not conflict based on the conflict verification received from the transaction log store, a response to the request is returned, the response confirming that the transaction has been committed for the hierarchical data structure.

2. The system of claim 1, wherein the request to submit the transaction includes one or more additional operations to be performed by the one storage node as part of the transaction, wherein the requested conflict verification further includes the one or more additional operations, and wherein in response to determining that the transaction does not conflict, the one storage node is further configured to perform the one or more additional operations on a version of the hierarchical data structure stored at the one storage node.

3. The system of claim 1, wherein one of the storage nodes is further configured to: Receive a request to begin a second transaction, which includes one or more additional operations on the hierarchical data structure; Perform the one or more additional operations on the hierarchical data structure; and Return the response to the request, indicating the status of the second transaction, as a transaction status token.

4. The system of claim 1, wherein one of the storage nodes is further configured to: Receive a request to submit another transaction for the hierarchical data structure, the other transaction including a transaction state generated by the other storage node among the plurality of storage nodes based on one or more other previous operations performed by the other storage node among the plurality of storage nodes; Request conflict verification of the other transaction, which includes the other one or more previous operations, from the transaction log storage area; as well as In response to determining that the other transaction has a conflict based on the conflict verification received from the transaction log store, a response indicating that the other transaction failed to commit is returned.

5. The system of claim 1, wherein the system is a network-based directory service, wherein the hierarchical data structure is stored on behalf of a client of the network-based directory service, wherein the storage node is a multi-tenant that stores one or more other hierarchical data structures maintained on behalf of one or more other clients of the network-based directory service, and wherein a version of the identified hierarchical data structure is maintained in the memory of the storage node.

6. A method comprising: Executed by one or more computing devices: At one of the multiple storage nodes of a distributed data store, a request to commit a transaction for a hierarchical data structure is received from a client of the distributed data store, wherein the request includes: A transaction state sent to the client by another storage node among the plurality of storage nodes, wherein the transaction state is generated by the other storage node among the plurality of storage nodes based on one or more previous operations performed by the other storage node among the plurality of storage nodes in response to an earlier request sent from the client to the other storage node among the plurality of storage nodes to perform one or more previous operations; and One or more additional operations to be performed as part of a transaction initiated in response to the earlier request for the other storage node; The storage node requests conflict verification of transactions, including the one or more previous operations and the one or more additional operations, from the transaction log of the hierarchical data structure; and In response to the determination that the transaction does not conflict based on the conflict verification received from the transaction log storage area, the storage node returns a response to the request, the response confirming that the transaction has been committed for the hierarchical data structure.

7. The method of claim 6, wherein the request to submit the transaction includes one or more additional operations to be performed by the storage node as part of the transaction, wherein the requested conflict verification also includes the one or more additional operations, and wherein in response to determining that the transaction does not conflict, the method further includes the storage node performing the one or more additional operations on a version of the hierarchical data structure stored at the storage node.

8. The method according to claim 6, further comprising: A request to begin a second transaction is received at the storage node, the second transaction comprising one or more additional operations on the hierarchical data structure; The storage node performs one or more additional operations on the hierarchical data structure; as well as The storage node returns a response to the request indicating the status of the second transaction as a transaction status token.

9. The method of claim 6, further comprising: At the storage node, a request to submit another transaction for the hierarchical data structure is received, the other transaction including a transaction state generated by the other storage node based on one or more other previous operations performed by the other storage node. The storage node requests conflict verification of the other transaction, which includes the other one or more previous operations, from the transaction log storage area; as well as In response to a determination that the transaction has a conflict based on the conflict verification received from the transaction log storage area, the storage node returns a response indicating that the other transaction has failed to commit.

10. The method of claim 6, wherein the distributed data storage is a network-based directory service, wherein the hierarchical data structure is stored on behalf of a client of the network-based directory service, wherein the storage node is a multi-tenant storing one or more other hierarchical data structures maintained on behalf of one or more other clients of the network-based directory service, and wherein a version of the identified hierarchical data structure is maintained in the memory of the storage node.

11. A non-transitory computer-readable storage medium containing one or more stored program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to perform: At one of the multiple storage nodes in a distributed data store, a request to commit a transaction for a hierarchical data structure is received from a client of the distributed storage, wherein the request includes: A transaction state sent to the client by another storage node among the plurality of storage nodes, wherein the transaction state is generated by the other storage node among the plurality of storage nodes based on one or more previous operations performed by the other storage node among the plurality of storage nodes in response to an earlier request sent from the client to the other storage node among the plurality of storage nodes to perform one or more previous operations; and One or more additional operations to be performed as part of a transaction initiated in response to an earlier request for another of the plurality of storage nodes; The storage node requests conflict verification of transactions, including the one or more previous operations and the one or more additional operations, from the transaction log of the hierarchical data structure. as well as In response to the determination that the transaction does not conflict based on the conflict verification received from the transaction log storage area, the storage node returns a response to the request, the response confirming that the transaction has been committed for the hierarchical data structure.

12. The one or more non-transitory computer-readable storage media of claim 11, wherein the request to submit the transaction includes one or more additional operations to be performed by the storage node as part of the transaction, wherein the requested conflict verification further includes the one or more additional operations, and wherein in response to determining that the transaction does not conflict, the one or more non-transitory computer-readable storage media also stores program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to also perform: the one or more additional operations by the storage node on a version of the hierarchical data structure stored at the storage node.

13. The one or more non-transitory computer-readable storage media of claim 11, further storing instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to also perform: A request to begin a second transaction is received at the storage node, the second transaction comprising one or more additional operations on the hierarchical data structure; The storage node performs one or more additional operations on the hierarchical data structure; as well as The storage node returns a response to the request indicating the status of the second transaction as a transaction status token.

14. The one or more non-transitory computer-readable storage media of claim 11, further storing instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to also perform: At the storage node, a request to submit another transaction for the hierarchical data structure is received, the other transaction including a transaction state generated by the other storage node based on one or more other previous operations performed by the other storage node. The storage node requests conflict verification of the other transaction, which includes the other one or more previous operations, from the transaction log storage area; as well as In response to determining that the transaction has a conflict based on the conflict verification received from the transaction log storage area for the other transaction, the storage node returns a response indicating that the other transaction failed to commit.

15. One or more non-transitory computer-readable storage media as claimed in claim 11, wherein the distributed data storage is a network-based directory service, wherein the hierarchical data structure is stored on behalf of a client of the network-based directory service, wherein the storage node is a multi-tenant storing one or more other hierarchical data structures maintained on behalf of one or more other clients of the network-based directory service, and wherein a version of the identified hierarchical data structure is maintained in the memory of the storage node.

Images