Techniques for achieving cache consistency across distributed storage clusters

Abstract

A leader control plane node in a set of control plane nodes of a node cluster can receive a request to store data in a distributed storage system that includes a set of access manager nodes. The leader control plane node can generate cache data that identifies an instruction from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instruction instructing the one or more access manager nodes to store data indicated in the request. The leader control plane node can then send a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data to respective caches of the one or more follower control plane nodes.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 543,064, filed December 6, 2021, entitled “TECHNIQUES TO ACHIEVE CACHE COHERENCY ACROSS DISTRIBUTED STORAGE CLUSTERS,” the entire disclosure of which is hereby incorporated by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to server systems and data processing, and more specifically, to techniques to achieve cache coherency across distributed storage clusters. BACKGROUND

[0004] Many users can employ a cloud platform (i.e., a computing platform for cloud computing) to use a shared network of remote servers to store, manage, and process data. Users can develop applications on the cloud platform to address the storage, management, and processing of data. In some cases, the cloud platform can utilize a multi-tenant database system. Users can access the cloud platform using various user devices (e.g., desktop computers, laptop computers, smartphones, tablet computers, or other computing systems, etc.). Computer networks allow for the transmission of data between interconnected computers. Data storage techniques using distributed storage clusters can be improved. SUMMARY

[0005] A method for data storage is described. The method can include receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes; generating, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; and sending, by the leader control plane node, replication instructions to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0006] An apparatus for data storage is described. The apparatus can include a processor, a memory coupled with the processor, and instructions stored in the memory. The instructions executable by the processor to cause the apparatus to: receive, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes; generate, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; and send, by the leader control plane node, replication instructions to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0007] Another apparatus for data storage is described. The apparatus can include means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes; means for generating, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; and means for sending, by the leader control plane node, replication instructions to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0008] A non-transitory computer-readable medium storing code for data storage is described. The code can include instructions executable by a processor to: receive, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes; generate, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; and send, by the leader control plane node, replication instructions to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0009] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for indicating, in accordance with the instructions, the one or more access manager nodes to store the data in the distributed storage system. In some examples of the method, apparatuses, and non-transitory computer-readable medium described herein, generating the cache data can include operations, features, means, or instructions for creating a replication log entry in a cache of the leader control plane node, the replication log entry including instructions for the one or more access manager nodes of the distributed storage system to store the data indicated in the request.

[0010] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for receiving, from the one or more access manager nodes, an acknowledgement message indicating that the data indicated in the request can have become data stored by the one or more access manager nodes, and updating the replicated log entry of the cache to indicate that the data indicated in the request can have become data stored by the one or more access manager nodes of the distributed storage system.

[0011] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for sending, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node was selected to be a new leader control plane node based at least in part on a failure of the leader control plane node.

[0012] In some examples of the method, apparatuses, and non-transitory computer-readable medium described herein, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples of the method, apparatuses, and non-transitory computer-readable medium described herein, the request to store the data in the distributed storage system includes a write request.

[0013] A method for data storage is described. The method can include receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node; receiving, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on a failure of the leader control plane node; processing, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request; and instructing the one or more access manager nodes of the distributed storage system to store the data according to the instructions retrieved from the cache.

[0014] An apparatus for data storage is described. The apparatus can include a processor, a memory coupled with the processor, and instructions stored in the memory. The instructions executable by the processor to cause the apparatus to receive, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node; receive, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on a failure of the leader control plane node; process, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request; and instruct the one or more access manager nodes of the distributed storage system to store the data according to the instructions retrieved from the cache.

[0015] Another apparatus for data storage is described. The apparatus can include means for receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node; means for receiving, at the follower control plane node, an indication that the follower control plane node is selected as a new leader control plane node based at least in part on a failure of the leader control plane node; means for processing, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request; and means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instructions retrieved from the cache.

[0016] A non-transitory computer-readable medium storing code for data storage is described. The code can include instructions executable by a processor to receive, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node; receive, at the follower control plane node, an indication that the follower control plane node is selected as a new leader control plane node based at least in part on a failure of the leader control plane node; process, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request; and instruct the one or more access manager nodes of the distributed storage system to store the data in accordance with the instructions retrieved from the cache.

[0017] Some examples of the method, apparatuses, and non-transitory computer- readable medium described herein can further include operations, features, means, or instructions for processing the cache to identify a replication log entry including instructions to store the data indicated in the request via the one or more access manager nodes of the distributed storage system; and instructing the one or more access manager nodes of the distributed storage system to store the data indicated in the request based at least in part on the replication log entry.

[0018] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for receiving, from the one or more access manager nodes, an acknowledgement message indicating that the data indicated in the request has become data stored by the one or more access manager nodes.

[0019] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for receiving, at the new leader control plane node, a second request to store data in the one or more access manager nodes; generating, by the new leader control plane node, second cache data, the second cache data identifying second instructions from the new leader control plane node to the one or more access manager nodes, the second instructions indicating that the one or more access manager nodes store data indicated in the second request; and sending, by the new leader control plane node, second replication instructions to one or more follower control plane nodes of the new leader control plane node to replicate the second cache data into respective caches of the one or more follower control plane nodes.

[0020] Some examples of the method, apparatuses, and non-transitory computer-readable medium described herein can further include operations, features, means, or instructions for indicating that the one or more access manager nodes store data in the distributed storage system in accordance with the second instructions.

[0021] In some examples of the method, apparatuses, and non-transitory computer- readable medium described herein, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples of the method, apparatuses, and non-transitory computer-readable medium described herein, the second cache of the new leader control plane node is configured to enable cluster failover for a defined amount of time. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 An example of a data storage system that supports techniques to enable cache consistency across a distributed storage cluster is shown in accordance with aspects of the present disclosure.

[0023] Figure 2 An example of a storage architecture that supports techniques to enable cache consistency across a distributed storage cluster is shown in accordance with aspects of the present disclosure.

[0024] Figure 3An example of a recovery architecture that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0025] Figure 4 An example of a distributed consensus protocol that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0026] Figure 5 An example of a process flow that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0027] Figure 6 A block diagram of an apparatus that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0028] Figure 7 A block diagram of a cache replication component that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0029] Figure 8 A diagram of a system that includes a device that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure.

[0030] Figures 9 to 12 A flow diagram of a method that supports techniques to enable cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0031] A platform of an online marketplace can use a distributed storage architecture to store data that scales with increasing data volume. The design goal of this storage architecture is to provide high throughput and low latency storage access. In some examples, the storage architecture can be implemented using a neuron cluster that includes a set of control plane nodes that store and replicate data for storage across data centers at different geographic locations. The set of control plane nodes can coordinate with a set of data plane nodes, referred to herein as access manager storage nodes. The access manager nodes can be used to store data in underlying hard disks controlled by respective access manager storage nodes. The neuron cluster can be designed to continue to function even if one of the control plane nodes fails. The neuron cluster can use a distributed consensus protocol to select a leader control plane node. With the leader control plane node elected, the other control plane nodes of the cluster are referred to as follower control plane nodes.

[0032] In some examples, a leader control plane node can manage a subset of storage nodes. Additionally or alternatively, the leader control plane node can instruct its storage nodes to perform data placement on underlying storage disks, and can coordinate with its storage nodes to retrieve data. The leader control plane node can coordinate with its subset of access manager storage nodes to store data at a first data center, and can coordinate with follower control plane nodes to replicate data for a second subset of access manager nodes to store data at a second data center. If there is a failure of a control plane node, the neuron cluster can select one of the follower control plane nodes to assume the leader role. In some data storage systems, when a leader fails, a cluster manager can read a local database and reconstruct at least some or all of the cluster state of the failed leader control plane node. The cluster manager can send a leader change request to the storage nodes, the leader change request indicating that the leader has changed from an old leader to a new control plane node. In addition, the cluster manager can provide information associated with the new leader to the storage nodes. The storage nodes can use this information for future communications with the new leader. However, this reconfiguration can take a long time (e.g., several minutes), and during this time, the storage nodes can not be able to communicate with the neuron cluster, resulting in latency.

[0033] One or more aspects of the present disclosure can provide cache data replication across multiple control plane nodes. Cache data can include a mapping between incoming data and access manager nodes that store the data. In some examples, a neuron cluster can include multiple control plane nodes (referred to as “neuron nodes”), where one control plane node can be a leader control plane node and the remaining control plane nodes can be follower control plane nodes. The leader control plane node can receive a request to store data in a distributed storage system. Upon receiving the request, the leader control plane node can generate and store cache data in its cache, the cache data identifying instructions for one or more access manager storage nodes of the distributed storage system to store or retrieve data indicated in the request at a first data center. For example, the cache data stored in the cache can be a set of log entries that indicate a set of one or more operations performed by the leader control plane node and its access manager storage nodes, and can indicate whether a particular operation has been completed. The leader control plane node can also replicate the request for the follower nodes to perform the same operations at a second data center on access manager storage nodes of its distributed storage system, such that data is replicated and stored at different data centers.

[0034] According to some aspects, a leader control plane node can replicate its cached data used for storage to the caches of one or more follower control plane nodes. By replicating the access instructions in each cache of each follower control plane node, the need to reconstruct at least one or all cluster states by a cluster manager when a leader fails before implementing a leadership change is reduced or eliminated, and thus the corresponding latency is reduced or eliminated. As described herein, each follower control plane node can include a copy of the cache of the leader control plane node (e.g., the cache can be consistent across the nodes of the cluster). In some examples, a new leader node can thus continue from where the old leader node stopped using its cache, resulting in reduced latency. Thus, the failover response time resulting from replicating the leader cache at each follower node provides reduced latency during a leadership change (e.g., when the current leader node fails). Thus, maintaining cache consistency between the leader control plane node and the follower control plane nodes can result in faster leader switching when a leader node fails, enabling the distributed storage system to continue a nominal interruption of storage replication operations.

[0035] Aspects of the disclosure are initially described in the context of an environment that supports on-demand database services. Aspects of the disclosure are further described in the context of storage architecture, recovery architecture, distributed consensus protocol, and processing flow. Aspects of the disclosure are further illustrated and described in reference to apparatus diagrams, system diagrams, and flow diagrams related to techniques that enable cache consistency across a distributed storage cluster.

[0036] Figure 1 An example of a system 100 for cloud computing that supports techniques to enable cache consistency across a distributed storage cluster is shown in accordance with various aspects of the present disclosure. The system 100 includes cloud clients 105, user devices 110, a cloud platform 115, and data centers 120. The cloud platform 115 can be an example of a public or private cloud network. The cloud clients 105 can access the cloud platform 115 through a network connection 135. The network can implement the Transmission Control Protocol and Internet Protocol (TCP / IP), such as the Internet, or can implement other network protocols. The cloud clients 105 can be examples of computing devices, such as a server (e.g., cloud client 105-a), a smartphone (e.g., cloud client 105-b), or a laptop computer (e.g., cloud client 105-c). In other examples, the cloud clients 105 can be a desktop computer, a tablet computer, a sensor, or another computing device or system capable of generating, analyzing, sending, or receiving communications. In some examples, the cloud clients 105 can be part of a business, enterprise, non-profit organization, start-up, or another organization type.

[0037] The cloud client 105 can facilitate communication between the data center 120 and one or more user devices 110 to enable an online marketplace. The network connections 130 can include communications, opportunities, purchases, sales, or other interactions between the cloud client 105 and the user devices 110. The cloud client 105 can access the cloud platform 115 to store, manage, and process data communicated via the one or more network connections 130. In some cases, the cloud client 105 can have an associated security or permission level. Based on the associated security or permission level, the cloud client 105 can access certain applications, data, and database information within the cloud platform 115 and can not have access to other applications, data, and database information.

[0038] The user devices 110 can interact with the cloud client 105 through the network connections 130. The network can implement the Transmission Control Protocol and Internet Protocol (TCP / IP), such as the Internet, or can implement other network protocols. The network connections 130 can facilitate data transfer via email, web pages, text messages, mail, or other appropriate forms of electronic interaction via a computer network (e.g., network connections 130-a, 130-b, 130-c, and 130-d). In an example, the user devices 110 can be computing devices such as a smartphone 110-a, a laptop computer 110-b, and can also be a server 110-c or a sensor 110-d. In other cases, the user devices 110 can be another computing system. In some cases, the user devices 110 can be operated by a user or a group of users. The user or group of users can be a customer associated with a business, a manufacturer, or other appropriate organization.

[0039] The cloud platform 115 can provide on-demand database services to the cloud clients 105. In some cases, the cloud platform 115 can be an example of a multi-tenant database system. In this case, the cloud platform 115 can serve multiple cloud clients 105 with a single software instance. However, other types of systems can be implemented, including but not limited to client-server systems, mobile device systems, and mobile network systems. In some cases, the cloud platform 115 can support online applications. This can include support for sales between buyers and sellers operating user devices 110, services, marketing of products posted by buyers, community interactions between buyers and sellers, analytics (e.g., user interaction metrics), applications (e.g., computer vision and machine learning), and the Internet of Things. The cloud platform 115 can receive data associated with the generation of online marketplaces from the cloud clients 105 over the network connection 135 and can store and analyze the data. In some cases, the cloud platform 115 can receive data directly from the user devices 110 and the cloud clients 105. In some cases, the cloud clients 105 can develop applications to run on the cloud platform 115. The cloud platform 115 can be implemented using remote servers. In some cases, the remote servers can be located at one or more data centers 120.

[0040] The data centers 120 can include multiple servers. The multiple servers can be used for data storage, management, and processing. The data centers 120 can receive data from the cloud platform 115 via the connection 140, or directly from the cloud clients 105 or via the network connection 130 between the user devices 110 and the cloud clients 105. The data centers 120 can utilize multiple redundancies for security purposes. In some cases, data stored at the data centers 120 can be backed up by data replicas at different data centers (not shown).

[0041] The server system 125 can include the cloud clients 105, the cloud platform 115, the cache replication component 145, and the data centers 120, which can coordinate with the cloud platform 115 and the data centers 120 to implement online marketplaces. In some cases, data processing can occur at one or more components of the server system 125, or at a combination of the components. In some cases, a server can perform data processing. The server can be a cloud client 105 or located at the data centers 120.

[0042] The cache replication component 145 can communicate with the cloud platform 115 via the connection 155 and can also communicate with the data centers 120 through the connection 150. The cache replication component 145 can receive signals and inputs from the user devices 110 via the cloud clients 105 and via the cloud platform 115 or the data centers 120.

[0043] Some e-commerce systems can provide a marketplace with billions of products listed by millions of sellers. The volume of new data creation can continue to grow, and large amounts of new data can be created every day. Such data can include product listing information, media files, various logs produced by services in production, machine learning, and analytics workloads, and / or other auxiliary sources that monitor these transactions. In some cases, some data can be ephemeral, while other data can persist for long periods of time to satisfy regulatory compliance. Such different use cases can be associated with different service level objectives in terms of latency and throughput for the underlying storage. In some aspects, some storage systems can provide block storage access.

[0044] Some conventional data storage solutions can provide a data storage architecture that includes a plurality of storage nodes. The data storage architecture can be implemented by a neuron cluster that includes a set of control plane nodes. The set of control plane nodes can store data in one or more access manager storage nodes. Nodes as described herein can be implemented by hardware, software, or a combination of both. Further, the neuron cluster can select one of the control plane nodes as a leader control plane node, while the remaining control plane nodes are designated as follower control plane nodes. In some examples, the leader control plane node can coordinate with its subset of access manager storage nodes to store data at a first data center, and can coordinate with the follower control plane nodes to replicate the data for a second subset of access manager nodes to store data at a second data center. However, if the leader control plane node fails, it can be challenging to seamlessly continue data storage operations. That is, upon failure of the leader control plane node, the neuron cluster selects a new control plane node as a new leader. In this case, the newly elected control plane node can not be aware of the cache of the previous leader control plane node. Thus, when the leader changes from the old leader to the new leader, reconfiguration can take a long time, resulting in latency.

[0045] In contrast, the system 100 implements processes and techniques for achieving cache consistency across a distributed storage cluster. In particular, the server system 125 can implement operations similar to those described herein. As described herein, one or more components of the server system 125, including the cache replication component 145, can operate to perform cache replication across multiple storage nodes. At a leader control plane node in a set of control plane nodes of a node cluster, the cache replication component 145 within the server system 125 can receive a request to store data in a distributed storage system including a set of access manager nodes. The cache replication component 145 within the server system 125 can generate, by the leader control plane node, cache data that identifies an instruction from the leader control plane node to one or more access manager nodes of the set of access manager nodes managed by the leader control plane node. In some examples, the instruction can instruct the one or more access manager nodes to store the data indicated in the request. The server system 125 and the cache replication component 145 can send, by the leader control plane node, the replication instruction to one or more follower control plane nodes in the set of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes. The cache data can include a mapping between incoming data and an access manager node that stores the data. In particular, the cache data can include an instruction for the access manager node to store the data. The instruction can be in the form of a replication log entry. For high availability and fault tolerance, the cluster manager (implemented by the cache replication component 145) can request replication of storage node state machine transitions to follower nodes. The leader node can serialize the replication log entry and store the replication log entry into a cache of the leader node. The leader node can then send the replication log entry to one or more follower nodes, where the log entry is deserialized and applied to a local key-value store.

[0046] In some examples, the cache data can indicate one or more access manager nodes of the distributed storage system that are managed by the leader node. For example, the cache data can include an indication of one or more access manager nodes used by the leader node to store data. Additionally or alternatively, the cache data can identify one or more storage volumes of the distributed storage system that are managed by the leader node, one or more storage pools of the distributed storage system that are managed by the leader node, or any combination thereof. A storage pool can include one or more storage nodes (and / or access manager nodes). In some examples, a storage pool can be an aggregation of storage from different failure domains. A failure domain can represent an aggregated storage resource under a common top of a rack switch. Additionally or alternatively, a storage pool can be used to store data from different tenants. For example, a storage pool can be uniquely identified by a combination of its name and an associated tenant identifier.

[0047] In some examples, the server system 125 and the cache replication component 145 can receive, at a follower control plane node of a set of control plane nodes of a cluster of nodes from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node. The server system 125 and the cache replication component 145 can receive, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based on a failure of the leader control plane node. The server system 125 and the cache replication component 145 can process the cache by the new leader control plane node to retrieve instructions from the leader control plane node to one or more access manager nodes of the distributed storage system. In some cases, the instructions can indicate that the one or more access manager nodes managed by the leader control plane node store data indicated in a request. The server system 125 and the cache replication component 145 can then instruct the one or more access manager nodes of the distributed storage system to store the data according to the instructions retrieved from the cache.

[0048] As one example, reduced availability of control plane nodes can increase failover response time by creating several local caches within each node (one local cache per state machine (e.g., storage node, storage volume, and storage pool)) that are updated from leader nodes to follower nodes when a state machine transition occurs at a leader node. Aspects of the present disclosure can use replication techniques and can schedule background threads to make various caches consistent across control plane nodes. This cache consistency between control plane nodes and storage nodes can effectively flip a few bits during a leader change operation to enable the cluster manager to start accepting incoming data and metrics traffic. With this change, the system 100 can reactivate cluster management operations almost immediately after a leader failover. For example, when a leader control plane node fails, and another control plane node becomes the leader, then the caches on that new leader node help the cluster manager achieve failover within a defined amount of time (e.g., instant cluster failover or cluster failover in a short amount of time).

[0049] Those skilled in the art will appreciate that one or more aspects of the present disclosure can be implemented in the system 100 to additionally or alternatively address other problems other than those described above. In addition, aspects of the present disclosure can provide technical improvements to “conventional” systems or processes as described herein. However, the specification and drawings only include example technical improvements resulting from implementing aspects of the present disclosure, and thus do not represent all technical improvements provided within the scope of the claims.

[0050] Figure 2 An example of a storage architecture 200 that supports techniques to implement cache consistency across a distributed storage cluster is shown in accordance with aspects of the present disclosure. In Figure 2 In an example, the storage architecture 200 is implemented using a neuron cluster 205. The neuron cluster 205 includes a set of servers (e.g., one or more servers, and in some cases can include five or seven servers). Such servers can be referred to as replicas. Each server can include a storage node, a cache, a database, or any combination thereof. The neuron cluster 205 can provide a storage architecture in which one or more replicas can continue group operations even in the event of a failure of at least one replica. If a replica fails, the storage architecture can provide for the automatic generation of a replacement replica. The replacement replica can then be added to the neuron cluster 205 (as a follower control plane node).

[0051] In some aspects, the cluster of neurons 205 can use a distributed consensus protocol to select a leader control plane node, and the other control plane nodes of the cluster are referred to as follower control plane nodes. During bootstrapping of the cluster of neurons 205, a workflow engine (bootstrap service 210) can create replicas, and each replica can include the ability to bootstrap as a leader control plane node. In some examples, the workflow engine can randomly select one of the replicas and designate it as the leader of the cluster of neurons. The workflow engine can add the remaining replicas to the leader control plane node as follower control plane nodes. In Figure 2 In examples, the cluster of neurons 205 can include node 1, node 2, and node 3 (although any number of nodes can be contemplated). The workflow engine can select node 1 as the leader control plane node and node 2 and node 3 as the follower control plane nodes. Once the cluster of neurons 205 is formed, the storage architecture 200 can implement a protocol for managing cluster operations. The storage architecture 200 can provide replication of log entries to enable faster failover in the event of a change in the leader node. Each node can have a local key-value store (database 1, database 2, and database 3) in which state machines are persisted. In some examples, the leader control plane node can initiate state machine replication, and subsequent read and write operations can flow from the leader node to the follower nodes.

[0052] As shown in Figure 2 The neuron architecture (e.g., the control plane service for neuron replication) can include five components: a cluster manager, a provider, a storage pool manager, an API server, and a raft manager. The cluster manager can configure cluster membership of storage nodes. The provider can manage storage volume related operations and can perform intelligent data placement. The storage pool manager can classify storage nodes based on tenants. The API server can provide external API interfaces. The raft manager can implement consensus protocols for leader election, can provide fault tolerance through state machine replication of cluster metadata, and can manage a consistent cache across all replicas.

[0053] As described in the example of Figure 2 The bootstrap service 210 can initiate the formation of the control plane nodes into the cluster of neurons 205. The bootstrap service 210 can then initiate one or more storage nodes (also referred to as access manager nodes). The set of control plane nodes (node 1, node 2, and node 3) can coordinate with the set of collection data plane nodes (referred to herein as access manager nodes) for storing data in underlying hard disks controlled by the respective access manager nodes. As described in the example of Figure 2As described in the example of , the bootstrap service 210 initiates access manager nodes (access managers 1 to N) in storage pool 1. The bootstrap service 210 can also initiate multiple storage pools (storage pools 1 to N). In some examples, the storage nodes associated with each access manager can register with a cluster manager running on a leader control plane node. In the example where the leader control plane node is node 1, the storage nodes register with node 1. The cluster manager residing on node 1 can assign a unique identifier to each of these nodes and can grant them a lease, which the storage node renews over a period of time (e.g., every few seconds). The storage manager node or access manager node can include three main components: a storage engine (also known as a Homestore), a control plane component, and a storage target. The storage engine can be a high-performance engine built using scalable b+ trees. The control plane component can have a server running to automatically create a storage volume that includes replicas and partitions located on several different storage nodes. In some examples, the storage engine can be built using a fully asynchronous threading model. The storage engine can run in user space and can be configured to bypass the kernel and use stream-based allocation (where the storage engine uses different types of block allocation schemes to balance speed of allocation, write amplification, and fragmentation).

[0054] According to aspects described in the present disclosure, a leader control plane node (node ​​1) may receive a request to store data in a distributed storage system comprising a set of access manager nodes (access manager 1, access manager 2, and access manager 3). For example, the leader control plane node may receive a request to store data from one or more compute nodes (compute nodes 1 to M). The leader control plane node (node ​​1) may generate cache data that identifies an instruction from the leader control plane node to one or more access manager nodes in the set of access manager nodes that are managed by the leader control plane node. Node 1 may generate cache data including the instruction and may store the cache data in cache 1. A cache as described herein may refer to a memory or other computer storage device capable of storing instructions, data, information, etc. In some examples, the instruction may instruct the one or more access manager nodes to store the data indicated in the request. For example, node 1 may receive a request to store data portions P1, P2, and P3. These instructions may instruct: access manager 1 to store data portion P1, access manager 2 to store data portion P2, and access manager 3 to store data portion P3. The leader control plane node (node ​​1) may then send a replication instruction to one or more follower control plane nodes to replicate the cached data into the corresponding caches of the one or more follower control plane nodes. Figure 2In the example of FIG. 1, node 1 can instruct node 2 and node 3 to replicate node 1’s cache data into respective caches of node 2 and node 3. Upon receiving the replication instruction, cache 2 of node 2 and cache 3 of node 3 can include instructions instructing the access manager nodes (access manager 1, access manager 2, and access manager 3) to store the data indicated in the request to node 1. By replicating the instructions across multiple caches, the techniques described herein provide for maintaining cache consistency across the set of control plane nodes to quickly establish a new leader node in the event of a failure of a leader node, resulting in reduced latency.

[0055] As described herein, storage architecture 200 can represent a central repository for a storage service. Storage architecture 200 can maintain storage cluster metadata and can replicate the metadata across multiple nodes. The number of replicas can depend on the amount of fault tolerance desired in each deployment environment.

[0056] Figure 3 An example of a recovery architecture 300 that supports techniques to implement cache consistency across a distributed storage cluster is shown, in accordance with aspects of the present disclosure. Recovery architecture 300 can be implemented using a neuron cluster 305 and a state machine 310. State machine 310 can include a local database 315 and an in-memory cache 320. As described with reference to Figure 1 and Figure 2 Neuron cluster 305 maintains a distributed storage system that includes a set of control plane nodes. Neuron cluster 305 can designate one control plane node as a leader control plane node and other control plane nodes as follower control plane nodes. For example, during bootstrapping, an administrator can select a leader and add followers to an existing leader. Follower control plane nodes can each maintain a copy of the cache data of the leader control plane node. If the leader changes, the new leader can broadcast information indicating the change to one or more registered storage nodes so that subsequent requests can be sent to the new leader. In some examples, the techniques described herein can use state machine replication techniques to maintain consistency between different control plane nodes. Each committed operation in a log can be treated as an action on a state machine.

[0057] For the recovery architecture 300, at 325, the recovery process can read the local database 315. At 330, the recovery process can write the contents to the in-memory cache 320. At 335, the recovery process can examine the incremental log entries 305 from the neuron cluster. Based on the examination of the incremental log entries, at 340, the recovery process can capture the incremental logs from the most recent local snapshot point. At 345, the recovery process can apply one or more missing committed log entries to the state machine 310. Thus, the recovery architecture 300 can use a replacement policy to manage the in-memory cache 320 to store both the access manager node and the storage volume related information (e.g., metadata) associated with each log entry to, for example, a newly elected leader to complete one or more outstanding tasks that correspond to log entries that were not completed by the previous leader due to, for example, a failure of the previous leader. For successful commits to the state machine 310, the updates are reflected in the in-memory cache 320 in addition to the local database 315.

[0058] In some examples, the metadata size can be small enough to keep the metadata in memory without a replacement policy. If one or more control plane nodes are restarted in the neuron cluster 305, it can first read the tables at the local database 315 (at 325), and then can load the metadata information into the in-memory cache 320. The neuron cluster 305 can then participate in the communication (e.g., API, consensus, etc.). Thereafter, the neuron cluster 305 can implement a protocol to check the commit sequence number with other replicas. The neuron cluster 305 can then synchronize with one or more other replicas by applying the uncommitted log entries to the state machine 310 (i.e., both the local database 315 and the in-memory cache 320).

[0059] Figure 4 An example of a distributed consensus protocol 400 that supports techniques to implement cache consistency across distributed storage clusters is shown in accordance with aspects of the present disclosure. According to the distributed consensus protocol 400, a neuron cluster can select one control plane node as a leader node and can select one or more control plane nodes (different from the leader node) as follower control plane nodes. The distributed consensus protocol 400 can employ a strategy where there is one leader at a time. The leader node can accept incoming write requests, and can then replicate the write requests to the follower nodes. The incoming write requests can correspond to individual log entries that are stored in separate log sections. The log entries can be associated with unique index numbers (e.g., log numbers) in a monotonically increasing order.

[0060] As Figure 4As described in the example of FIG. 4, the cluster of neurons can select a leader node 405, a first follower node 410, and a second follower node 415. The leader node 405 can receive a write request. Once the leader node receives the write request, at 420, the leader node 405 can append a log entry to its log section. For example, the leader node 405 can append log 5. At 425, the leader node 405 can replicate the request to the follower nodes (e.g., the first follower node 410 and the second follower node 415). Each follower node can receive the replication request from the leader node 405. At Figure 4 In the example of FIG. 4, the first follower node 410 and the second follower node 415 can receive the log entry (log 5). At 430, each follower node can append the log to its log section. For example, the first follower node 410 and the second follower node 415 can append log 5 and commit log 4. At 435, the follower nodes can return a response (e.g., an acknowledgement) to the leader node 405. The distributed consensus protocol 400 can use a write strategy where, once the leader node 405 obtains a response from a majority of the follower nodes, including itself, the leader node 405 commits the log entry and applies the log entry to the state machine. At 440, the leader node 405 can commit log 6.

[0061] After the commit, the leader node 405 can replicate the committed log number to the follower nodes, and the follower nodes can then commit the log corresponding to the given log number and apply it to their state machines. In some examples, the log replication request can include the committed log number of the previous log replication, such that they are always pipelined. The leader node 405 can periodically send one or more updates to the follower nodes (e.g., the follower node 410 and the follower node 415). In some examples, each follower node can maintain a random timer, the lower bound of which can be greater than the update interval. When a follower node receives an update, the follower node can reset its timer with a new random expiration time. If the timer expires due to no updates for a threshold period of time, the follower node can determine that the leader node has failed and can request a vote from all other follower nodes to select a next leader. Once a majority of the nodes vote for a node to initiate the selection, that node becomes the leader and starts serving write requests. In some examples, the node that initiates the selection can become the leader based on a threshold number of nodes voting for that node. In other examples, a node can become the selected leader based on the node having sufficient or the greatest processing bandwidth, the node having sufficient processing power to perform leader operations, the node being able to meet at least one latency criterion in performing leader operations, etc.

[0062] In some examples, a conflict can occur during a time when the leader node is not present or the network is disconnected. To resolve the conflict, techniques described herein can provide a counter referred to as a term that is incremented when a new leader election is initiated. In some examples, the follower nodes are configured to have a term value that is the same as the term value of the current leader. Each replication request or update (from the leader node) can include the term value of the current leader node. Additionally or alternatively, each log entry can retain a term value of the time when the log entry was produced. If the current leader receives a message with a higher term value from a follower node, the message with the higher term value can mean that a new leader has been elected by a majority of the nodes. In this case, the leader node can immediately relinquish the leadership role and can become a follower. A conflict can occur when the leader node successfully appends a new log entry to its log portion, but fails to replicate the new log entry to the followers due to various reasons such as server crash or network isolation. In this case, a new leader is elected, and the new leader can serve the new log. If the previous leader recovers from the failure and rejoins as a follower, the previous leader and the current leader can see different log entries with the same log number. In this case, the previous leader can find the most recent log entry whose term is the same as the term of the corresponding log entry in the current leader. The previous leader can then initiate overwriting log entries from that point using the current leader’s log entries.

[0063] Figure 5 An example of a process flow 500 that supports techniques to implement cache coherency across distributed storage clusters is shown in accordance with aspects of the present disclosure. The process flow 500 can include a user device 505 and a server system 510. The server system 510 can be an example of the server system 125 described with reference to Figure 1 The user device 505 can be an example of the user device 110 described with reference to Figure 1 The server system 510 can receive data for storage from the user device 505. The server 510 can include a leader control plane node (node 1), a follower control plane node (node 2), and an access manager node. Although a single follower node is shown, it should be understood that the process flow 500 can support multiple follower nodes. Additionally or alternatively, although a single access manager node is shown, it should be understood that the process flow 500 can support multiple access manager nodes.

[0064] The server can represent a collection of computing components, data storage components, and the like as described herein. In some examples, the operations shown in process flow 500 can be performed by hardware (e.g., including circuitry, processing blocks, logic components, and other components), code (e.g., software or firmware) executed by a processor, or any combination thereof. Alternative examples can be implemented in which some steps are performed in a different order than described or not performed at all. In some cases, steps can include additional features not mentioned below, or other steps can be added.

[0065] At 515, the leader control plane node (node 1) can receive a request to store data in a distributed storage system that includes a collection of access manager nodes. As described herein, the leader control plane node can manage one or more access manager nodes. At 520, the leader control plane node can generate cache data that identifies an instruction from the leader control plane node to the one or more access manager nodes managed by the leader control plane node. In some examples, the instruction can instruct the one or more access manager nodes to store the data indicated in the request. In some examples, the leader control plane node can create a replication log entry in a cache of the leader control plane node that includes the instruction for the one or more access manager nodes of the distributed storage system to store the data indicated in the request.

[0066] At 525, the leader control plane node can send the instruction to the access manager nodes. The leader control plane node can instruct the one or more access manager nodes to store the data in the distributed storage system according to the instruction. At 530, the leader control plane node can receive a confirmation message from the one or more access manager nodes that indicates that the data indicated in the request has become data stored by the one or more access manager nodes. In some examples, the leader control plane node can update the replicated log entry in the cache to indicate that the data indicated in the request has become data stored by the one or more access manager nodes of the distributed storage system or otherwise indicate that the access manager has completed the requested instruction.

[0067] At 535, the leader control plane node can send replication instructions to one or more follower control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes. The follower control plane node (node 2) can receive a replication instruction from the leader control plane node to replicate the cache data of the leader control plane node into a cache of the follower control plane node. At 540, the follower control plane node can replicate the cache data.

[0068] At 545, the follower control plane node can receive an indication that the follower control plane node is selected to be a new leader control plane node based on a failure of a leader control plane node. In some cases, the follower control plane node can receive the selection indication from elsewhere in the cluster of neurons if, for example, the previous leader control plane node fails. At 550, the follower control plane node can process its replicated cache to retrieve instructions from the leader control plane node to one or more access manager nodes of the distributed storage system. In some examples, the instructions can indicate that the one or more access manager nodes managed by the leader control plane node store data indicated in a request. In some examples, the follower control plane node can identify replicated log entries associated with the previous leader node. The follower control plane node can identify from the replicated log entries associated with the previous leader node an incomplete log entry (e.g., due to a failure of the previous leader node). For example, the follower control plane node can identify that at least one instruction has not been executed by the access manager node. At 555, the follower control plane node can instruct the one or more access manager nodes of the distributed storage system to store data according to the instructions retrieved from the cache. For example, the follower control plane node can instruct the access manager node to complete a task that was not completed at the previous leader node.

[0069] Figure 6 A block diagram 600 of a device 605 that supports techniques to enable cache consistency across a distributed storage cluster in accordance with aspects of the present disclosure is shown. The device 605 can include an input module 610, an output module 615, and a cache replication component 620. The device 605 can also include a processor. Each of these components can be in communication with one another (e.g., via one or more buses).

[0070] The input module 610 can manage input signals for the device 605. For example, the input module 610 can identify input signals based on interactions with a modem, a keyboard, a mouse, a touchscreen, or similar devices. These input signals can be associated with user inputs or processing at other components or devices. In some cases, the input module 610 can utilize an operating system such as iOS®, ANDROID®, WINDOWS®, or other known operating systems to process the input signals. The input module 610 can send aspects of these input signals to other components of the device 605 for processing. For example, the input module 610 can send input signals to the cache replication component 620 to support techniques to enable cache consistency across a distributed storage cluster. In some cases, the input module 610 can be a component of the I / O controller 810 as described with reference to FIG. 8. Figure 8

[0071] ​The output module 615 can manage output signals of the device 605. For example, the output module 615 can receive signals from other components of the device 605 (e.g., the cache replication component 620) and can send those signals to other components or devices. In some examples, the output module 615 can send output signals for display in a user interface, for storage in a database or data store, for further processing at a server or cluster of servers, or for any other processing at any number of devices or systems. In some cases, the output module 615 can be a component of the I / O controller 810 as described with reference to Figure 8

[0072] For example, the cache replication component 620 can include a request component 625, an instruction component 630, a replication instruction component 635, a leader selection component 640, or any combination thereof. In some examples, the cache replication component 620 or various components thereof can be configured to perform various operations (e.g., receiving, monitoring, sending) using, in cooperation with, or otherwise in cooperation with the input module 610, the output module 615, or both. For example, the cache replication component 620 can receive information from the input module 610, send information to the output module 615, or in combination with the input module 610, the output module 615, or both, to receive information, send information, or perform various other operations described herein.

[0073] The cache replication component 620 can support a data store in accordance with examples as disclosed herein. The request component 625 can be configured to or otherwise support means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system including a plurality of access manager nodes. The instruction component 630 can be configured to or otherwise support means for generating, by the leader control plane node, cache data identifying instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store the data indicated in the request. The replication instruction component 635 can be configured to or otherwise support means for sending, by the leader control plane node to one or more follower control plane nodes of the plurality of control plane nodes, replication instructions to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0074] ​Additionally or alternatively, the cache replication component 620 can support data storage in accordance with examples as disclosed herein. The replication instruction component 635 can be configured as or otherwise support a means for receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, a replication instruction to replicate cache data of the leader control plane node into a cache of the follower control plane node. The leader selection component 640 can be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on a failure of the leader control plane node. The instruction component 630 can be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction indicating that one or more access manager nodes managed by the leader control plane node store data indicated in a request. The instruction component 630 can be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

[0075] Figure 7 A block diagram 700 showing a cache replication component 720 that supports techniques to enable cache consistency across a distributed storage cluster in accordance with aspects of the present disclosure is shown. The cache replication component 720 can be an example of aspects of a cache replication component or the cache replication component 620, or both, as described herein. The cache replication component 720, or various components thereof, can be examples of means for performing various aspects of techniques to enable cache consistency across a distributed storage cluster, as described herein. For example, the cache replication component 720 can include a request component 725, an instruction component 730, a replication instruction component 735, a leader selection component 740, a log entry component 745, a failure component 750, an acknowledgement component 755, or any of their arbitrary combinations. Each of these components can communicate, directly or indirectly, with one another (e.g., via one or more buses).

[0076] The cache replication component 720 can support data storage in accordance with examples as disclosed herein. The request component 725 can be configured as or otherwise support a means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The instruction component 730 can be configured as or otherwise support a means for generating, by the leader control plane node, cache data that identifies an instruction from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The replication instruction component 735 can be configured as or otherwise support a means for sending, by the leader control plane node to one or more follower control plane nodes of the plurality of control plane nodes, a replication instruction to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0077] In some examples, the instruction component 730 can be configured as or otherwise support a means for instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the instruction. In some examples, to support generating the cache data, the log entry component 745 can be configured as or otherwise support a means for creating, in a cache of the leader control plane node, a replication log entry comprising the instruction for the one or more access manager nodes of the distributed storage system to store the data indicated in the request.

[0078] In some examples, the acknowledgement component 755 can be configured as or otherwise support a means for receiving, from the one or more access manager nodes, an acknowledgement message indicating that the data indicated in the request has become data stored by the one or more access manager nodes. In some examples, the log entry component 745 can be configured as or otherwise support a means for updating the replicated log entry of the cache to indicate that the data indicated in the request has become data stored by the one or more access manager nodes of the distributed storage system.

[0079] In some examples, the failure component 750 can be configured as or otherwise support a means for determining a failure of the leader control plane node. In some examples, the leader selection component 740 can be configured as or otherwise support a means for sending, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on the failure of the leader control plane node.

[0080] In some examples, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples, the request to store data in the distributed storage system includes a write request.

[0081] Additionally or alternatively, the cache replication component 720 can support data storage in accordance with the examples disclosed herein. In some examples, the replication instruction component 735 can be configured as or otherwise support a means for receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, from a leader control plane node, replication instructions to replicate cache data of the leader control plane node into a cache of the follower control plane node. The leader selection component 740 can be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on a failure of the leader control plane node. In some examples, the instruction component 730 can be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request. In some examples, the instruction component 730 can be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instructions retrieved from the cache.

[0082] In some examples, the log entry component 745 can be configured as or otherwise support a means for processing the cache to identify a replication log entry, the replication log entry including instructions to store data indicated in a request via one or more access manager nodes of a distributed storage system. In some examples, the instruction component 730 can be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data indicated in the request based at least in part on the replication log entry.

[0083] In some examples, the acknowledgement component 755 can be configured as or otherwise support a means for receiving, from the one or more access manager nodes, an acknowledgement message indicating that the data indicated in the request has become data stored by the one or more access manager nodes.

[0084] In some examples, the request component 725 can be configured as or otherwise support a means for receiving, at a new leader control plane node, a second request to store data in the one or more access manager nodes. In some examples, the instruction component 730 can be configured as or otherwise support a means for generating, by the new leader control plane node, second cache data that identifies second instructions from the new leader control plane node to the one or more access manager nodes that instruct the one or more access manager nodes to store data indicated in the second request. In some examples, the instruction component 730 can be configured as or otherwise support a means for sending, by the new leader control plane node, the second replication instructions to one or more follower control plane nodes of the new leader control plane node to replicate the second cache data into respective caches of the one or more follower control plane nodes.

[0085] In some examples, the instruction component 730 can be configured as or otherwise support a means for instructing the one or more access manager nodes to store data in the distributed storage system in accordance with the second instructions. In some examples, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples, the second cache of the new leader control plane node can be configured to enable cluster failover for a defined amount of time.

[0086] Figure 8 A diagram illustrates a system 800 including a device 805 that supports techniques to enable cache consistency across a distributed storage cluster in accordance with aspects of the present disclosure. The device 805 can be an example of or include the components of device 605 as described herein. The device 805 can include components for bi-directional data communication, including components for transmitting and receiving communications, such as a cache replication component 820, an I / O controller 810, a database controller 815, memory 825, a processor 830, and a database 835. These components can be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more buses, such as a bus 840.

[0087] I / O controller 810 can manage input signals 845 and output signals 850 of device 805. I / O controller 810 can also manage peripheral devices that are not integrated into device 805. In some cases, I / O controller 810 can represent a physical connection or port to an external peripheral device. In some cases, I / O controller 810 can use a controller such as or other known operating systems. In other cases, I / O controller 810 may represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, I / O controller 810 may be implemented as part of processor 830. In some examples, a user may interact with device 805 via I / O controller 810 or via hardware components controlled by I / O controller 810.

[0088] Database controller 815 can manage the storage and processing of data in database 835. In some cases, a user can interact with database controller 815. In other cases, database controller 815 can operate automatically without user interaction. Database 835 can be an example of a single database, a distributed database, multiple distributed databases, a data store, a data lake, or an emergency backup database.

[0089] The memory 825 may include random access memory (RAM) and ROM. The memory 825 may store computer-readable, computer-executable software including instructions that, when executed, cause the processor 830 to perform the various functions described herein. In some cases, the memory 825 may also contain BIOS, which may control basic hardware or software operations (e.g., interaction with peripheral components or devices).

[0090] The processor 830 may include an intelligent hardware device (e.g., a general-purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor 830 may be configured to operate the memory array using a memory controller. In other cases, the memory controller may be integrated into the processor 830. The processor 830 may be configured to execute computer-readable instructions stored in the memory 825 to perform various functions (e.g., functions or tasks that support techniques for implementing cache coherence across a distributed storage cluster).

[0091] The cache replication component 820 can support data storage in accordance with examples as disclosed herein. For example, the cache replication component 820 can be configured as or otherwise support a means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The cache replication component 820 can be configured as or otherwise support a means for generating, by the leader control plane node, cache data that identifies an instruction from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The cache replication component 820 can be configured as or otherwise support a means for sending, by the leader control plane node to one or more follower control plane nodes of the plurality of control plane nodes, a replication instruction to replicate the cache data into respective caches of the one or more follower control plane nodes.

[0092] Additionally or alternatively, the cache replication component 820 can support data storage in accordance with examples as disclosed herein. For example, the cache replication component 820 can be configured as or otherwise support a means for receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction from a leader control plane node to replicate cache data of the leader control plane node into a cache of the follower control plane node. The cache replication component 820 can be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on a failure of the leader control plane node. The cache replication component 820 can be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. The cache replication component 820 can be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

[0093] By including or configuring the cache replication component 820 in accordance with examples as described herein, the device 805 can support techniques for improved data storage functionality, reduced latency, improved user experience related to efficient data processing, and more efficient utilization of storage resources.

[0094] Figure 9A flow chart illustrating a method 900 for supporting techniques for achieving cache consistency across a distributed storage cluster according to aspects of the present disclosure is shown. The operations of the method 900 may be implemented by the storage architecture described herein or components thereof. For example, the operations of the method 900 may be implemented by reference to Figures 1 to 8 In some examples, the storage architecture may execute an instruction set to control functional elements of the storage architecture to perform the described functions. Additionally or alternatively, the storage architecture may use dedicated hardware to perform various aspects of the described functions.

[0095] At 905, the method may include: receiving, at a leader control plane node among a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The operations of 905 may be performed according to examples disclosed herein. In some examples, aspects of the operations of 905 may be as described in reference to Figure 7 The described request component 725 is executed.

[0096] At 910, the method may include: generating, by the leader control plane node, cache data, the cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node among the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The operations of 910 may be performed according to the examples disclosed herein. In some examples, aspects of the operations of 910 may be performed as described in reference to Figure 7 The described instruction component 730 is executed.

[0097] At 915, the method may include: sending, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cached data to corresponding caches of the one or more follower control plane nodes. The operations of 915 may be performed according to the examples disclosed herein. In some examples, aspects of the operations of 915 may be described with reference to Figure 7 The described copy instruction component 735 is executed.

[0098] Figure 10 A flow chart illustrating a method 1000 for supporting techniques for achieving cache coherence across a distributed storage cluster according to aspects of the present disclosure is shown. The operations of the method 1000 may be implemented by the storage architecture described herein or components thereof. For example, the operations of the method 1000 may be implemented by reference to Figures 1 to 8 In some examples, the storage architecture may execute an instruction set to control functional elements of the storage architecture to perform the described functions. Additionally or alternatively, the storage architecture may use dedicated hardware to perform various aspects of the described functions.

[0099] At 1005, the method can include receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The operations of 1005 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1005 can be performed by a request component 725 as described with reference to Figure 7

[0100] At 1010, the method can include generating, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions indicating the one or more access manager nodes to store the data indicated in the request. The operations of 1010 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1010 can be performed by an instruction component 730 as described with reference to Figure 7

[0101] At 1015, the method can include sending, by the leader control plane node, replication instructions to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data into respective caches of the one or more follower control plane nodes. The operations of 1015 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1015 can be performed by a replication instruction component 735 as described with reference to Figure 7

[0102] At 1020, the method can include indicating the one or more access manager nodes to store the data in the distributed storage system in accordance with the instructions. The operations of 1020 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1020 can be performed by an instruction component 730 as described with reference to Figure 7

[0103] Figure 11 A flow diagram illustrating a method 1100 that supports implementing cache consistency across distributed storage clusters in accordance with aspects of the present disclosure is shown. The operations of method 1100 can be implemented by a storage architecture or its components as described herein. For example, the operations of method 1100 can be performed by a storage architecture as described with reference to Figures 1 to 8 In some examples, aspects of the operations of method 1100 can be performed by a storage architecture as described with reference to FIGS. 1 through 8. Additionally or alternatively, the storage architecture can execute sets of instructions to control the functional elements of the storage architecture to perform the described functions. The storage architecture can also utilize dedicated hardware to perform aspects of the described functions.

[0104] ​​​​At 1105, the method can include receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system that includes a plurality of access manager nodes. The operations of 1105 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1105 can be performed by a request component 725 as described with reference to Figure 7 FIG. 7.

[0105] At 1110, the method can include generating, by the leader control plane node, cache data that identifies instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions indicating the one or more access manager nodes to store the data indicated in the request. The operations of 1110 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1110 can be performed by an instruction component 730 as described with reference to Figure 7 FIG. 7.

[0106] At 1115, the method can include sending, by the leader control plane node to one or more follower control plane nodes of the plurality of control plane nodes, replication instructions to replicate the cache data into respective caches of the one or more follower control plane nodes. The operations of 1115 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1115 can be performed by a replication instruction component 735 as described with reference to Figure 7 FIG. 7.

[0107] At 1120, the method can include determining a failure of the leader control plane node. The operations of 1120 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1120 can be performed by a failure component 750 as described with reference to Figure 7 FIG. 7.

[0108] At 1125, the method can include sending, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node is selected to be a new leader control plane node based at least in part on the failure of the leader control plane node. The operations of 1125 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1125 can be performed by a leader selection component 740 as described with reference to Figure 7 FIG. 7.

[0109] Figure 12A flow diagram illustrating a method 1200 that supports enabling cache consistency across distributed storage clusters in accordance with aspects of the present disclosure is shown. The operations of method 1200 can be implemented by a storage architecture or its components as described herein. For example, the operations of method 1200 can be performed by a storage architecture as described with reference to Figures 1 to 8 The storage architecture described can execute a set of instructions to control the functional elements of the storage architecture to perform described functions. Additionally or alternatively, the storage architecture can perform aspects of the described functions using special-purpose hardware.

[0110] At 1205, the method can include receiving, at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction from a leader control plane node to replicate cache data of the leader control plane node into a cache of the follower control plane node. The operations of 1205 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1205 can be performed by a replication instruction component 735 as described with reference to Figure 7 The storage architecture described can execute a set of instructions to control the functional elements of the storage architecture to perform described functions. Additionally or alternatively, the storage architecture can perform aspects of the described functions using special-purpose hardware.

[0111] At 1210, the method can include receiving, at the follower control plane node, an indication that the follower control plane node was selected to be a new leader control plane node based at least in part on a failure of the leader control plane node. The operations of 1210 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1210 can be performed by a leader selection component 740 as described with reference to Figure 7 The storage architecture described can execute a set of instructions to control the functional elements of the storage architecture to perform described functions. Additionally or alternatively, the storage architecture can perform aspects of the described functions using special-purpose hardware.

[0112] At 1215, the method can include processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of the distributed storage system, the instruction indicating that the one or more access manager nodes managed by the leader control plane node store data indicated in a request. The operations of 1215 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1215 can be performed by an instruction component 730 as described with reference to Figure 7 The storage architecture described can execute a set of instructions to control the functional elements of the storage architecture to perform described functions. Additionally or alternatively, the storage architecture can perform aspects of the described functions using special-purpose hardware.

[0113] At 1220, the method can include indicating that the one or more access manager nodes of the distributed storage system store data in accordance with the instruction retrieved from the cache. The operations of 1220 can be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of 1220 can be performed by an instruction component 730 as described with reference to Figure 7 The storage architecture described can execute a set of instructions to control the functional elements of the storage architecture to perform described functions. Additionally or alternatively, the storage architecture can perform aspects of the described functions using special-purpose hardware.

[0114] It should be noted that the methods described above describe possible implementations, and that the operations and the steps can be rearranged or otherwise modified and that other implementations can be possible. Further, aspects from two or more of the methods can be combined.

[0115] The description set forth herein describes example configurations, but does not represent all of the possible implementations or configurations thereof. Aspects used herein are meant to be examples and not limitations. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration," and not "preferred" or "advantageous over other examples." The detailed description includes specific details for the purpose of providing an understanding of the described techniques. These techniques, however, can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described examples.

[0116] In the appended figures, similar components or features can have the same reference label. Further, various components of the same type can be distinguished by adding a dash and a second label that distinguishes among the components of the same type in the appended figures. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.

[0117] Information and signals described herein can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0118] The various illustrative blocks and modules described in connection with the disclosure herein can be implemented or performed with a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).

[0119] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope of the disclosure and appended claims. For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations thereof. Features implementing functions can also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, "or" as used in a list of items (for example, a list of items prefaced by a phrase such as "at least one of" or "one or more of") indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Also, as used herein, the phrase "based on" shall not be construed as a reference to a closed set of conditions. For example, an exemplary step that is described as "based on condition A" can be based on both a condition A and a condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase "based on" shall be construed in the same manner as the phrase "based at least in part on."

[0120] Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium can be any available medium that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, non-transitory computer-readable media can comprise RAM, ROM, electrically erasable programmable ROM (EEPROM), compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of computer-readable media.

[0121] The description herein is presented to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A computer-implemented method for data storage, comprising: receiving, at a leader control plane node among a plurality of control plane nodes of the node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes at a first data center; generating, by the leader control plane node, cache data, the cache data identifying instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; as well as The leader control plane node sends a replication instruction to one or more follower control plane nodes among the multiple control plane nodes to replicate the cached data into the corresponding caches of the one or more follower control plane nodes, and the one or more follower control plane nodes instruct one or more access manager nodes at the second data center of the distributed storage system to store the data in the distributed storage system according to the request, so that the data is replicated and stored in the first data center and the second data center.

2. The method according to claim 1, further comprising: The one or more access manager nodes are instructed to store data in the distributed storage system according to the instructions.

3. The method according to claim 1, wherein Generating the cache data further includes: A replication log entry is created in a cache of the leader control plane node, the replication log entry including instructions for the one or more access manager nodes of the distributed storage system to store the data indicated in the request.

4. The method according to claim 3, further comprising: receiving a confirmation message from the one or more access manager nodes, the confirmation message indicating that the data indicated in the request has become data stored by the one or more access manager nodes; as well as A cached replication log entry is updated to indicate that the data indicated in the request has become data stored by the one or more access manager nodes of the distributed storage system.

5. The method according to claim 1, further comprising: determining a failure of the leader control plane node; as well as An indication is sent to a follower control plane node of the one or more follower control plane nodes that the follower control plane node was selected as a new leader control plane node based at least in part on the failure of the leader control plane node.

6. The method according to claim 1, wherein The cached data indicates: one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof.

7. The method according to claim 1, wherein The request to store data in the distributed storage system comprises a write request.

8. A computer-implemented method for data storage, comprising: At a follower control plane node among multiple control plane nodes in a node cluster, receiving a replication instruction from a leader control plane node to replicate cache data of the leader control plane node into a cache of the follower control plane node, instructing one or more access manager nodes of a distributed storage system at a second data center to store the data in the distributed storage system according to a request received by the leader control plane node, so that the data is replicated and stored in a first data center and a second data center, wherein the cache data identifies an instruction from the leader control plane node to one or more access manager nodes of the distributed storage system at the first data center managed by the leader control plane node, the instruction instructing the one or more access manager nodes of the distributed storage system at the first data center to store the data indicated in the request; receiving, at the follower control plane node, an indication that the follower control plane node is selected as a new leader control plane node based at least in part on a failure of the leader control plane node; processing, by the new leader control plane node, the cache to retrieve instructions from the leader control plane node to one or more access manager nodes of a distributed storage system, the instructions instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in the request; and The one or more access manager nodes of the distributed storage system are instructed to store the data according to the instructions retrieved from the cache.

9. The method according to claim 8, further comprising: processing the cache to identify a replication log entry comprising instructions to store data indicated in the request via the one or more access manager nodes of the distributed storage system, and The one or more access manager nodes of the distributed storage system are instructed to store data indicated in the request based at least in part on the replication log entry.

10. The method according to claim 9, further comprising: A confirmation message is received from the one or more access manager nodes, the confirmation message indicating that the data indicated in the request has become data stored by the one or more access manager nodes.

11. The method according to claim 8, further comprising: receiving, at the new leader control plane node, a second request to store data in the one or more access manager nodes; generating, by the new leader control plane node, second cache data, the second cache data identifying a second instruction from the new leader control plane node to the one or more access manager nodes, the second instruction instructing the one or more access manager nodes to store data indicated in the second request; as well as The new leader control plane node sends a second replication instruction to one or more follower control plane nodes of the new leader control plane node, so as to replicate the second cache data to corresponding caches of the one or more follower control plane nodes.

12. The method according to claim 11, further comprising: Instruct the one or more access manager nodes to store the data in the distributed storage system according to the second instruction.

13. The method according to claim 8, wherein The cached data indicates: one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof.

14. The method according to claim 8, wherein The new leader control plane node's second cache is configured to achieve cluster failover within a defined amount of time.

15. A device for data storage, comprising: processor; a memory coupled to the processor; as well as instructions stored in the memory and executable by the processor to cause the apparatus to: receiving, at a leader control plane node among a plurality of control plane nodes of the node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes at a first data center; generating, by the leader control plane node, cache data, the cache data identifying instructions from the leader control plane node to one or more access manager nodes of the plurality of access manager nodes managed by the leader control plane node, the instructions instructing the one or more access manager nodes to store data indicated in the request; as well as The leader control plane node sends a replication instruction to one or more follower control plane nodes among the multiple control plane nodes to replicate the cached data into the corresponding caches of the one or more follower control plane nodes, and the one or more follower control plane nodes instruct one or more access manager nodes at the second data center of the distributed storage system to store the data in the distributed storage system according to the request, so that the data is replicated and stored in the first data center and the second data center.

16. The device according to claim 15, wherein The instructions are further executable by the processor to cause the apparatus to: The one or more access manager nodes are instructed to store the data in the distributed storage system according to the instruction.

17. The device according to claim 15, wherein The instructions for generating the cache data are further executable by the processor to cause the apparatus to: A replication log entry is created in a cache of the leader control plane node, the replication log entry including instructions for the one or more access manager nodes of the distributed storage system to store the data indicated in the request.

18. The device according to claim 17, wherein The instructions are further executable by the processor to cause the apparatus to: receiving a confirmation message from the one or more access manager nodes, the confirmation message indicating that the data indicated in the request has become data stored by the one or more access manager nodes; as well as A cached replication log entry is updated to indicate that the data indicated in the request has become data stored by the one or more access manager nodes of the distributed storage system.

19. The device according to claim 15, wherein The instructions are further executable by the processor to cause the apparatus to: determining a failure of the leader control plane node; and An indication is sent to a follower control plane node of the one or more follower control plane nodes that the follower control plane node was selected as a new leader control plane node based at least in part on the failure of the leader control plane node.

20. The apparatus according to claim 15, wherein The cached data indicates: one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof.

Images