Placement of data path service partitions

Abstract

The disclosure generally relates to placement of datapath service partitions. In some examples, a scheduler of a control plane schedules control plane entities in a cluster of compute nodes, where placement of the control plane entities in respective compute nodes reflects locations where respective data path service partitions are placed. A first control plane entity in a first compute node activates a first data path service partition in the first compute node. The first data path service partition is used for accessing data of a first data fragment in the data storage, wherein the data of the data storage is partitioned into a plurality of data fragments associated with each data path service partition. Based on the condition of the compute node cluster, the placement of the second control plane entity is modified to change the placement of a second datapath service partition associated with the second control plane entity in the compute node cluster.

Description

Background Technology

[0001] Data storage includes pools of storage devices used to store data. In some examples, storage devices may include flash memory devices arranged in groups of "just a bunch of flash" (JBOF). In other examples, storage devices may include disk-based storage devices or other types of storage devices. Attached Figure Description

[0002] Some embodiments of this disclosure are described with reference to the following drawings.

[0003] Figure 1 It is a block diagram based on some examples of the layout of a cluster of compute nodes, a control system, and a storage system.

[0004] Figure 2A and Figure 2B It is a block diagram based on some examples of layouts containing different numbers of computing nodes.

[0005] Figure 3 It is a block diagram of a storage medium with machine-readable instructions based on some examples of storage.

[0006] Figure 4 It is a block diagram of a system based on some examples.

[0007] Figure 5 It is a flowchart based on some examples.

[0008] Throughout the accompanying drawings, the same reference numerals designate similar but not necessarily identical elements. These drawings are not necessarily drawn to scale, and some dimensions may be enlarged for clarity of the examples shown. Furthermore, the drawings provide examples and / or embodiments consistent with the specification; however, the specification is not limited to the examples and / or embodiments provided in the drawings. Detailed Implementation

[0009] Distributed data partitioning can be applied to data in a data store to create data shards, where a "data shard" can refer to a partition or segment of data in the data store. Clusters of compute nodes can be deployed to allow distributed access to data shards in the data store. Each compute node can execute one or more data path service partitions, where each data path service partition supports input / output (I / O) access to the corresponding data shard. Multiple data path service partitions can access their respective data shards in parallel.

[0010] Determining how to allocate data path service partitions across compute nodes can be complex. The allocation of data path service partitions across a compute node cluster can include the initial scheduling of the data path service partitions across selected compute nodes. After the initial scheduling, the placement of data path service partitions across compute nodes can be modified as conditions change within the compute node cluster. These changes may include failures or errors in compute nodes, loss of connectivity between compute nodes and data stores, unbalanced workload distribution (where one compute node may be performing more workload than another), the addition or removal of compute nodes, or any other factors that may affect the workload load in the compute node cluster.

[0011] For distributed access to data shards in data storage, other issues associated with using compute node clusters may include any one or a combination of the following: the possibility of data corruption associated with multiple instances of the same data shard accessing the data path service partition; placing the data path service partition on an unhealthy compute node; determining when to disable a compute node; excessively long failover time when moving workloads from an unhealthy compute node to another compute node; or other issues.

[0012] According to some embodiments of this disclosure, instead of placing data path service partitions directly on compute nodes, the scheduler of the control plane of the compute node cluster associated with data storage can schedule agents for data path service partitions on each compute node. The agent is a control entity responsible for activating and deactivating the corresponding data path service partition in the compute node. The agent (referred to as a "data partition agent") is part of the control plane, while the data path service partition activated or deactivated by the data partition agent is part of the data plane.

[0013] In some examples, each data partition agent executes within a container, which is a virtual compute entity. In specific examples, the data partition agent executes within a container of a Kubernetes pod. In other examples, the data partition agent may execute within a virtual machine (VM). In further examples, the data partition agent may include a program. By placing data partition agents on individual compute nodes, the scheduler does not need to be configured to support data path service partitions. Instead, the scheduler schedules data partition agents on selected compute nodes, where the placement of the data partition agent reflects the desired location of the corresponding data path service partition. In other words, the scheduler indirectly places data path service partitions based on placing data partition agents on compute nodes.

[0014] A “compute node” can refer to one or more physical computers. A “data path service partition” refers to any set of modules and data structures used to access data shards in response to a request from a requester (e.g., a read request or a write request). Examples of modules that can be included in a data path service partition include any or a combination of the following: a caching module that manages data caching (e.g., caching write data in a write buffer), an indexing module that manages data indexing, or a garbage collector that manages the removal of data from the storage system after a delete operation for deleting data. Examples of data structures that can be included in a data path service partition include any or a combination of the following: a write buffer for caching write data to be written to the storage system, an index that maps objects to storage locations in the storage system, or information related to data redundancy in the storage system (e.g., redundant array of independent disks (RAID) redundancy). Although example modules and data structures have been listed above, in further examples, a data path service partition may include additional or alternative modules or data structures.

[0015] In some examples, data path service partitions form vertical slices of the storage system's I / O stack. Multiple data path service partitions thus form multiple vertical slices of the I / O stack, which can independently access their respective data slices in parallel. The I / O stack includes the modules and data structures involved in the data access operations for the corresponding data slice. A vertical slice of the I / O stack refers to a corresponding instance of the module and data structure for that data slice.

[0016] Figure 1 This is a block diagram of an example layout of a cluster containing compute nodes 102-1, 102-2, and 102-3. Although Figure 1 The example shows three compute nodes, but in different examples, the cluster may include a different number of compute nodes.

[0017] Compute nodes 102-1 to 102-3 communicate with storage system 106 via interconnect 104. In some examples, storage system 106 includes an arrangement of storage devices (such as flash memory devices, disk-based storage devices, or other types of storage devices). "Interconnect" can refer to the communication medium for data communication. Examples of interconnect 104 may include any or a combination of the following: local area network (LAN), storage area network (SAN), wide area network (WAN), or any other type of communication link.

[0018] In some examples, data communication on the interconnect 104 follows protocols such as the Non-volatile Memory Express (NVMe) over-Fabric (NVMeoF) protocol, the Internet Small Computer System Interface (iSCSI) protocol, the Infiniband protocol, or any other protocol, whether standardized, open source, or proprietary.

[0019] The control system 108 is associated with a cluster of computing nodes 102-1 to 102-3. The control system 108 can be implemented using one or more computers. In some cases, elements of the control system 108 may be implemented in one or more of the computing nodes 102-1 to 102-3. In other examples, the control system 108 may be separate from the computing nodes 102-1 to 102-3.

[0020] The control system 108 includes a scheduler 110 for scheduling data partition agents in compute nodes 102-1 through 102-3, which controls the placement of data path service partitions. Although referred to as a single entity, the scheduler 110 may include multiple modules. For example, if the control system 108 is a Kubernetes control system, the scheduler 110 may include the following modules: a Kubernetes scheduler, a Kubernetes de-scheduler, and a mapper. Typically, the Kubernetes scheduler is responsible for placing pods in compute nodes, where a pod may include one or more containers. The Kubernetes scheduler places pods in compute nodes based on a pod specification, which includes constraints that guide the Kubernetes scheduler on how to place pods. The Kubernetes de-scheduler complements the actions of the Kubernetes scheduler by rebalancing pods across compute nodes in response to specific events, such as compute node failures. Furthermore, the de-scheduler can check (e.g., periodically) the current pod placement and can verify that the pod placement meets the constraints specified by the pod specification. If a container placement does not meet constraints, the de-scheduler can create an eviction plan to remove the container from the compute node, and the de-scheduler can send information about the evictioned container to the Kubernetes scheduler to schedule the eviction. The mapper configures the container during cluster setup or cluster changes (e.g., adding or removing compute nodes).

[0021] In other examples, scheduler 110 may be implemented using different techniques. Typically, scheduler 110 may be implemented using machine-readable instructions executable on the processing resources of control system 108.

[0022] The control system 108 also includes an interface server 118, through which components in compute nodes 102-1 to 102-3 can communicate. In some examples, the interface server 118 includes an application programming interface (API) server, which provides APIs accessible to components in compute nodes 102-1 to 102-3. Routines using the APIs enable communication between the components in compute nodes 102-1 to 102-3 and the control system 108. In other examples, the interface server 118 may provide different types of interfaces accessible to components in compute nodes 102-1 to 102-3.

[0023] In some examples, data may be stored as blocks on the storage devices of storage system 106. A “block” can refer to a data block or any other portion of data. These data blocks are stored in their respective storage locations within storage system 106. Although data is stored as data blocks in storage system 106, client 114 can access objects, where objects comprise an arrangement of data blocks stored in storage system 106. For example, an object may be part of object storage (such as object storage provided by Amazon Simple Storage Service (S3)). S3 object storage will store the object's data. Figure 1 In the example, the global object namespace 112 is presented to client 114. The global object namespace includes the names of objects that client 114 can access. "Client" can refer to an electronic device or program.

[0024] In the example using S3 object storage, objects can be contained in bucket 116. Bucket 116 can contain one or more objects. Objects in each bucket 116 can be partitioned into data shards, which can be accessed by data path service partitions in compute nodes 102-1, 102-2, and 102-3.

[0025] In other examples, other types of object storage can be used to store data accessible to client 114. For example, another type of object storage can store files, in which case the global object namespace 112 can be replaced by a file system. Regardless of the type of object storage used, the object data in the object storage is partitioned across compute nodes 102-1 to 102-3 into data shards accessible to the corresponding data path service partitions.

[0026] Each data path service partition is represented as "DSP (data path service partition)". Figure 1In the example, compute node 102-1 has two data path service partitions (DSP A and DSP B), compute node 102-2 has one data path service partition (DSP C), and compute node 102-3 has two data path service partitions (DSP D and DSP E). Although a specific number of DSPs is shown in each compute node, in other examples, compute nodes may run different numbers of DSPs.

[0027] Scheduler 110 places data partition agents in each compute node (in Figure 1 (represented as a DSP agent in the middle). Figure 1 In the example, DSP agent A and DSP agent B are placed in compute node 102-1 by scheduler 110, DSP agent C is placed in compute node 102-2 by scheduler 110, and DSP agent D and DSP agent E are placed in compute node 102-3 by scheduler 110.

[0028] The DSP agent is part of the control plane, while the Data Path Service Partition (DSP) is part of the data plane. The data plane defines the path taken by data between the storage system 106 and clients requesting access (read or write) to the data. The control plane includes the control functions that manage the data plane.

[0029] Scheduler 110 places DSP agents in computing nodes 102-1, 102-2, and 102-3 based on constraint information 120 stored in memory 124 of control system 108. Constraint information 120 includes one or more rules that specify the locations where DSP agents should be placed to achieve a goal, such as evenly distributing DSPs across computing nodes 102-1 to 102-3, avoiding placing DSPs on specific computing nodes, or any other rules.

[0030] In some examples, the DSP is associated with a service daemon (SD) procedure. An SD procedure can be a user-space procedure that executes in the user space of the compute node to invoke a service, such as the DSP. Figure 1 In the example, SD process 122-1 is executed on compute node 102-1, SD process 122-2 is executed on compute node 102-2, and SD process 122-3 is executed on compute node 102-3.

[0031] In some examples, the SD process is containerized (containing one or more containers) and the data plane is implemented in the compute node. The DSP runs within a container of the SD process. In other examples, the DSP can be implemented as a microservice managed by the SD process. In yet another example, the DSP can run in a VM.

[0032] exist Figure 1 In the example, DSP A and DSP B are associated with SD process 122-1 (e.g., DSP A and DSP B run in the corresponding containers of SD process 122-1), DSP C is associated with SD process 122-2, and DSP D and DSP E are associated with SD process 122-3.

[0033] Once a DSP agent is placed in a compute node, it communicates with the corresponding SD process to activate the corresponding DSP in the compute node. For example, DSP agent A instructs SD process 122-1 to activate DSP A, and DSP agent B instructs SD process 122-1 to activate DSP B. Similarly, DSP agent C instructs SD process 122-2 to activate DSP C, DSP agent D instructs SD process 122-3 to activate DSP D, and DSP agent E instructs SD process 122-3 to activate DSP E. The activated DSP begins running in the compute node and uses the compute node's resources (including processing, memory, and communication resources).

[0034] The DSP agent can also communicate with the SD process to deactivate the DSP. Activating the DSP can mean starting the DSP or sending a run signal to the DSP to initiate operation. Deactivating the DSP can mean shutting down the DSP or sending a signal to the DSP to stop operation. Communication between the DSP agent and the SD process can be accomplished through a message interface (such as a remote procedure call (RPC) interface). An example of an RPC interface is the gRPC interface. Another message interface that can be used is the representational state transfer (REST) ​​interface.

[0035] In some examples, the SD process is stateless; in other words, the SD process does not maintain persistent information about the active DSPs associated with it. If the SD process restarts (shuts down and restarts), the DSP agent on the same compute node detects the restart and can instruct the SD process to reactivate the corresponding DSP. In some examples, the DSP agent maintains persistent information about the active DSPs, allowing the SD process to reactivate the DSPs upon restart.

[0036] In other examples, the SD process is omitted. In such additional examples, the DSP agent can directly activate or deactivate the DSP without interacting with the SD process.

[0037] Scheduler 110 loads DSP agents in each of compute nodes 102-1 to 102-3 during cluster setup or when compute nodes are added. After cluster setup, as the status of compute node cluster 102-1 to 102-3 changes, scheduler 110 can also add more DSP agents to or remove DSP agents from compute nodes. Scheduler 110 can also migrate DSP agents from one compute node to another, for example, to balance the load on compute nodes 102-1 to 102-3, or more generally, when scheduler 110 detects that the current placement of DSPs in compute nodes 102-1 to 102-3 does not satisfy one or more rules in constraint information 120.

[0038] The DSP agent can handle error conditions when attempting to activate or deactivate a DSP in a compute node. For example, if the SD process in the compute node returns an error (e.g., a resource exhaustion error indicating that the SD process has detected that the compute node's resources have been exhausted), the DSP agent can terminate itself within the compute node. Self-termination of the DSP agent includes the DSP agent shutting itself down within the compute node and notifying scheduler 110 so that the DSP agent can be rescheduled by scheduler 110 on another compute node. As part of self-termination, the DSP agent may also deactivate the corresponding DSP.

[0039] In some examples, each compute node also includes a node problem detector, which can be implemented by machine-readable instructions. For example, compute node 102-1 includes node problem detector 130-1, compute node 102-2 includes node problem detector 130-2, and compute node 102-3 includes node problem detector 130-3. Each node problem detector is capable of monitoring the status of the compute node to determine if the compute node is unhealthy. For example, a node problem detector can detect if a compute node has lost connection to storage system 106, such as due to hardware failure or software error. In further examples, a node problem detector can detect if certain programs have stopped running or are not functioning correctly. For example, if the compute node cluster forms a Kubernetes cluster, the node problem detector can be able to detect whether a kubelet is running on the compute node. The kubelet is a node agent that performs checks to ensure that containers are functioning correctly.

[0040] In other examples, node problem detectors can monitor various metrics, such as those related to resource usage (e.g., processing resources, memory resources, communication resources, etc.), resource performance, error metrics, or other types of metrics. Based on these metrics, node problem detectors can detect or predict unhealthy conditions of compute nodes. Typically, an "unhealthy" condition refers to the state of a compute node where it is or will fall below a target level, or is experiencing or will experience errors or failures.

[0041] If the node problem detector detects an unhealthy compute node, it can send a notification to the control system 108 (e.g., to the interface server 118). The notification can be a message, signal, information element, or any other indicator indicating the unhealthy state of the compute node. In response to this notification, the scheduler 110 can migrate the DSP agent from the unhealthy compute node to one or more target compute nodes.

[0042] In some examples, the compute node may also include a lease manager (e.g., implemented with machine-readable instructions) that manages the node leases of the compute node. Figure 1 In this example, compute node 102-1 includes lease manager 140-1, compute node 102-2 includes lease manager 140-2, and compute node 102-3 includes lease manager 140-3. A "node lease" is a health indicator used to indicate whether a compute node is functioning correctly. As long as the compute node is functional and communicating with other compute nodes in the cluster (i.e., the compute node is not isolated from the cluster), the lease manager maintains the node lease and continues to refresh the watchdog timer. Watchdog timer 142-1 resides in compute node 102-1, watchdog timer 142-2 resides in compute node 102-2, and watchdog timer 142-3 resides in compute node 102-3. In some examples, the watchdog timer is a hardware timer that runs for a specified count before expiring.

[0043] Refreshing the watchdog timer can include signaling the watchdog timer to restart its count. If the lease manager detects that a given compute node has lost communication with the cluster, the lease manager stops restarting the watchdog timer for that given compute node. The expiration of the watchdog timer will trigger a reset for that given compute node.

[0044] The lease manager in a given compute node also maintains a representation of node leases, for example, by storing information elements in the memory of that given compute node. Information elements set to a specified value indicate that a node lease on that given compute node is active.

[0045] The control system 108 and other compute nodes in the cluster can check the node lease of a given compute node. If the control system 108 or other compute nodes do not detect the node lease of a given compute node (e.g., because the given compute node has lost its connection to the cluster, is shut down, or is restarting, and therefore cannot access the node lease), the control system 108 or other compute nodes can determine that the given compute node is no longer available. The scheduler 110 in the control system 108 will not place any new DSP agents on the given compute node, and further, the scheduler 110 can reschedule any DSP agents on the given compute node to one or more other compute nodes in the cluster.

[0046] although Figure 1 The example shows both a node problem detector and a lease manager in each compute node, but in other examples, the lease manager may be omitted. In such other examples, the node problem detector will also check the connectivity of a given compute node to the cluster. As long as the node problem detector confirms that the given compute node is connected to the cluster, it can send a heartbeat message or another indicator to the control system 108 and other compute nodes. If a given compute node becomes unavailable (e.g., due to loss of connectivity to the cluster, shutdown of the given compute node, or restart of the given compute node), the node problem detector will stop sending heartbeat messages or another indicator to the control system 108 and other compute nodes. Failure to receive this heartbeat message or other indicator indicates that the given compute node is unavailable.

[0047] Figure 2A An example arrangement is shown, including a control system 108 and two computing nodes 202-1 and 202-2 as part of a computing node cluster. Computing node 202-1 includes DSP agent A, DSP agent B, and DSP agent C, which are placed in the control plane 204-1 of computing node 202-1 by scheduler 110. DSP agent A has activated DSP A, DSP agent B has activated DSP B, and DSP agent C has activated DSP C. DSP A, DSP B, and DSP C are part of the data plane 206-1 of computing node 202-1.

[0048] Similarly, compute node 202-2 includes DSP agents D, E, and F, which are placed in the control plane 204-2 of compute node 202-2 by scheduler 110. DSP agent D is activated for DSP D, DSP agent E is activated for DSP E, and DSP agent F is activated for DSP F. DSP D, DSP E, and DSP F are part of the data plane 206-2 of compute node 202-2.

[0049] Figure 2BAn example is shown where another compute node 202-3 has been added to the cluster. Due to the addition of compute node 202-3, scheduler 110 detected a deviation in the placement of DSP agents, as each of compute nodes 202-1 and 202-2 was running three DSP agents, while the added compute node 202-3 was not running any DSP agents. This deviation may violate constraint information 120. Figure 1 The rules specify that the DSP will be evenly distributed across compute nodes in the cluster. Therefore, scheduler 110 evicts DSP agent C from compute node 202-1 and DSP agent F from compute node 202-2. Scheduler 110 places DSP agent C and DSP agent F in the control plane 204-3 of the added compute node 202-3.

[0050] DSP agent C, placed in compute node 202-3, can retrieve information about the last location of DSP C. In some examples, the SD process in the compute nodes of the cluster can store information in shared memory identifying the compute nodes on which each DSP is running. This information can be retrieved by a DSP agent placed in a new compute node by interacting with the SD process in that new compute node. Once DSP agent C identifies that DSP C was last located in compute node 202-1, DSP agent C in compute node 202-3 can send a stop notification to the SD process in compute node 202-1 to terminate DSP C in compute node 202-1. In response to receiving confirmation that DSP C has been terminated in compute node 202-1 (in the form of a notification from the SD process in compute node 202-1), DSP agent C in compute node 202-3 starts DSP C in compute node 202-3 by sending a start notification to the SD process in compute node 202-3. DSP agent F, placed in compute node 202-3, performs a similar process. Typically, a DSP agent placed in a new compute node will not activate the DSP in the new compute node until the DSP agent receives confirmation that the DSP in the previous compute node has been terminated.

[0051] In some examples using node leases, the DSP agent placed on the new compute node also checks if the node lease for the previous compute node is active. If the node lease is not active (indicating that the previous compute node is unavailable), the DSP agent can skip sending a stop notification to the SD process for the previous compute node.

[0052] If from Figure 2BIf a compute node (e.g., 202-1) is removed from the cluster in the configuration, scheduler 110 will evict DSP agents A and B from compute node 202-1, and scheduler 110 will place DSP agents A and B in the remaining compute nodes 202-2 and 202-3, respectively. DSP agent A placed in compute node 202-2 activates DSP A in compute node 202-1, and DSP agent B placed in compute node 202-3 activates DSP A in compute node 202-3.

[0053] The goal of scheduler 110 (referred to as Goal A) is to avoid activating multiple DSPs on multiple compute nodes for the same data shard (a situation known as a split-brain), thus preventing data corruption caused by inconsistent modifications to the data shard by multiple DSPs. In the scenarios described above, associated with adding or removing compute nodes from the cluster, multiple instances of a DSP agent may exist for a specific DSP (identified by a specific DSP identifier) ​​during the process of evicting the DSP agent from the source compute node and placing it on the target compute node. To achieve Goal A, locks can be used to prevent multiple instances of a DSP agent for a specific DSP identifier from being active simultaneously. In the example using a Kubernetes cluster, locks can include variables included in the configuration data of a configuration map, which is an API object that stores data as key-value pairs. In other examples, locks can be implemented using any locking mechanism, such as a mutex or a semaphore. Locks are used to arbitrate among multiple instances of a DSP agent for a specific DSP identifier, allowing only one instance of the DSP agent to be active. An instance of the DSP agent placed on the target compute node is allowed to acquire the lock (e.g., by creating a configuration map) in response to the lock being released by an instance of the DSP agent on the source compute node (e.g., by deleting a configuration map).

[0054] Another objective (referred to as Objective B) is the DSP's liveness objective to avoid any DSP becoming unavailable, as an unavailable DSP means that data in the data shards associated with that unavailable DSP will be inaccessible to clients. A DSP may become unavailable if a compute node fails or is isolated (due to loss of connection to the compute node cluster). In some examples, node problem detectors (e.g., Figure 1 130-1 to 130-3 in the relevant documents) and / or the lease manager (e.g., Figure 1 Versions 140-1 to 140-3 allow the identification of unhealthy or isolated compute nodes, enabling the DSP agent on unhealthy or isolated compute nodes to be evicted and placed on the remaining compute nodes in the cluster.

[0055] In some examples, multiple compute nodes may be isolated, resulting in a cluster divided into a first subset of compute nodes that can communicate with each other and a second subset of compute nodes that can communicate with each other. However, compute nodes in the first subset cannot communicate with compute nodes in the second subset. In this scenario, the subset with more compute nodes is considered the majority subset, and the subset with fewer compute nodes is considered the minority subset. Scheduler 110 evicts DSP agents from the minority subset of compute nodes and places the evictped DSP agents in the majority subset of compute nodes.

[0056] A further objective (referred to as Objective C) is to reduce DSP failover time when evicting the DSP agent from an unavailable compute node (failed or disconnected) and placing it on the target compute node. If the DSP failover time is too long, clients may time out while waiting for requested data, which will degrade the user experience or cause errors. Objective C can be satisfied by adjusting scheduler 110 to launch a new instance of the DSP agent on the target compute node with minimal latency in response to the detection of an unavailable compute node. The combination of using a lock (for Objective A) and the fact that the previous instance of the DSP agent was located on an unavailable compute node means that quickly launching a new instance of the DSP agent on the target compute node is unlikely to result in an error condition where multiple DSP agent instances are active simultaneously.

[0057] The cluster of compute nodes executing individual DSPs provides a decomposed architecture for accessing data storage through data shards. As cluster resources change (compute nodes are added or removed), DSPs can scale accordingly. For example, DSPs can be migrated to different compute nodes. If new data shards are added, additional DSPs can be created. High availability is achieved by the ability to detect unhealthy or isolated compute nodes, allowing DSPs to be evicted from unhealthy or isolated nodes and placed on other compute nodes in the cluster. DSP agent-based placement, with DSPs indirectly placed by the scheduler in the control system, allows the scheduler to be modified to directly support DSPs without requiring modification, reducing scheduler development time.

[0058] Figure 3 This is a block diagram of a non-transitory machine-readable or computer-readable storage medium 300 that stores machine-readable instructions for the control plane of a cluster of computing nodes. For example, the machine-readable instructions for the control plane may be part of a control system 108 and located in... Figure 1 The computation nodes are 102-1 to 102-3.

[0059] Machine-readable instructions include control plane entity scheduling instructions 302, used to schedule control plane entities in the compute node cluster using the control plane scheduler. Control plane entities include, for example... Figure 1 and Figure 2A-2B The DSP agent is shown. The placement of control plane entities within each compute node reflects the placement of each data path service partition. Examples of data path service partitions include... Figure 1 and Figure 2A-2B DSP.

[0060] The machine-readable instructions include a first control plane entity activation instruction 304, for activating a first data path service partition in a first compute node by a first control plane entity in a first compute node in a compute node cluster.

[0061] The machine-readable instructions include data shard access instructions 306, used to access data in a first data shard in a data store using a first data path service partition in a first compute node, wherein the data in the data store is partitioned into multiple data shards associated with each data path service partition. The first data shard is accessed using the modules and data structures of the first data path service partition.

[0062] The machine-readable instructions include a control plane entity placement modification instruction 308, used to modify the placement of a second control plane entity within the control plane entity based on the status of the compute node cluster, thereby changing the layout of the second data path service partition associated with the second control plane entity in the compute node cluster. Note that the second control plane entity may be the same as or different from the first control plane entity, and the second data path service partition may be the same as or different from the first data path service partition.

[0063] In some examples, the data path service partition comprises distinct slices of the I / O stack in the data path between the compute node cluster and the data store.

[0064] In some examples, the first data path service partition includes elements selected from the following: a cache module for managing data caching, an index manager, a garbage collector, a buffer for storing data, an index that maps objects to storage locations in the data store, or redundancy information related to data redundancy in the data store.

[0065] In some examples, modifying the placement of the second control plane entity is part of a failover process (failover from unhealthy or isolated compute nodes) or a load balancing process (in response to detecting a deviation in the placement of the control plane entity).

[0066] In some examples, the control plane entity includes Kubernetes boxes that are mapped to various data path service partitions.

[0067] In some examples, the first control plane entity activates a first data path service partition in the first compute node based on the interaction between the first control plane entity and the data plane procedure (e.g., SD procedure) of the first compute node, wherein the first data path service partition includes elements that are part of the data plane.

[0068] In some examples, in order to modify the placement of the second control plane entity, machine-readable instructions remove an existing instance of the second control plane entity from a given compute node in the compute node cluster and launch a new instance of the second control plane entity in another compute node in the compute node cluster.

[0069] In some examples, an existing instance of the second control plane entity deactivates an existing instance of the second data path service partition in a given compute node, and a new instance of the second control plane entity activates a new instance of the second data path service partition in another compute node.

[0070] In some examples, during the placement of a modified second control plane entity, an existing instance of the second control plane entity and a new instance of the second control plane entity coexist, and machine-readable instructions can be arbitrated between the existing instance of the second control plane entity and the new instance of the second control plane entity using locks to allow only one of the existing instance of the second control plane entity and the new instance of the second control plane entity to proceed.

[0071] In some examples, the first control plane entity may terminate itself in the first compute node in response to an error condition. As part of the termination, the first control plane entity deactivates the first data path service partition in the first compute node.

[0072] In some examples, the scheduling of control plane entities in individual compute nodes is performed during the setup or modification of the compute node cluster (due to the addition of new compute nodes or the removal of existing compute nodes).

[0073] In some examples, agents on each compute node can monitor the health of each compute node. An "agent" can be, for example... Figure 1The node problem detector and / or lease manager are shown. Based on the first agent detecting that the first compute node is unhealthy, a health indicator indicating the first compute node's unhealthiness is provided. For example, the health indicator may include a notification from the node problem detector. As another example, the health indicator may include a node lease (a missing node lease indicates that the corresponding compute node is isolated from the cluster). In response to the health indicator indicating the first compute node is unhealthy, a process of disabling the first compute node is initiated, and any control plane entities are evictped from the first compute node. For example, the first compute node can be disabled by no longer refreshing the watchdog timer in the first compute node. In another example, the first compute node can be disabled by no longer placing a control plane entity and the corresponding data path service partition on the first compute node. In response to the health indicator indicating the first compute node is unhealthy, machine-readable instructions change the placement of the first control plane entity to another compute node in the compute node cluster.

[0074] In some examples, based on monitoring performed by an agent, machine-readable instructions can detect that a first subset of compute nodes in a compute node cluster is isolated from a second subset of compute nodes in the same cluster. The machine-readable instructions can then compare a first number of compute nodes in the first subset with a second number of compute nodes in the second subset. If the first number is greater than the second, the machine-readable instructions can disable the second subset of compute nodes.

[0075] In some examples, machine-readable instructions can select which compute nodes in a compute node cluster to schedule control plane entities based on health indicators from the agent.

[0076] Figure 4 This is a block diagram of system 400 based on some examples. System 400 includes a cluster 402 of computing nodes 403-1 and 403-2 (including hardware processors 404-1 and 404-2). The hardware processors may include microprocessors, the cores of multi-core microprocessors, microcontrollers, programmable integrated circuits, programmable gate arrays, or other hardware processing circuitry.

[0077] System 400 includes a scheduler 406, which can be executed within the control system to perform various tasks. The tasks of scheduler 406 include control plane entity scheduling task 408, used to schedule control plane entities 410-1 and 410-2 across the various compute nodes of cluster 402. The control plane entities are proxies for the corresponding data path service partitions 412-1 and 412-2 in the respective compute nodes.

[0078] The scheduler 406 has a task including a control plane entity placement modification task 414, which modifies the placement of the first control plane entity 410-1 based on the status of cluster 402, to change the placement of the first data path service partition 412-1 associated with the first control plane entity 410-1 in the compute node cluster 402.

[0079] The first control plane entity 410-1 activates the data path service partition 412-1 in the first compute node 403-1. The first data path service partition 412-1 in the first compute node 403-1 accesses data in the first data shard in the data store, wherein the data in the data store is partitioned into multiple data shards associated with each data path service shard.

[0080] Figure 5 This is a flowchart of process 500, which can be controlled by a control system (such as...). Figure 1 108 in the middle) and computing nodes (such as Figure 1 Execute steps 102-1 to 102-3 in the middle.

[0081] Process 500 includes using a control plane scheduler to schedule (at 502) control plane entities in the compute node cluster, wherein the control plane entities are part of the control plane and the placement of the control plane entities on each compute node reflects the placement of each data path service partition, which is part of the data plane and is used to access the corresponding data shard in the data store.

[0082] Process 500 includes activating (at 504) the first data path service partition in the first compute node by a first control plane entity in the control plane entity of the first compute node in the compute node cluster. Activation may include, for example, sending a startup notification to the SD procedure in the first compute node.

[0083] Process 500 includes accessing (at 506) data of a first data shard in a data store using a first data path service partition in a first compute node, wherein the data in the data store is partitioned into multiple data shards associated with each data path service partition.

[0084] Process 500 includes modifying (at 508) the placement of the first control plane entity based on the changed state of the compute node cluster, to evict the first control plane entity from the first compute node, and to place the first control plane entity in the second compute node in the compute node cluster.

[0085] In some examples, each compute node in the compute node cluster includes an SD procedure with which the control plane entity interacts to activate or deactivate a corresponding data path service partition. Procedure 500 also includes, after placing the first control plane entity in the second compute node, the first control plane entity in the second compute node determining that an existing instance of the first data path service partition is present in the first compute node. The first control plane entity in the second compute node sends a stop notification to the SD procedure in the first compute node to terminate the existing instance of the first data path service partition. Based on the confirmation received that the existing instance of the first data path service partition has been terminated, the first control plane entity in the second compute node activates a new instance of the first data path service partition in the second compute node by sending a start notification to the SD procedure in the second compute node.

[0086] The term "memory" as used herein can be implemented using one or more storage devices. A memory device may include any or a combination of the following: a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, an erasable programmable read-only memory (EPROM) device, an electrically erasable programmable read-only memory (EEPROM) device, a flash memory device, or any other type of memory device.

[0087] "Processing resources" can include one or more hardware processors.

[0088] Machine-readable instructions that can be executed on a hardware processor can refer to instructions that can be executed on a single hardware processor or instructions that can be executed on multiple hardware processors.

[0089] Storage media (e.g., Figure 3 The 300 in the specification can include any or a combination of the following: semiconductor storage devices, such as DRAM or SRAM, EPROM, EEPROM, or flash memory; disks, such as fixed disks, floppy disks, and removable disks; another magnetic medium, including magnetic tape; optical media, such as optical discs (CDs) or digital video discs (DVDs); or another type of storage device. Note that the instructions discussed above can be provided on a computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system that may have multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single or multiple manufactured components. One or more storage media can be located in a machine that executes the machine-readable instructions or at a remote site from which the machine-readable instructions can be downloaded via a network for execution.

[0090] In this disclosure, unless the context clearly indicates otherwise, the use of the terms “a,” “an,” or “the” is also intended to include the plural form. Furthermore, the terms “including,” “comprising,” “containing,” or “having” as used in this disclosure specify the presence of the referenced element but do not exclude the presence or addition of other elements.

[0091] In the foregoing description, numerous details have been set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations to the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A non-transitory machine-readable storage medium comprising instructions for the control plane of a cluster of computing nodes, the instructions causing the system, when executed, to: The control plane scheduler is used to schedule the control plane entities in each compute node of the compute node cluster, wherein the placement of the control plane entities in each compute node reflects the location of each data path service partition. The first data path service partition in the first computing node is activated by the first control plane entity in the control plane entity of the first computing node in the computing node cluster. Access data in a first data shard in a data store using the first data path service partition in the first computing node, wherein the data in the data store is partitioned into multiple data shards associated with each of the data path service partitions; as well as Based on the status of the compute node cluster, modify the placement of the second control plane entity in the control plane entity to change the placement of the second data path service partition associated with the second control plane entity in the compute node cluster.

2. The non-transitory machine-readable storage medium of claim 1, wherein the data path service partition comprises distinct slices of the input / output (I / O) stack in the data path between the compute node cluster and the data storage.

3. The non-transitory machine-readable storage medium of claim 2, wherein the first data path service partition comprises an element selected from the following: a cache module for managing data caching, an index manager, a garbage collector, a buffer for storing data, an index for mapping objects to storage locations in the data storage, or redundancy information relating to data redundancy in the data storage.

4. The non-transitory machine-readable storage medium of claim 1, wherein the modification of the placement of the second control plane entity is part of a failover process or a load balancing process.

5. The non-transitory machine-readable storage medium of claim 1, wherein the control plane entity comprises a Kubernetes enclosure mapped to the corresponding data path service partition.

6. The non-transitory machine-readable storage medium according to claim 1, wherein, The first control plane entity activates the first data path service partition in the first computing node based on the interaction between the first control plane entity of the first computing node and the data plane process, wherein the first data path service partition includes elements that are part of the data plane.

7. The non-transitory machine-readable storage medium of claim 1, wherein the instructions, when executed, cause the system to: The placement of the second control plane entity is modified by removing an existing instance of the second control plane entity from a given compute node in the compute node cluster and launching a new instance of the second control plane entity in another compute node in the compute node cluster.

8. The non-transitory machine-readable storage medium of claim 7, wherein the instructions, when executed, cause the system to: The existing instance of the second data path service partition in the given compute node is activated by the existing instance of the second control plane entity; and The new instance of the second data path service partition in the other computing node is activated by the new instance of the second control plane entity.

9. The non-transitory machine-readable storage medium of claim 7, wherein during the modification of the placement of the second control plane entity, both the existing instance of the second control plane entity and the new instance of the second control plane entity coexist, and wherein the instructions, when executed, cause the system to: Arbitration is performed between the existing instance and the new instance of the second control plane entity using a lock, allowing only one of the existing instance and the new instance of the second control plane entity to proceed.

10. The non-transitory machine-readable storage medium of claim 1, wherein the instructions, when executed, cause the system to: In response to a fault condition in the first compute node, the first control plane entity is automatically terminated in the first compute node, wherein, as part of the automatic termination, the first control plane entity deactivates the first data path service partition in the first compute node.

11. The non-transitory machine-readable storage medium according to claim 1, wherein, During the setup or modification of the compute node cluster, the control plane entity performs scheduling among the various compute nodes.

12. The non-transitory machine-readable storage medium of claim 1, wherein the instructions, when executed, cause the system to: The health status of each computing node is monitored by an agent in each computing node; Based on the first agent detecting that the first computing node is unhealthy, a health indicator indicating that the first computing node is unhealthy is provided, and the process of disabling the first computing node is initiated; as well as In response to the health indicator indicating that the first compute node is unhealthy, the placement of the first control plane entity is changed to another compute node in the compute node cluster.

13. The non-transitory machine-readable storage medium of claim 12, wherein the instructions, when executed, cause the system to: Based on monitoring performed by the agent, the isolation between the first subset of compute nodes in the compute node cluster and the second subset of compute nodes in the compute node cluster is detected; Compare the first number of computing nodes in the first subset of computing nodes with the second number of computing nodes in the second subset of computing nodes; as well as Based on the fact that the first quantity is greater than the second quantity, the second subset of computing nodes is disabled.

14. The non-transitory machine-readable storage medium of claim 12, wherein the instructions, when executed, cause the system to: Based on the health indicator from the agent, select which compute nodes in the compute node cluster to schedule the control plane entity.

15. A system comprising: A computing node cluster, the computing node cluster including hardware processors; and Scheduler, the scheduler is used for: The control plane entities in each compute node of the compute node cluster are scheduled, wherein the control plane entities are proxies for each data path service partition in each compute node, and Based on the status of the computing node cluster, the placement of the first control plane entity in the control plane entity is modified to change the placement of the first data path service partition associated with the first control plane entity in the computing node cluster. A first control plane entity is used to activate the first data path service partition in the first computing node in the computing node cluster. as well as The first data path service partition in the first computing node is used to access data in the first data shard in the data storage, wherein the data in the data storage is partitioned into multiple data shards associated with each of the data path service partitions.

16. The system of claim 15, wherein the computing nodes of the computing node cluster include node problem detectors, the node problem detectors being used to: The compute node is detected to be unhealthy, and A notification indicating that the compute node is unhealthy is sent to the control system, wherein the control system includes the scheduler, and the status of the cluster of compute nodes placed by the first control plane entity is modified to include the unhealthy compute node.

17. The system of claim 15, wherein the first control plane entity is part of the control plane of the first computing node, and the first data path service partition is part of the data plane of the first computing node.

18. The system according to claim 15, wherein, When the first control plane entity is placed in the first compute node, it is used for: Identify the second compute node where the existing instance of the first data path service partition is located. Send a stop notification to the second compute node to terminate the existing instance of the first data path service partition in the second compute node, and In response to receiving confirmation that the existing instance of the first data path service partition has been terminated in the second compute node, a new instance of the first data path service partition is activated in the first compute node.

19. A method comprising: The control plane scheduler is used to schedule control plane entities in each compute node of the compute node cluster, wherein the control plane entities are part of the control plane and the placement of the control plane entities in each compute node reflects the placement of each data path service partition, which is part of the data plane used to access the corresponding data shard in the data storage. The first data path service partition in the first computing node is activated by the first control plane entity in the control plane entity of the first computing node in the computing node cluster. The data in the data storage is accessed using the first data path service partition in the first computing node, wherein the data in the data storage is partitioned into multiple data shards associated with each of the data path service partitions. as well as Based on the changed state of the computing node cluster, the placement of the first control plane entity is modified to evict the first control plane entity from the first computing node, and the first control plane entity is placed in the second computing node of the computing node cluster.

20. The method of claim 19, wherein each computing node in the computing node cluster includes a service daemon, and a control plane entity interacts with the service daemon to activate or deactivate a corresponding data path service partition, the method comprising: After the first control plane entity is placed in the second compute node: The existing instance of the first data path service partition in the first computing node is determined by the first control plane entity in the second computing node. The first control plane entity in the second compute node sends a stop notification to the service daemon in the first compute node to terminate the existing instance of the first data path service partition. Based on the confirmation received that the existing instance of the first data path service partition has been terminated, the first control plane entity in the second compute node activates a new instance of the first data path service partition in the second compute node by sending a start notification to the service daemon in the second compute node.

Images