PLACEMENT OF DATA PATH SERVICE PARTITIONS

Abstract

In some examples, a control plane scheduler schedules control plane units in a cluster of compute nodes, with the placement of the control plane units in the respective compute nodes reflecting where the respective data path service partitions are to be placed. A first control plane unit in a first compute node activates a first data path service partition in the first compute node. The first data path service partition is used to access data of a first data unit in a data store, where the data of the data store is partitioned across a multitude of data units that are associated with the respective data path service partitions. Based on a state of the compute node cluster, the placement of a second control plane unit is modified to change the placement of a second data path service partition associated with the second control plane unit in the compute node cluster.

Description

background

[0001] A data storage system comprises a pool of storage devices used to store data. In some examples, the storage devices might include flash memory devices arranged in groups of "Just a Bunch of Flash" (JBOF). In other examples, the storage devices might include disk-based storage devices or other types of storage devices. Brief description of the drawings

[0002] Some embodiments of the present disclosure are described with reference to the following figures. Fig. is a block diagram of an arrangement with a cluster of computing nodes, a control system, and a storage system, according to some examples. Fig. and Fig. These are block diagrams of example configurations with different sets of computation nodes, according to some examples. Fig. is a block diagram of a storage medium that stores machine-readable instructions according to some examples. Fig. This is a block diagram of a system, based on some examples. Fig. This is a flowchart of a process, based on some examples.

[0003] In the drawings, identical reference numbers denote similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated for better illustration of the example shown. Furthermore, the drawings contain examples and / or embodiments that correspond to the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0004] Distributed data partitioning can be applied to data in a datastore to create data shards, where a "data shard" can refer to a partition or segment of the data in the datastore. A cluster of compute nodes can be used to enable distributed access to the data shards in the datastore. Each compute node can run one or more data path service partitions, with each data path service partition supporting input / output access to a corresponding data shard. Multiple data path service partitions can access their respective data shards concurrently.

[0005] Deciding how to allocate data path service partitions to compute nodes can be complex. Allocating data path service partitions across a cluster of compute nodes may involve initial planning of the data path service partitions on selected compute nodes. Following this initial planning, the placement of the data path service partitions on the compute nodes may be changed if conditions within the compute node cluster change. Changing conditions might include failures or errors in a compute node, a compute node losing connectivity to the data storage, an unbalanced workload distribution where one compute node might be performing more workloads than another, the addition or removal of compute nodes, or any other factor that can affect how well workloads perform within the compute node cluster.

[0006] Other problems associated with using a cluster of compute nodes for distributed access to data sets in the data store may include one or a combination of the following: the potential for data corruption associated with multiple instances of a data path service partition accessing the same data shard, the placement of data path service partitions in unhealthy compute nodes, determining when compute nodes need to be disabled, excessive failover times when transitioning workloads from one unhealthy compute node to another, or other issues.

[0007] In accordance with some implementations of this disclosure, a control plane planner for a cluster of compute nodes connected to a data store can schedule data path service partition proxies on the respective compute nodes, instead of placing data path service partitions directly on the compute nodes. A proxy is a control unit responsible for enabling and disabling a particular data path service partition on the compute node. The proxy (referred to as a "data partition proxy") is part of the control plane, while a data path service partition enabled or disabled by the data partition proxy is part of a data plane.

[0008] In some examples, each data partition proxy runs in a container that is a virtual machine. In certain examples, the data partition proxy runs in a container within a Kubernetes pod. In other examples, a data partition proxy can run in a virtual machine (VM). In still other examples, a data partition proxy can contain a program. By placing data partition proxies in appropriate compute nodes, the scheduler does not need to be configured to support data path service partitions. Instead, the scheduler schedules the data partition proxies in selected compute nodes, with the placement of the data partition proxies reflecting the schedule for where the respective data path service partitions should be placed.In other words, the scheduler indirectly places data path service partitions in compute nodes based on the placement of data partition proxies in the compute nodes.

[0009] A "computer node" can refer to one or more physical computers. A "data path service partition" refers to any collection of modules and data structures used to access a data shard in response to a request (such as a read or write request) from a requester. Examples of modules that may be included in a data path service partition include any one or a combination of the following: a caching module that manages the temporary storage of data (such as temporarily storing write data in a write buffer), an indexing module that manages the indexing of data, or a garbage collector that manages the removal of data from a storage system after a delete operation.Examples of data structures that may be contained in a data path service partition include any one or a combination of the following elements: . A write buffer for temporarily storing write data to be written to the storage system, an index that maps objects to storage locations in the storage system, or redundancy information relating to data redundancy in the storage system, such as Redundant Array of Independent Disks (RAID) redundancy. Although example modules and data structures are listed above, a data path service partition may contain additional or alternative modules or data structures in other examples.

[0010] In some examples, a data path service partition forms a vertical slice of an I / O stack for a storage system. Multiple data path service partitions thus form multiple vertical slices of the I / O stack, which can access their respective data slots independently and in parallel. An I / O stack comprises the modules and data structures involved in data access operations for a corresponding data slot. A vertical slice of the I / O stack refers to a specific instance of the modules and data structures for the corresponding data shard.

[0011] Fig. is a block diagram of an example arrangement comprising a cluster of compute nodes 102-1, 102-2, and 102-3. Although in Fig. In one example, three computing nodes are represented; in another example, a cluster may comprise a different number of computing nodes.

[0012] Computing nodes 102-1 to 102-3 communicate with a storage system 106 via a connection 104. In some examples, the storage system 106 comprises an array of storage devices, such as flash storage devices, disk-based storage devices, or other types of storage devices. An "interconnect" can refer to a communication medium over which data is transmitted. Examples of the connection 104 could be one or a combination of the following: a local area network (LAN), a storage area network (SAN), a wide area network (WAN), or another type of communication link.

[0013] In some examples, data communication via Interconnect 104 takes place according to a protocol such as the Non-volatile Memory Express (NVMe)-over-Fabric (NVMEoF) protocol, the Internet Small Computer Systems Interface (iSCSI) protocol, the Infiniband protocol, or another standardized, open-source, or proprietary protocol.

[0014] A control system 108 is assigned to the cluster of compute nodes 102-1 to 102-3. The control system 108 can be implemented on one or more computers. In some cases, the elements of the control system 108 can be implemented on one or more of the compute nodes 102-1 to 102-3. In other examples, the control system 108 can be separate from the compute nodes 102-1 to 102-3.

[0015] Control system 108 includes a scheduler 110, which schedules data partition proxies in compute nodes 102-1 through 102-3 and controls where data path service partitions are placed. Although referred to as a single unit, scheduler 110 can comprise multiple modules. For example, if control system 108 is a Kubernetes control system, scheduler 110 might include the following modules: a Kubernetes scheduler, a Kubernetes descheduler, and a mapper. Generally, a Kubernetes scheduler is responsible for placing pods in compute nodes, where a pod can contain one or more containers. The Kubernetes scheduler places pods in compute nodes based on a pod specification (which contains constraints that determine how the Kubernetes scheduler should place the pods). A Kubernetes descheduler complements the actions of the Kubernetes scheduler by creating pods in response to certain events, such as...Compute node failures are handled by re-distributing pods across the remaining compute nodes. Additionally, the descheduler can (e.g., at regular intervals) review the current pod placements and ensure they comply with the constraints defined in the pod specification. If the pod placements do not meet the constraints, the descheduler can create a eviction plan to remove pods from a compute node and send information about the removed pods to the Kubernetes scheduler for rescheduling. The mapper provisiones pods during cluster setup or modification (e.g., adding or removing a compute node).

[0016] In other examples, Scheduler 110 can be implemented using a different technology. Generally, Scheduler 110 can be implemented using machine-readable instructions that are executable on a processing resource of Control System 108.

[0017] The control system 108 also includes an interface server 118, with which the components in the compute nodes 102-1 to 102-3 can communicate. In some examples, the interface server 118 includes an API server (Application Programming Interface) that provides an API which the components in the compute nodes 102-1 to 102-3 can access. Communication between the components in the compute nodes 102-1 to 102-3 and the control system 108 can be carried out using routines from the API. In other examples, the interface server 118 can provide a different type of interface for access by the components in the compute nodes 102-1 to 102-3.

[0018] In some examples, data can be stored as blocks in the storage devices of Storage System 106. A "block" can refer to a chunk or other piece of data. The chunks are stored in appropriate storage locations within Storage System 106. Although the data is stored as chunks in Storage System 106, clients can access objects, where an object comprises an arrangement of chunks stored in Storage System 106. An object can, for example, be part of object storage, such as object storage provided by the Amazon Simple Storage Service (S3). S3 object storage stores data as objects. In the example of Fig. Clients 114 are presented with a global object namespace 112. This global object namespace contains the names of objects that clients 114 can access. A "client" can refer to an electronic device or a program.

[0019] In examples using S3 object storage, objects can be contained in buckets 116. A bucket 116 can contain one or more objects. The objects in each bucket 116 can be divided into data shards, which the data path service partitions in compute nodes 102-1, 102-2, and 102-3 can access.

[0020] In other examples, different types of object storage can be used to store data that clients 114 can access. For instance, another type of object storage can store files, and the global object namespace 112 can be replaced by a file system in this case. Regardless of the type of object storage used, the data for the objects in the object storage is split into data splitters via compute nodes 102-1 to 102-3, which are then accessed by the respective data path service partitions.

[0021] Each data path service partition is divided into Fig. represented as "DSP". In the example of Fig. Two data path service partitions (DSP A and DSP B) are used in compute node 102-1, one data path service partition (DSP C) in compute node 102-2, and two data path service partitions (DSP D and DSP E) in compute node 102-3. Although a specific number of DSPs is specified for each compute node, a compute node may operate a different number of DSPs in other examples.

[0022] Scheduler 110 places data partition proxies (in Fig. (represented as DSP proxies) in corresponding compute nodes. In the example of Fig. DSP-Proxy A and DSP-Proxy B are placed by planner 110 in compute node 102-1, DSP-Proxy C is placed by planner 110 in compute node 102-2, and DSP-Proxy D and DSP-Proxy E are placed by planner 110 in compute node 102-3.

[0023] A DSP proxy is part of the control plane, while a Data Path Service Partition (DSP) is part of the data plane. The data plane defines the path that data takes between the storage system and a client requesting access to the data (read or write). A control plane includes control functions that manage the data plane.

[0024] The scheduler 110 places DSP proxies in the compute nodes 102-1, 102-2, and 102-3 according to the constraint information 120 stored in a memory 124 of the control system 108. The constraint information 120 contains one or more rules that determine where DSP proxies should be placed to achieve a goal, such as distributing DSPs evenly across the compute nodes 102-1 to 102-3, avoiding the placement of a DSP on a particular compute node, or any other rule.

[0025] In some examples, a DSP is associated with a service daemon (SD) process. The SD process can be a user-space process that runs in the user space of a compute node to call services, such as a DSP. In the example of Fig. An SD process 122-1 is executed in computer node 102-1, an SD process 122-2 in computer node 102-2 and an SD process 122-3 in computer node 102-3.

[0026] In some examples, an SD process is containerized (to enclose one or more containers) and implements the data plane in a compute node. A DSP runs in a container of the SD process. In other examples, DSPs can be implemented as microservices managed by the SD process. In still other examples, DSPs can run in VMs.

[0027] In the example of Fig. DSP A and DSP B are assigned to SD process 122-1 (e.g., DSP A and DSP E are executed in corresponding containers of SD process 122-1), DSP C is assigned to SD process 122-2, and DSP D and DSP E are assigned to SD process 122-3.

[0028] Once a DSP proxy is placed in a compute node, the DSP proxy communicates with a corresponding SD process to activate a corresponding DSP in the compute node. For example, DSP proxy A instructs SD process 122-1 to activate DSP A, and DSP proxy B instructs SD process 122-1 to activate DSP B. Similarly, DSP proxy C instructs SD process 122-2 to activate DSP C, DSP proxy D instructs SD process 122-3 to activate DSP D, and DSP proxy E instructs SD process 122-3 to activate DSP E. An activated DSP begins running in a compute node and uses the compute node's resources (including processing, memory, and communication resources).

[0029] A DSP proxy can also communicate with an SD process to disable a DSP. Enabling a DSP can refer to starting a DSP or signaling the DSP to run. Disabling a DSP can refer to closing a DSP or signaling the DSP to stop running. Communication between a DSP proxy and an SD process can occur via a messaging interface, such as a Remote Procedure Call (RPC) interface. An example of an RPC interface is a gRPC interface. Another messaging interface that can be used is a Representational State Transfer (REST) ​​interface.

[0030] In some examples, an SD process is stateless, meaning it does not maintain persistent information about enabled DSPs associated with it. When the SD process restarts (shuts down and restarts), DSP proxies on the same compute node detect the restart and can instruct the SD process to re-enable the corresponding DSPs. In some examples, the DSP proxies persistently retain the information about enabled DSPs, allowing the SD process to re-enable the DSPs upon restarting.

[0031] In other examples, an SD process is omitted. In these other examples, a DSP proxy can directly enable or disable DSPs without interacting with an SD process.

[0032] Scheduler 110 starts DSP proxies on the respective compute nodes 102-1 to 102-3 during cluster setup or when a compute node is added. After cluster setup, Scheduler 110 can also add more DSP proxies to a compute node or remove DSP proxies from a compute node if the conditions of the cluster of compute nodes 102-1 to 102-3 change. Scheduler 110 can also migrate a DSP proxy from one compute node to another, for example, to balance the load of compute nodes 102-1 to 102-3, or more generally, if Scheduler 110 determines that a current placement of DSPs on compute nodes 102-1 to 102-3 does not comply with the rule(s) of constraint information 120.

[0033] A DSP proxy is capable of handling an error condition when attempting to activate or deactivate a DSP on a compute node. For example, the DSP proxy can self-terminate on a compute node if the SD process on the compute node returns an error (such as an out-of-resource error, indicating that the SD process has detected that the compute node's resources are exhausted). Self-termination of a DSP proxy involves the DSP proxy closing itself on the compute node and notifying Scheduler 110 so that Scheduler 110 can reschedule the DSP proxy on a different compute node. As part of the self-termination process, the DSP proxy also deactivates the corresponding DSP.

[0034] In some examples, each compute node also contains a node problem detector, which may be implemented as machine-readable instructions. For example, compute node 102-1 contains a node problem detector 130-1, compute node 102-2 contains a node problem detector 130-2, and compute node 102-3 contains a node problem detector 130-3. Each node problem detector is capable of monitoring the conditions of a compute node to determine if the compute node is unhealthy. For example, the node problem detector can detect if the compute node has lost connection to storage system 106, perhaps due to a hardware or software error. In other examples, the node problem detector can detect if certain programs have crashed or are malfunctioning.For example, if the cluster of compute nodes forms a Kubernetes cluster, the node problem detector can detect whether a kubelet is running on a compute node. The kubelet is a node agent that verifies whether containers are running correctly.

[0035] In other examples, the node problem detector is able to monitor various metrics, such as resource usage metrics (e.g., processing resources, memory resources, communication resources, etc.), resource performance metrics, error metrics, or other types of metrics. Based on these metrics, the node problem detector can detect or predict an unhealthy state of the compute node. Generally, an "unhealthy" state refers to a condition of a compute node where its performance is or will fall below a target value, or where errors or malfunctions will occur.

[0036] A node problem detector can send a notification to the control system 108 (e.g., to the interface server 118) when it detects an unhealthy compute node. The notification can be in the form of a message, a signal, an information element, or another indicator that specifies the unhealthy state of a compute node. In response to the notification, the scheduler 110 can migrate DSP proxies from the unhealthy compute node to one or more target compute nodes.

[0037] In some examples, a compute node may additionally contain a lease manager (e.g., implemented as machine-readable instructions) that manages a node lease of the compute node. Fig. Compute node 102-1 contains a lease manager 140-1, compute node 102-2 contains a lease manager 140-2, and compute node 102-3 contains a lease manager 140-3. A "node lease" refers to a status indicator that shows whether the compute node is functioning correctly. As long as the compute node is functional and communicating with other compute nodes in the cluster (i.e., the compute node is not isolated from the cluster), the lease manager maintains the node lease and continues to update the watchdog timer. A watchdog timer 142-1 is located in compute node 102-1, a watchdog timer 142-2 is located in compute node 102-2, and a watchdog timer 142-3 is located in compute node 102-3.In some examples, a watchdog timer is a hardware timer that runs for a specific period of time, after which the watchdog timer expires.

[0038] Refreshing the watchdog timer can involve signaling the watchdog timer to restart its counter. If the lease manager detects that a specific compute node has lost communication with the cluster, it stops the watchdog timer restart for that node. The watchdog timer's expiration triggers a reset of the affected node.

[0039] The lease manager in the given compute node also maintains a representation of the node lease, for example, by storing an information element in a memory of the given compute node. The information element, which is set to a specific value, indicates that the node lease is active on the given compute node.

[0040] Control System 108 and the other compute nodes in the cluster can check the node rental of a given compute node. If Control System 108 or the other compute nodes do not detect the node rental of a given compute node (for example, the node rental is inaccessible because the given compute node has lost its connection to the cluster, has failed, or is rebooting), then Control System 108 or the other compute nodes can determine that the given compute node is no longer available. Scheduler 110 in Control System 108 would not place any new DSP proxies on the given compute node, and furthermore, Scheduler 110 can reschedule all DSP proxies on the given compute node to one or more other compute nodes in the cluster.

[0041] Although the example of Fig. While the presence of both the Node Problem Detector and the Lease Manager on each compute node is indicated in one example, the Lease Manager can be omitted in other examples. In such other examples, the Node Problem Detector would also verify the connectivity of a given compute node to the cluster. As long as the Node Problem Detector confirms that the compute node in question is connected to the cluster, it can send a heartbeat message or other indicator to Control System 108 and the other compute nodes. If the given compute node becomes unavailable (for example, due to loss of connection to the cluster, failure of the given compute node, or restart of the given compute node), then the Node Problem Detector stops sending the heartbeat message or other indicator to Control System 108 and the other compute nodes.If this heartbeat message or other indicator is not received, this is an indication that the relevant compute node is unavailable.

[0042] Fig. Figure 1 shows an example configuration with the control system 108 and two compute nodes 202-1 and 202-2, which are part of a compute node cluster. Compute node 202-1 includes DSP Proxy A, DSP Proxy B, and DSP Proxy C, which were placed by the scheduler 110 in a control plane 204-1 of compute node 202-1. DSP Proxy A has DSP A enabled, DSP Proxy B has DSP B enabled, and DSP Proxy C has DSP C enabled. DSP A, DSP B, and DSP C are part of a data plane 206-1 of compute node 202-1.

[0043] Similarly, compute node 202-2 contains a DSP proxy D, a DSP proxy E, and a DSP proxy F, which were placed by scheduler 110 in a control plane 204-2 of compute node 202-2. DSP proxy D has enabled DSP D, DSP proxy E has enabled DSP E, and DSP proxy F has enabled DSP F. DSP D, DSP E, and DSP F are part of a data plane 206-2 of compute node 202-2.

[0044] Fig. This shows an example where another compute node, 202-3, has been added to the cluster. Due to the addition of compute node 202-3, scheduler 110 detects an imbalance in the placement of the DSP proxies, since three DSP proxies are running on each of compute nodes 202-1 and 202-2, and no DSP proxies are running on the added compute node 202-3. This imbalance can be corrected against a rule in constraint information 120 ( Fig. The rule that the DSPs should be evenly distributed across the cluster's compute nodes was violated. As a result, Scheduler 110 removed DSP proxy C from compute node 202-1 and DSP proxy F from compute node 202-2. Scheduler 110 then placed DSP proxy C and DSP proxy F in a control plane 204-3 added to compute node 202-3.

[0045] The DSP proxy C on compute node 202-3 can retrieve information about the last location of DSP C. In some examples, SD processes on the cluster's compute nodes can store information identifying the compute nodes on which the respective DSPs are running in a shared memory. This information can be retrieved by a DSP proxy placed on a new compute node by interacting with the SD process on that new compute node. Once DSP proxy C determines that DSP C was last located on compute node 202-1, DSP proxy C on compute node 202-3 can send a stop notification to the SD process on compute node 202-1 to terminate DSP C on compute node 202-1.In response to receiving confirmation (in the form of a notification from the SD process in compute node 202-1) that DSP C in compute node 202-1 has terminated, DSP proxy C in compute node 202-3 starts DSP C in compute node 202-3 by sending a start notification to the SD process in compute node 202-3. DSP proxy F in compute node 202-3 performs a similar process. Generally, a DSP proxy placed in a new compute node will not activate a DSP in the new compute node until the DSP proxy receives confirmation that the DSP in a previous compute node has terminated.

[0046] In some examples where node leases are used, a DSP proxy placed on a new compute node also checks whether the node lease of the previous compute node is active. If the node lease is not active (meaning the previous compute node is unavailable), the DSP proxy can skip sending the stop notification to the SD process of the previous compute node.

[0047] If a compute node (e.g., 202-1) from the cluster in the arrangement of Fig. If the DSP proxy is removed, then Scheduler 110 removes DSP proxies A and B from compute node 202-1, and Scheduler 110 places DSP proxies A and B in the remaining compute nodes 202-2 and 202-3, respectively. DSP proxy A placed in compute node 202-2 activates DSP A in compute node 202-1, and DSP proxy B placed in compute node 202-3 activates DSP A in compute node 202-3.

[0048] One goal (referred to as Goal A) of Scheduler 110 is to prevent the activation of multiple DSPs on multiple compute nodes for the same data shard (a split-brain condition) to avoid data corruption caused by multiple DSPs making inconsistent changes to the data shard. In the scenarios mentioned above, which involve adding a compute node to or removing a compute node from a cluster, there may be multiple instances of the DSP proxy for a specific DSP (identified by a particular DSP identifier) ​​during the process of evacuating a DSP proxy from a source compute node and placing the DSP proxy on a target compute node. To achieve Goal A, a lock can be used to prevent the multiple instances of the DSP proxy for the specific DSP identifier from being active simultaneously.In examples using a Kubernetes cluster, the lock can be a variable contained in the configuration data of a ConfigMap, which is an API object for storing data as key-value pairs. In other examples, the lock can be implemented using any locking mechanism, such as a mutex or semaphore. The lock is used to mediate between multiple instances of the DSP proxy for the specific DSP identifier, ensuring that only one of the multiple DSP proxy instances can proceed. The DSP proxy instance on the target compute node can acquire the lock (e.g., by creating a ConfigMap) when the lock is released by the DSP proxy instance on the source compute node (e.g., by deleting the ConfigMap).

[0049] Another goal (referred to as goal B) is a liveness goal for DSPs to prevent a DSP from becoming unavailable, as an unavailable DSP would mean that the data in the data shard connected to the unavailable DSP would be inaccessible to clients. A DSP can become unavailable if a compute node fails or is isolated (due to loss of connectivity to the cluster of compute nodes). In some examples, node problem detectors (e.g., 130-1 to 130-3 in Fig. ) and / or lease managers (e.g., 140-1 to 140-3 in Fig. ) the identification of unhealthy compute nodes or isolated compute nodes, so that DSP proxies can be evacuated to the unhealthy compute node or isolated compute node and placed on the remaining compute nodes of the cluster.

[0050] In some examples, multiple compute nodes can be isolated, so the cluster is divided into a first subset of compute nodes that can communicate with each other and a second subset of compute nodes that can communicate with each other. However, the compute nodes in the first subset cannot communicate with the compute nodes in the second subset. In such a scenario, the subset with the larger number of compute nodes is considered the majority subset, and the subset with the smaller number of compute nodes is considered the minority subset. Scheduler 110 causes the DSP proxies in the minority subset of compute nodes to be evicted, and Scheduler 110 places the evicted DSP proxies in the majority subset of compute nodes.

[0051] Another goal (referred to as Goal C) is to reduce the DSP failover time when a DSP proxy is evacuated from an unavailable compute node (which has failed or lost connection) and placed on a target compute node. If the DSP failover time is too long, clients may experience timeouts while waiting for requested data, which can negatively impact usability or lead to errors. Goal C can be achieved by configuring Scheduler 110 to start a new instance of the DSP proxy on the target compute node with minimal delay as soon as the unavailable compute node is detected.The combination of using the lock (for target A) and the fact that the previous instance of the DSP proxy is located on an unavailable compute node means that a fast start of the new instance of the DSP proxy on the target compute node is unlikely to result in an error condition where multiple DSP proxy instances are active simultaneously.

[0052] A cluster of compute nodes running appropriate DSPs provides a disaggregated architecture for accessing data shards of a data store. As cluster resources change (computation nodes are added or removed), the DSPs can scale accordingly. For example, DSPs can be migrated to other compute nodes. Additional DSPs can be created when new data shards are added. High availability is achieved by detecting unhealthy or isolated compute nodes, allowing DSPs to be removed from these nodes and placed on other nodes within the cluster. Indirectly placing DSPs through a scheduler in a control system based on DSP proxy placement eliminates the need to modify the scheduler to directly support DSPs, thus reducing scheduler development time.

[0053] Fig. is a block diagram of a non-transitory machine-readable or computer-readable storage medium 300 that stores machine-readable instructions of a control plane of a cluster of compute nodes. For example, the machine-readable instructions of the control plane can be part of the control system 108 and the compute nodes 102-1 to 102-3 of Fig. be.

[0054] The machine-readable instructions include instructions 302 for scheduling control plane units, in order to schedule control plane units in corresponding compute nodes of the cluster of compute nodes using a control plane scheduler. The control plane units include DSP proxies, as shown in Fig. and Fig. The placement of the control plane units in the respective compute nodes reflects where the respective data path service partitions are to be placed. Examples of data path service partitions are the DSPs in Fig. and Fig. .

[0055] The machine-readable instructions include first control plane entity activation instructions 304 to activate a first data path service partition in the first compute node by a first control plane entity of the control plane entities in a first compute node of the cluster of compute nodes.

[0056] The machine-readable instructions include Data Set Access Instructions 306 to access data on a first data shard in a data store using the first data path service partition in the first compute node. The data in the data store is partitioned across a plurality of data shards, each associated with its respective data path service partition. Access to the first data shard utilizes modules and data structures from the first data path service partition.

[0057] The machine-readable instructions include instruction 308 for changing the placement of control plane entities. This involves changing the placement of a second control plane entity, based on the state of the compute node cluster, and changing the placement of a second data path service partition associated with that second control plane entity within the compute node cluster. Note that the second control plane entity can be identical or different from the first control plane entity, and the second data path service partition can be identical or different from the first data path service partition.

[0058] In some examples, the data path service partitions each comprise different slices of an I / O stack in a data path between the cluster of compute nodes and the data storage.

[0059] In some examples, the first data path service partition contains elements selected from: a caching module to manage data caching, an index manager, a garbage collector, a buffer to store data, an index that maps objects to storage locations in the data store, or redundancy information relating to data redundancy in the data store.

[0060] In some examples, changing the placement of the second control plane unit is part of a failover process (to fail over from an unhealthy or isolated compute node) or a load balancing process (in response to the detection of an imbalance in the placement of control plane units).

[0061] In some examples, the control plane units comprise Kubernetes pods that are associated with the respective data path service partitions.

[0062] In some examples, the first control plane unit activates the first data path service partition in the first compute node based on an interaction between the first control plane unit and a data plane process (e.g., an SD process) of the first compute node, where the first data path service partition contains elements that are part of a data plane.

[0063] In some examples, the machine-readable instructions to change the placement of the second control plane unit remove an existing instance of the second control plane unit from a given compute node of the cluster of compute nodes and start a new instance of the second control plane unit in another compute node of the cluster of compute nodes.

[0064] In some examples, the existing instance of the second control plane unit disables an existing instance of the second data path service partition in the given compute node, and the new instance of the second control plane unit enables a new instance of the second data path service partition in the further compute node.

[0065] In some examples, during the change of the placement of the second control plane entity, the existing instance of the second control plane entity and the new instance of the second control plane entity exist simultaneously, and the machine-readable instructions can mediate using a lock between the existing instance of the second control plane entity and the new instance of the second control plane entity to allow only one of the existing instance of the second control plane entity and the new instance of the second control plane entity to proceed.

[0066] In some examples, the first control plane unit in the first compute node may terminate itself in response to an error condition in the first compute node. As part of the self-termination, the first control plane unit disables the first data path service partition in the first compute node.

[0067] In some examples, the scheduling of the control plane entities in the respective compute nodes is performed during the setup of the compute node cluster or a change to the compute node cluster (due to adding a new compute node or removing an existing compute node).

[0068] In some examples, agents in the respective compute nodes can monitor the state of those nodes. An "agent" can be a node problem detector and / or a lease manager, as in Fig. As shown, based on a first agent's detection that the first compute node is not healthy, a health indicator is provided to show that the first compute node is not healthy. The health indicator might, for example, include a notification from a node problem detector. As another example, the health indicator might include a node lease (the absence of a node lease indicates that the respective compute node has been isolated from the cluster). In response to the health indicator showing that the first compute node is unhealthy, a process is initiated to disable the first compute node and remove all control plane units from it. For example, the first compute node might be disabled by stopping a watchdog timer on the first compute node from updating.In another example, the first compute node can be disabled by no longer placing control plane units and corresponding data path service partitions on the first compute node. In response to the health indicator showing that the first compute node is unhealthy, the machine-readable instructions change the placement of the first control plane unit to another compute node in the cluster.

[0069] In some examples, the machine-readable instructions, based on agent monitoring, can detect that a first subset of compute nodes in the cluster is isolated from a second subset of compute nodes in the same cluster. The machine-readable instructions can compare a first set of compute nodes in the first subset with a second set of compute nodes in the second subset. Based on the first set being larger than the second, the machine-readable instructions can disable the second subset of compute nodes.

[0070] In some examples, the machine-readable instructions can select which compute nodes of the cluster of compute nodes the control plane units should schedule based on agent state indicators.

[0071] Fig. This is a block diagram of a System 400 according to some examples. The System 400 comprises a cluster 402 of compute nodes 403-1 and 403-2, which contain hardware processors 404-1 and 404-2. A hardware processor can include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or other hardware processing circuitry.

[0072] System 400 includes a scheduler 406, which can be executed in a control system to perform various tasks. One of the tasks of scheduler 406 is task 408, which schedules control plane entities 410-1 and 410-2 in their respective compute nodes of cluster 402. The control plane entities are proxies for the respective data path service partitions 412-1 and 412-2 in the respective compute nodes.

[0073] The tasks of scheduler 406 include a task 414 to change the placement of the control plane unit, in order to change a placement of the first control plane unit 410-1 based on a state of cluster 402, in order to change a placement of compute nodes of the first data path service partition 412-1, which is associated with the first control plane unit 410-1, in cluster 402.

[0074] The first control plane unit 410-1 activates the data path service partition 412-1 in the first compute node 403-1. The first data path service partition 412-1 in the first compute node 403-1 accesses data from a first data shard in a data store, the data of which is partitioned across a plurality of data shards connected to the respective data path service partitions.

[0075] Fig. is a flowchart of a process 500 controlled by a control system (e.g., 108 in Fig. ) and computer nodes (e.g., 102-1 to 102-3 in Fig. ) can be carried out.

[0076] Process 500 involves scheduling (at 502), using a control plane scheduler, control plane entities in respective compute nodes of the cluster of compute nodes, wherein the control plane entities are part of the control plane and reflect the placements of the control plane entities in the respective compute nodes where respective data path service partitions are to be placed, wherein the respective data path service partitions are part of a data plane to access corresponding data shards in a data store.

[0077] Process 500 involves the activation (at 504) of a first data path service partition in the first compute node by a first control plane unit of the control plane units in a first compute node of the cluster of compute nodes. The activation can, for example, involve sending a start message to an SD process in the first compute node.

[0078] Process 500 involves accessing (at 506) using the first data path service partition in the first compute node to data of a first data shard in the data store, wherein the data of the data store is partitioned across a plurality of data shards connected to the respective data path service partitions.

[0079] Process 500 involves modifying (at 508), based on a changed state of the cluster of compute nodes, placing the first control plane unit, removing the first control plane unit from the first compute node, and placing the first control plane unit in a second compute node of the cluster of compute nodes.

[0080] In some examples, each compute node in the cluster contains an SD process that interacts with a control plane unit to enable or disable a corresponding data path service partition. Process 500 further includes, after the first control plane unit is placed in the second compute node, the determination by the first control plane unit in the second compute node that an existing instance of the first data path service partition exists in the first compute node. The first control plane unit in the second compute node then sends a stop message to the SD process in the first compute node to terminate the existing instance of the first data path service partition.Based on receiving confirmation that the existing instance of the first data path service partition has been terminated, the first control plane unit in the second compute node activates a new instance of the first data path service partition in the second compute node by sending a start message to the SD process in the second compute node.

[0081] As used here, a "memory" can be implemented with one or more memory chips. A memory chip can comprise any one or a combination of the following: dynamic random-access memory (DRAM), static random-access memory (SRAM), erasable and programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of memory chip.

[0082] A "processing resource" can include one or more hardware processors.

[0083] Machine-readable instructions that can be executed on a hardware processor can refer to instructions that can be executed on a single hardware processor or on multiple hardware processors.

[0084] A storage medium (e.g. 300 in Fig.The storage medium may contain any or a combination of the following: a semiconductor storage device such as DRAM or SRAM, EPROM, EEPROM, or flash memory; a magnetic disk such as a hard disk, floppy disk, or removable disk; another magnetic medium, including tape; an optical medium such as a compact disc (CD) or digital video disc (DVD); or some other type of storage device. It should be noted that the instructions described above may be provided on a single computer- or machine-readable storage medium, or alternatively, on multiple computer- or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer- or machine-readable storage medium or media are considered part of an article (or a manufactured article).An article or manufactured item can refer to any single component or multiple components manufactured. The storage medium or media can be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded for execution over a network.

[0085] In this disclosure, the use of the terms "a", "an", or "the" also includes the plural forms, unless the context clearly indicates otherwise. Likewise, the terms "contains", "including", "comprises", "encompasses", "have", or "have", when used in this disclosure, specify the presence of the elements indicated, but do not exclude the presence or addition of other elements.

[0086] The preceding description presents numerous details to facilitate an understanding of the topic disclosed herein. However, implementations without some of these details are also possible. Other implementations may include modifications and variations of the details described above. The attached claims are intended to cover such modifications and variations.

Claims

[1] Non-transitory, machine-readable storage medium containing instructions of a control plane of a cluster of compute nodes, wherein the instructions, when executed, cause a system to: Planning, using a control plane scheduler, of control plane units in respective compute nodes of the cluster of compute nodes, wherein the placements of the control plane units in the respective compute nodes reflect where respective data path service partitions are to be placed; Activation of a first data path service partition in a first compute node of the cluster of compute nodes by a first control plane unit of the control plane units in the first compute node using the first data path service partition in the first compute node to access data of a first data shard in a data store, wherein the data of the data store is partitioned across a plurality of data shards that are associated with the respective data path service partitions; and Based on the state of the cluster of compute nodes, modify the placement of a second control plane entity of the control plane entities to change the placement of a second data path service partition associated with the second control plane entity in the cluster of compute nodes. [2] Non-transitory machine-readable storage medium according to claim 1, wherein the data path service partitions each comprise different slices of an input / output (I / O) stack in a data path between the cluster of compute nodes and the data storage. [3] Non-transitory machine-readable storage medium according to claim 2, wherein the first data path service partition comprises elements selected from: a caching module for managing the caching of data, an index manager, a garbage collector, a buffer for storing data, an index that maps objects to storage locations in the data storage, or redundancy information relating to data redundancy in the data storage [4] Non-transient machine-readable storage medium according to claim 1, wherein the change in the placement of the second control plane unit is part of a failover process or a load balancing process. [5] Non-transitory machine-readable storage medium according to claim 1, wherein the control plane units comprise Kubernetes pods that are mapped to the respective data path service partitions. [6] Non-transitory machine-readable storage medium according to claim 1, wherein the first control plane unit activates the first data path service partition in the first compute node based on an interaction between the first control plane unit and a data plane process of the first compute node, wherein the first data path service partition comprises elements that are part of a data plane [7] Non-transitory machine-readable storage medium according to claim 1, wherein the instructions, upon execution, cause the system to: to modify the placement of the second control plane unit, remove an existing instance of the second control plane unit from a given compute node of the cluster of compute nodes and start a new instance of the second control plane unit in another compute node of the cluster of compute nodes. [8] Non-transitory machine-readable storage medium according to claim 7, wherein the instructions, upon execution, cause the system to: Deactivating an existing instance of the second data path service partition in the given compute node by the existing instance of the second control plane unit; and Activate a new instance of the second data path service partition in the additional compute node through the new instance of the second control plane unit. [9] Non-transitory machine-readable storage medium according to claim 7, wherein during the modification of the placement of the second control plane unit the existing instance of the second control plane unit and the new instance of the second control plane unit are simultaneously present, and wherein the instructions, upon execution, cause the system to: mediate using a lock between the existing instance of the second control plane unit and the new instance of the second control plane unit, in order to allow only one of the existing instance of the second control plane unit and the new instance of the second control plane unit to proceed. [10] Non-transitory machine-readable storage medium according to claim 1, wherein the instructions, upon execution, cause the system to: Self-termination of the first control plane unit in the first compute node in response to an error condition in the first compute node, wherein as part of the self-termination the first control plane unit disables the first data path service partition in the first compute node. [11] Non-transitory machine-readable storage medium according to claim 1, wherein the scheduling of the control plane units in the respective compute nodes is carried out during the setup of the cluster of compute nodes or a change of the cluster of compute nodes. [12] Non-transitory machine-readable storage medium according to claim 1, wherein the instructions, upon execution, cause the system to: to monitor the state of the respective computing nodes through agents in the respective computing nodes; based on a first agent that detects that the first compute node is unhealthy, provide a health indicator showing that the first compute node is unhealthy, and initiate a process to disable the first compute node; and In response to the health indicator showing that the first compute node is unhealthy, change the placement of the first control plane unit to another compute node in the cluster of compute nodes. [13] Non-transitory machine-readable storage medium according to claim 12, wherein the instructions, upon execution, cause the system to: Based on monitoring by the agents, recognize that a first subset of compute nodes of the cluster of compute nodes is isolated from a second subset of compute nodes of the cluster of compute nodes; Comparing a first set of computation nodes in the first subset of computation nodes with a second set of computation nodes in the second subset of computation nodes; and Based on the fact that the first size is larger than the second size, the second subset of compute nodes is deactivated. [14] Non-transitory machine-readable storage medium according to claim 12, wherein the instructions, upon execution, cause the system to: Selection of the compute nodes of the cluster of compute nodes to plan the tax plane entities based on the agents' health indicators. [15] System comprising the following: a cluster of computing nodes comprising hardware processors; and a planner Planning of control plane units in respective compute nodes of the cluster of compute nodes, wherein the control plane units are proxies for respective data path service partitions in the respective compute nodes, and Based on the state of the cluster of compute nodes, modify the placement of a first control plane entity of the control plane entities to change the placement of a first data path service partition associated with the first control plane entity in the cluster of compute nodes; the first control plane unit to activate the first data path service partition in the first compute node of the cluster of compute nodes; and the first data path service partition in the first compute node to access data of a first data shard in a data store, wherein the data of the data store is partitioned across a plurality of data shards that are connected to the respective data path service partitions. [16] System according to claim 15, wherein a compute node of the cluster of compute nodes comprises a node problem detector to: recognize that the compute node is unhealthy, and Sending a notification to a control system indicating that the compute node is unhealthy, where the control system includes the scheduler and the state of the cluster of compute nodes, which triggers the modification of the placement of the first control plane unit, includes the fact that the compute node is unhealthy. [17] System according to claim 15, wherein the first control plane unit is part of a control plane of the first compute node and the first data path service partition is part of a data plane of the first compute node. [18] System according to claim 15, wherein the first control plane unit, when placed in the first compute node, serves to: Identify a second compute node containing an existing instance of the first data path service partition, Sending a stop notification to the second compute node to terminate the existing instance of the first data path service partition on the second compute node, and In response to receiving confirmation that the existing instance of the first data path service partition in the second compute node has been terminated, activate a new instance of the first data path service partition in the first compute node. [19] Procedure comprising the following: Planning, using a control plane planner, of control plane entities in respective compute nodes of a cluster of compute nodes, wherein the control plane entities are part of the control plane and reflect the placements of the control plane entities in the respective compute nodes where respective data path service partitions are to be placed, wherein the respective data path service partitions are part of a data plane to access corresponding data shards in a data store; Activation of a first data path service partition in a first compute node of the cluster of compute nodes by a first control plane unit of the control plane units in the first compute node; Accessing, using the first data path service partition in the first compute node, data of a first data shard in the data store, wherein data of the data store is partitioned across a plurality of data shards that are associated with the respective data path service partitions; and Based on a changed state of the cluster of compute nodes, modify the placement of the first control plane unit to remove the first control plane unit from the first compute node and place the first control plane unit in a second compute node of the cluster of compute nodes. [20] Method according to claim 19, wherein each compute node of the cluster of compute nodes comprises a service daemon process with which a control plane unit interacts to enable or disable a respective data path service partition, wherein the method comprises: after the first control plane unit was placed in the second compute node: Determine, through the first control plane unit in the second compute node, that an existing instance of the first data path service partition exists in the first compute node. Sending a stop notification to the services daemon process in the first compute node by the first control plane unit in the second compute node to terminate the existing instance of the first data path service partition, Based on receiving confirmation that the existing instance of the first data path service partition has been terminated, the first control plane unit in the second compute node activates a new instance of the first data path service partition by sending a start notification to the services daemon process in the second compute node.

Images