HIGHLY AVAILABLE COMPUTING CLUSTERS

Abstract

In a sample implementation that conforms to the features disclosed here, high availability of a stateful application is achieved through the orchestration of multiple computer clusters. A stateful application is deployed to a first application operator and a second application operator of a first computing cluster and a second computing cluster, respectively. The first application operator includes a first control loop that modifies the first computing cluster based on its first resources. The second application operator has a second control loop that modifies the second computing cluster based on its second resources. Synchronization of the first resources with the second resources is configured, for example, via an object storage service. The first application operator and the second application operator scale to non-zero and zero, respectively.A load balancer is configured to forward requests to the first computing cluster.

Description

BACKGROUND

[0001] Container orchestration can be used to automate the deployment, scaling, and management of applications. A container management system can be used to perform container orchestration. A container management system can contain a set of primitives used collectively for container orchestration in a compute cluster of compute nodes. A compute cluster includes one or more manager nodes (which are part of a control plane) and one or more worker nodes (which are part of a data plane). A manager node of a compute cluster can distribute workloads among worker nodes of the compute cluster, manage the transfer of workloads between the worker nodes, scale workloads up or down, and / or similar tasks by orchestrating application containers on the worker nodes.Application containers are a form of operating system virtualization in which a container contains the minimal operating system resources, memory, and dependencies needed to run an application. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying figures. The Fig. 1A-1B are schematic diagrams of a computer cluster according to some embodiments. Fig. Figure 2 is a schematic diagram of an active-passive computing cluster system according to some implementations. Fig. 3 is a diagram of a computer cluster group deployment method according to some implementations. The Fig. 4A-4B are schematic representations of an active-passive computer cluster system during operation according to some implementations. Fig. 5 is a diagram of a method for reconfiguring computer cluster groups according to some implementations.

[0003] Corresponding numbers and symbols in the various figures generally refer to corresponding parts unless otherwise noted. The figures are intended to illustrate the relevant aspects of the disclosure and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0004] The following disclosure provides many different examples of implementing various features. To simplify the present disclosure, specific examples of components and arrangements are described below. These are, of course, only examples and are not intended to be limiting.

[0005] Container management systems are suitable for orchestrating cloud computing workloads and can therefore be deployed in public (publicly accessible) and / or private (non-publicly accessible) clouds. For example, a container management system in a company's on-premises data center can be used to orchestrate containers in a cluster of local nodes. The company can use such a container management system to orchestrate applications in a private cloud. An application orchestrated on a computing cluster can be a stateless application or a stateful application. A stateless application operates without domain-specific knowledge of the computing cluster. A stateful application operates with domain-specific knowledge of the computing cluster. A company may want to achieve high availability (HA) of a stateful application, e.g.High availability across different locations (e.g., geographic areas or countries), but the use of domain-specific knowledge by the stateful application presents a challenge to achieving high availability. Furthermore, some stateful applications are single-instance statefuls, limited to one instance per compute cluster. One technique to achieve high availability of a single-instance stateful application is to use a stretched compute cluster, that is, a compute cluster orchestrated to span multiple locations, but network constraints can cause a stretched compute cluster to exhibit poor performance.

[0006] In an example implementation consistent with the features disclosed herein, high availability of a single-instance stateful application is achieved by orchestrating multiple compute clusters at different locations. One of the compute clusters is referred to as the active compute cluster, and another of the compute clusters is referred to as the passive compute cluster. A single-instance stateful application is deployed on each of the compute clusters. The resources used by the stateful application are synchronized between the compute clusters. The stateful application on the active compute cluster scales up, while the stateful application on the passive compute cluster scales down.Specifically, the stateful application on the active computing cluster is scaled to a value greater than zero so that it is deployed on at least one worker node of the active computing cluster, while the stateful application on the passive computing cluster is scaled to zero so that it is not deployed on any worker nodes of the passive computing cluster. Thus, the stateful application runs on the active computing cluster but not on the passive computing cluster. If the active computing cluster fails (e.g., due to a failure, maintenance, etc.), the active / passive computing cluster mapping is reversed, and the stateful application is scaled up on the new active computing cluster (e.g., the former passive computing cluster).Because the resources used by the stateful application are synchronized between the compute clusters, the new active compute cluster has a copy of the resources used by the stateful application and can therefore start up with minimal delay. Accordingly, high availability of the stateful application (especially active-passive high availability) can be achieved across different locations. This can improve the operation of a private cloud.

[0007] To enable client access to the stateful application, a load balancer is deployed and configured. Specifically, the load balancer is configured to forward requests to the compute cluster currently designated as the active compute cluster. If the designation of the active compute cluster changes, the load balancer is updated to point to the currently active compute cluster.

[0008] Computing clusters can be provided by a container management system. Such systems provide a platform for automating the deployment, scaling, and operation of application containers in computing clusters of worker nodes. Resources or components of computing clusters can be created, grouped, updated, and / or deleted, and these operations can be automated. The basic planning unit of a container management system is a pod, which forms a higher level of abstraction by grouping container components. A pod comprises containers that reside on the same worker node and can share resources. A service (e.g., a computing cluster service), which represents an abstraction of the resources of a computing cluster, can be created in a computing cluster. A service comprises one or more pods that work together to provide a function.A computing cluster may also include one or more controllers. A controller (e.g., a computing cluster controller) includes a control loop that monitors the state of a computing cluster to track a computing cluster resource and automatically makes or requests changes to the computing cluster based on the resource's state. A controller may contain one or more pods that implement the control loop. A cluster may also contain one or more operators. Operators are software extensions of a cluster that use custom resources to manage applications and their components. An operator includes a controller whose control loop monitors the state of a cluster to track a custom resource of the cluster and automatically makes or requests changes to the cluster based on the custom resource's state.

[0009] The Fig. 1A-1B are schematic diagrams of a computer cluster 100 according to some embodiments. Fig. 1A is a diagram of the components of the computer cluster 100 and Fig. 1B is a diagram of a computer node of the computer cluster 100. Referring to Fig. 1A, the compute cluster 100 includes a manager node 102 responsible for managing the compute cluster 100, and a plurality of worker nodes 104 (including a first worker node 104A and a second worker node 104B) within which the components of the compute cluster 100 are adapted to perform a requested cluster operation. Examples of such requested cluster operations may include operations to create an application deployment, delete an application deployment, update an application deployment, and the like.

[0010] The manager node 102 is the entry point for management tasks of the computing cluster 100 and is responsible for controlling the worker nodes 104, which house the components of the computing cluster 100 to create cluster operation. The manager node 102 includes an API server 106, which provides both the internal and external interfaces for accessing the computing cluster 100 via the manager node 102. The API server 106 receives commands from a management interface 108. The commands can be REST (Representational State Transfer) command requests. The API server 106 processes the commands from the management interface 108, validates the commands, and executes the logic specified in the commands.

[0011] The results of the commands processed by the API server 106 may be stored in a storage component 110. The storage component 110 may be a distributed key-value store, such as an etcd store component, which may be integrated with the manager node 102. The storage component 110 stores configuration data of the compute cluster 100 that represents the state of the compute cluster 100 (e.g., which pods are present, which pods should be running, on which nodes the pods should be running, etc.). The storage component 110 stores the commands received from the API server 106 for performing CRUD (create-read-update-and-delete) operations, as well as an interface for registering observers on specific nodes, thereby providing a reliable way to inform the rest of the compute cluster 100 of configuration changes within the compute cluster 100.For example, the information in the storage component 110 enables the manager node 102 to be informed of configuration changes such as scheduled, created and deployed jobs, pod / service details and states, namespaces and replication information, etc.

[0012] The manager node 102 also includes a resource scheduler 112 and a controller manager 114. The resource scheduler 112 is capable of deploying pods (and thus applications) to the worker nodes 104. The resource scheduler 112 contains information about available resources on the computing cluster 100 as well as the resources used for the applications to be executed. This information is used by the resource scheduler 112 to make decisions about where to deploy a particular application. The controller manager 114 manages the controllers of the computing cluster 100. A controller uses the API server 106 to monitor the state of one or more resources of the computing cluster 100 and to automatically make changes to the computing cluster 100 based on the state of the resource(s).For example, a controller may use the APl server 106 to make changes to the current state of the compute cluster 100, such as changing the current state to another state, rebuilding a failed pod, removing an unscheduled pod, etc. Additionally, the manager node 102 may include a DNS server 116 that provides DNS entries for the components (e.g., pods and services) of the compute cluster 100. The node agents of the worker nodes 104 may use the DNS server 116 to resolve domain names.

[0013] Pods 118 (including first pods 118A and second pods 118B) are co-located on each of the worker nodes 104. Containers 120 (including first containers 120A and second containers 120B) are co-located in the respective pods 118. The containers 120 are co-located on the respective worker nodes 104 on which the respective pods 118 run and can share resources. A pod 118 is a group of containerized components that share resources such as storage, namespaces, control groups, IP addresses, etc. Each of the pods 118 is assigned an IP address within the computer cluster 100. A pod 118 can contain a disk, such as a local disk directory or a network disk, and can share the disk with the containers 120 within the pod 118.The pods 118 can be managed manually via the API server 106, or the management of the pods 118 can be performed automatically by a controller (managed by the controller manager 114).

[0014] Containers 120 contain the minimal operating system resources, memory, and dependencies required to run an application. Dependencies include, for example, files, environment variables, libraries, and the like. The host operating system for a worker node 104 restricts access of containers 120 to the physical resources of the worker node 104, such as CPUs, storage, memory, and the like. The worker nodes 104 may use virtualization to run containers 120.

[0015] The pods 118 contained in a worker node 104 are created, destroyed, and recreated based on the state of the compute cluster 100. Therefore, the pods 118 may not be persistent or may not exist for a long period of time. Due to the relatively short lifespan of the pods 118, the IP address under which they are deployed may change. To facilitate communication with the pods 118, even when their IP addresses change, a service may be defined for specific pods 118. A service is an abstraction of a group of pods 118, typically using a proxy. A service may be assigned a virtual IP address so that other components can communicate with the service through the virtual IP address. At least some of the pods 118 may be load balanced so that the pods 118 can be made accessible via a service.Pods 118 can be recreated, and their corresponding IP protocol can be changed without changing the service's virtual IP address. Therefore, a service can be created with a stable IP address and a DNS name that other pods 118 can use to communicate with the service. For example, consider an image processing backend running with three replicas. These replicas are fungible because a frontend doesn't care which image processing backend is used. While the pods 118 that make up the backend set can change, the frontend clients that communicate with the backend through a service cannot be aware of these changes, so the frontend clients don't keep track of the backend set's list. Each service in containers 120 can be assigned a DNS name that identifies the pods 118 in which the service resides.

[0016] Each of the worker nodes 104 includes a node agent 122 (including a first node agent 122A and a second node agent 122B). A node agent 122 communicates with the management node 102 and receives details about the configuration of the pods 118 from the API server 106. The node agent 122 uses the received details to ensure that the containers 120 are built and running as intended. In addition, the node agent 122 may also receive information about services from the storage component 110 to obtain information about services and create details about newly created services.

[0017] In addition, each of the worker nodes 104 includes a proxy 124 (including a first proxy 124A and a second proxy 124B). Each proxy 124 acts as a network proxy or hub through which requests are transmitted and as a load balancer for a service on a worker node 104 to redirect and distribute network traffic across the containers 120. The proxies 124 are used to increase the capacity and reliability of applications and to perform network routing for TCP (Transmission Control Protocol) and UDP (User Data Protocol) packets. The proxies 124 forward traffic to the appropriate container 120 to enable access to a service based on a virtual IP address of the service. The proxies 124 can also perform numbering of incoming requests, and this information can be used to establish cluster operation.In this way, the components of the worker nodes 104 can be combined and identified with each other so that when an application is deployed, the components for building and running the application are located on all worker nodes 104. If one of the worker nodes 104 is added or removed, the computer cluster 100 can build or deploy the application by combining components from different worker nodes 104 or using a combination of different components within the worker nodes 104.

[0018] To perform cluster operations in a container management system, a deployment configuration containing instructions for creating and updating components to perform a cluster operation can be entered into the management node 102 via the management interface 108. Once the management node 102 receives the instructions for creating and updating the components to perform the cluster operation, the API server 106 schedules the cluster operation on the worker nodes 104 to perform the cluster operation using a combination of several different components in several different containers 120 of several different pods 118. In this way, the cluster operation is performed using a combination of components located in several containers 120 located in one or more of the pods 118 in one or more of the worker nodes 104.

[0019] Once a cluster operation has been scheduled, the manager node 102 monitors the pods 118. If the manager node 102 detects that a resource used for the cluster operation, located in one of the containers 120 of the pods 118, fails or is deleted, the manager node 102 replaces the deleted or non-functional pod 118 associated with the cluster operation with a different combination of the currently available resources in the containers 120 of the pods 118. In this way, the API server 106 monitors the functionality of the pods 118, and if the pods 118 no longer function as intended, the pod 118 is recreated.

[0020] The management node 102 also includes one or more computing cluster operators, e.g., an application operator 126. A computing cluster operator is a software extension for the container management system that enables the behavior of the computing cluster 100 to be extended, e.g., by extending the API server 106. An operator uses custom resources to manage an application and its components. Custom resources are resources of the computing cluster 100 that are tailored to an application and defined by custom resource definitions (CRDs). The custom resources can be stored in the storage component 110. The configuration and settings for an application can be stored in a custom resource for an operator.An operator comprises an application-specific controller (and one or more associated pods 118) managed by the controller manager 114. An operator's controller is associated with application-specific custom resources. Specifically, an operator's controller includes a control loop that tracks the application-specific custom resources and performs actions based on those resources. The application operator 126 is an operator for an operator-based application. An operator-based application is a stateful, single-instance application that operates on custom resources. Therefore, the operator-based application can be directly controlled through the management interface 108 to interact with the custom resources.In an example implementation, the compute cluster 100 is a Kubernetes® cluster, the application operator 126 is a Kubernetes® operator, and the operator-based application is an application for monitoring the compute cluster 100, such as Prometheus.

[0021] The manager node 102 also includes one or more computing cluster controllers, such as a cluster group controller 128, managed by the controller manager 114. A computing cluster controller includes a control loop that tracks a resource of the computing cluster 100 and automatically makes or requests changes to the computing cluster 100 based on the resource's state. As described in more detail below, multiple computing clusters 100 are used to achieve high availability for the operator-based application of the application operator 126. The cluster group controllers 128 of the computing clusters 100 are used to coordinate the computing clusters 100 such that the operator-based application is deployed on each of the computing clusters 100, but only one instance of the operator-based application is running across the computing clusters 100 at a given time.The operation of the cluster group controller 128 is described in more detail below.

[0022] The nodes of the computing cluster 100 (e.g., the manager node 102 and the worker nodes 104) may each include suitable components. Suitable components include a processor, an application-specific integrated circuit, a microcontroller, memory, and the like. The nodes of the computing cluster 100 may each be physical devices, e.g., computers. For example, the manager node 102 and the worker nodes 104, as shown in Fig. 1B, each include a processor 132 and a memory 134. Memory 134 may be a non-transitory computer-readable medium that stores instructions for execution by processor 132. One or more modules within the nodes of computer cluster 100 may be implemented partially or entirely in software and / or hardware to perform the functions described herein.

[0023] Fig. 2 is a schematic diagram of an active-passive computing cluster system 200 according to some implementations. In the active-passive computing cluster system 200, multiple computing clusters 100 (including a first computing cluster 100A and a second computing cluster 100B) are grouped and used to achieve high availability of a stateful, single-instance application, such as an operator-based application. The first computing cluster 100A may be located at a first location (e.g., location A) and the second computing cluster 100B may be located at a second location (e.g., location B) that is different from the first location. In particular, the first location may be in a different geographic area than the second location, such that the first computing cluster 100A and the second computing cluster 100B are located in different geographic areas.

[0024] The active-passive computing cluster system 200 includes a cluster group manager 202 and an object storage service 204. The cluster group manager 202 may be a computer cluster controller deployed on a management computer cluster 100C. The cluster group manager 202 controls the computer clusters 100 that have been registered as group members, so that they are grouped into a high-availability cluster group. The cluster group controllers 128 (including a first cluster group controller 128A and a second cluster group controller 128B) of the respective computer clusters 100 process deployment commands from the cluster group manager 202.The cluster group manager 202 sends the deployment commands to the cluster group controllers 128 to coordinate the computer clusters 100 such that an operator-based application is deployed on each of the computer clusters 100, but only one instance of the operator-based application is running on the computer clusters 100 at a given time. The storage components 110 (including a first storage component 110A and a second storage component 110B) of the respective computer clusters 100 store custom resources defined for the operator-based application. The object storage service 204 is used to facilitate the synchronization of the custom resources for the operator-based application between the storage components 110. The object storage service 204 may be a cloud object storage service such as an Amazon Simple Storage Service (S3) bucket or the like.In an example implementation, the storage components 110 are not synchronized via the object storage service 204, but only the custom resources defined for the operator-based application.

[0025] First, the operator-based application, along with its user-defined resource definitions (CRDs), is deployed to each of the computer clusters 100. Thus, the computer clusters 100 may include application operators 126 (including a first application operator 126A and a second application operator 126B) on which the operator-based application is deployed. The cluster group manager 202 designates one site as the active site and the other sites as passive sites. In this example, the active-passive computer cluster system 200 includes one passive site, but multiple passive sites may be used. The selection of the active / passive sites may be performed via established rules or by manual enforcement (e.g., by an administrator of the active-passive computer cluster system 200 via a management interface for the management computer cluster 100C).The cluster group controller 128 at the passive site then shuts down the application operator 126 of the respective computing cluster 100 to zero. Specifically, the application operators 126 include pods, and the number of pods for the application operator 126 at the passive site is set to zero. Furthermore, the cluster group controller 128 at the active site scales up the application operator 126 of the respective computing cluster 100 to a value greater than zero. Specifically, the number of pods for the application operator 126 at the active site is set to a value greater than zero, for example, to one. As a result, only one instance of the operator-based application is running in the computing clusters 100 at any given time.

[0026] The application operators 126 include control loops that modify the computer clusters 100 based on the operator-based application's custom resources stored in the storage components 110. Synchronization of the custom resources between the computer clusters 100 is performed via the object storage service 204. The cluster group controllers 128 synchronize the custom resources of the respective storage components 110 with the object storage service 204, and vice versa. Consequently, custom resources written to the storage component 110 of one computer cluster are synchronized with the storage component 110 of the other computer cluster.During operation, the cluster group controller 128 in the active site writes to its storage component 110, while the cluster group controller 128 in the passive site does not write to its storage component 110 because that cluster group controller 128 is scaled to zero (e.g., not running). As a result, the custom resources at the passive site are non-functional copies of the custom resources at the active site.

[0027] The active-passive computing cluster system 200 also includes a load balancer 206. The load balancer 206 may be a computing cluster service deployed on the managing computing cluster 100C. A computing cluster service includes one or more pods 118 (see Fig. 1) that cooperate to provide a function, such as load balancing in the case of load balancer 206. Load balancer 206 is configured to forward requests from clients 208 to the active site computing cluster 100.

[0028] Requests that interact with the custom resources at the active site result in the data for the custom resources being synchronized with the passive site. If the active site fails (e.g., due to a malfunction, maintenance, etc.), the cluster group manager 202 designates the former passive site as the new active site, thus reversing the roles of the former active / passive sites. The cluster group controller 128 at the new active site increases the application operator 126 running in the respective compute cluster 100 to greater than zero, while the cluster group controller 128 at the former active site decreases the application operator 126 running in the respective compute cluster 100 to zero. The load balancer 206 is reconfigured to forward requests from the clients 208 to the compute cluster 100 of the new active site.In this way, high availability of the operator-based application can be achieved at the application layer. The availability of the operator-based application can thus be improved across sites without requiring any changes to the site infrastructure.

[0029] Fig. 3 is a diagram of a method for provisioning computer cluster groups (300) according to some implementations. The method for establishing computer cluster groups 300 is used in conjunction with Fig. 2. The method 300 for provisioning computer cluster groups may be performed by the cluster group manager 202 when setting up the first computer cluster 100A and the second computer cluster 100B for high availability.

[0030] In step 302, a stateful application is deployed to a first application operator 126A and a second application operator 126B of a first computing cluster 100A and a second computing cluster 100B, respectively. The first application operator 126A includes a first control loop that modifies the first computing cluster 100A based on first user-defined resources of the first computing cluster 100A. The second application operator 126B includes a second control loop that modifies the second computing cluster 100B based on the second user-defined resources of the second computing cluster 100B. The stateful application may be an operator-based application.The cluster group manager 202 may deploy the stateful application by sending deployment commands instructing the first cluster group controller 128A and the second cluster group controller 128B to deploy a copy of the stateful application to the first application operator 126A and the second application operator 126B, respectively.

[0031] In step 304, the synchronization of the first custom resources with the second custom resources is configured. The first custom resources are stored in a first storage component 110A of the first computing cluster 100A. The second custom resources are stored in a second storage component 110B of the second computing cluster 100B. The first storage component 110A and the second storage component 110B may each be configured to synchronize with each other via an object storage service 204. In an example implementation where the first storage component 110A is a first key-value store and the second storage component 110B is a second key-value store, the key-value stores are synchronized with each other.The cluster group manager 202 may configure the synchronization of the custom resources by sending provisioning commands instructing the first cluster group controller 128A and the second cluster group controller 128B to begin synchronizing the custom resources stored in the first computer cluster 100A and the second storage component 110B, respectively, with the object storage service 204.

[0032] In step 306, the first application operator 126A and the second application operator 126B are scaled to non-zero and zero, respectively. The first control loop of the first application operator 126A may be executed by a first manager node of the first compute cluster 100A, while the first application operator 126A may include first pods deployed on first worker nodes of the first compute cluster 100A. Scaling the first application operator 126A to a non-zero value may include setting a first set of the first pods to a value greater than zero, for example, one. Similarly, the second control loop of the second application operator 126B may be executed by a second manager node of the second compute cluster 100B, while the second application operator 126B may include second pods deployed on second worker nodes of the second compute cluster 100B.Scaling the second application operator 126B to zero may involve setting a second set of the second pods to zero. Thus, the first set of the first pods is larger than the second set of the second pods. The cluster group manager 202 may scale the first application operator 126A and the second application operator 126B by sending provisioning commands that command the first cluster group controller 128A to increase the size of the first application operator 126A and the second cluster group controller 128B to decrease the size of the second application operator 126B.

[0033] In step 308, a load balancer 206 is configured to forward requests to the first computing cluster 100A. At this time, the first computing cluster 100A is configured as an active computing cluster and the second computing cluster 100B is configured as a passive computing cluster. Thus, requests from clients 208 are forwarded to the active computing cluster. The cluster group manager 202 can configure the load balancer 206 by sending provisioning commands that instruct the load balancer 206 to change the load balancing weights so that the active computing cluster is given more weight in load balancing decisions than the passive computing cluster. In one example, the load balancing weights are set to zero for the passive computing cluster and to a non-zero value for the active computing cluster.

[0034] Additional steps may be performed. In an example implementation, the cluster group manager 202 deploys the first cluster group controller 128A and the second cluster group controller 128B on the first compute cluster 100A and the second compute cluster 100B, respectively, before performing other steps. For example, if the first cluster group controller 128A and the second cluster group controller 128B are compute cluster controllers, they may be orchestrated on the first compute cluster 100A and the second compute cluster 100B via the respective API servers for the compute clusters.

[0035] The method 300 for provisioning computer cluster groups is initially performed during the provisioning of the active-passive computer cluster system 200. Upon completion, the active-passive computing cluster system 200 provides high availability of the operator-based application deployed at the first application operator 126A and the second application operator 126B. During operation, the current active computing cluster may fail (e.g., due to a malfunction, maintenance, etc.). If the current active computing cluster fails, the cluster group manager 202 designates a new computing cluster as the active computing cluster and a new computing cluster as the passive computing cluster. The roles of the previous active / passive computing clusters are thus reversed.

[0036] The Fig. 4A-4B are schematic representations of the active-passive computer cluster system 200 during operation according to some implementations. An application operator 126 for a stateful application is deployed on each of the computer clusters 100.

[0037] In a first active-passive configuration, which Fig. As illustrated in Figure 4A, the first compute cluster 100A is referred to as the active compute cluster, while the second compute cluster 100B is referred to as the passive compute cluster. The first application operator 126A running on the first compute cluster 100A is scaled to a value greater than zero, while the second application operator 126B running on the second compute cluster 100B is scaled to a value of zero. The first application operator 126A interacts with user-defined resources stored in the first storage component 110A, and changes to these user-defined resources are synchronized with the second storage component 110B. The load balancer 206 is configured to forward requests to the first compute cluster 100A.

[0038] In a second active-passive configuration, which Fig. As illustrated in Figure 4B, the first compute cluster 100A is referred to as a passive compute cluster, while the second compute cluster 100B is referred to as an active compute cluster. The first application operator 126A running on the first compute cluster 100A is scaled to zero, while the second application operator 126B running on the second compute cluster 100B is scaled to greater than zero. The second application operator 126B interacts with user-defined resources stored in the second storage component 110B, and changes to these user-defined resources are synchronized with the first storage component 110A. The load balancer 206 is configured to forward requests to the second compute cluster 100B.

[0039] As previously mentioned, the custom resources for the operator-based application are synchronized between the first storage component 110A and the second storage component 110B. When the active / passive computing cluster designation is changed, the application operator 126 of the new active computing cluster maintains a copy of the custom resources used by the operator-based application. This allows the operator-based application to execute with minimal latency on the computing cluster to which it is scaled up. Furthermore, the state of the operator-based application is preserved when the active / passive data center designation is changed.

[0040] Fig. 5 is a diagram of a computer cluster group reconfiguration method 500 according to some implementations. The computer cluster group reconfiguration method 500 is used in conjunction with the Fig. 4A-4B. The method 500 for reconfiguring computer cluster groups may be performed by the cluster group manager 202 during failover, e.g., when reconfiguring the first computer cluster 100A and the second computer cluster 100B for high availability.

[0041] In step 502, a first computing cluster 100A is designated as the first active computing cluster and a second computing cluster 100B is designated as the first passive computing cluster. A stateful application is deployed to a first application operator 126A and a second application operator 126B of the first computing cluster 100A and the second computing cluster 100B, respectively. The stateful application may be an operator-based application. The first application operator 126A of the first active computing cluster and the second application operator 126B of the first passive computing cluster are scaled to a non-zero value and zero, respectively. Additionally, a load balancer 206 is configured to forward requests to the first active computing cluster (e.g., the first computing cluster 100A). Thus, the active-passive computing cluster system 200 is set to a first active-passive configuration (see Fig. 4A).

[0042] In step 504, a failure of the first active compute cluster (e.g., the first compute cluster 100A) is detected by the cluster group manager 202. The first active compute cluster may fail due to a malfunction, maintenance, etc. The cluster group manager 202 may detect the failure by receiving a failure notification from the first cluster group controller 128A, by detecting a loss of connectivity to the first cluster group controller 128A, etc. In an example implementation, the cluster group manager 202 detects the failure based on information from the container management system, such as information stored in the storage components 110 of the compute clusters. In particular, user-defined resources of the compute clusters may be considered when detecting the failure.The information in the storage components 110 can be used to detect a lack of availability of an application due to a lack of resources or a failure.

[0043] In step 506, in response to the detection of the failure, the first compute cluster 100A is designated as the second passive compute cluster and the second compute cluster 100B is designated as the second active compute cluster. The first application operator 126A of the second passive compute cluster and the second application operator 126B of the second active compute cluster are scaled to zero and non-zero, respectively. Furthermore, the load balancer 206 is configured to forward requests to the second active compute cluster (e.g., the second compute cluster 100B). Thus, the active-passive compute cluster system 200 is set to a second active-passive configuration (see Fig.4B), which differs from the first active-passive configuration. At this time, the designation of the active / passive computing clusters is reversed. Thus, requests from clients 208 are forwarded to the second active computing cluster (formerly the first passive computing cluster) instead of the first active computing cluster (now the second passive computing cluster). When the designation of the active / passive computing clusters is reversed, the second active computing cluster has a copy of the custom resources used by the operator-based application and can therefore be brought into operation with little delay. The operator-based application thus starts on the second active computing cluster in the same state in which it previously ran on the first active computing cluster.

[0044] Several variations are conceivable. In one example implementation, the operator-based application also uses data stored on a persistent disk (e.g., the pods 118) to maintain state. In such an implementation, the persistent disk may be migrated from the previous active compute cluster to the new active compute cluster during a failover. The migration may be performed using a volume migration method instead of volume synchronization by the object storage service 204. For example, the persistent volume may be copied from the respective pods of the first compute cluster 100A to the respective pods of the second compute cluster 100B. Migrating disks may increase the resiliency delay but may be more efficient than disk synchronization due to network constraints.In another example implementation, multiple computer clusters can be grouped and the groups can be considered a single unit for active / passive determination. An active group of multiple active computer clusters can run multiple operator-based applications. If a failure is detected in one of the applications in the active group, the compute clusters in the active group can be demoted and the corresponding compute clusters in a previous passive group can be promoted. The load balancer 206 can then be reconfigured to use the operator-based applications in the new active group to service customer requests.

[0045] In an example implementation consistent with the features disclosed herein, a method comprises: deploying a stateful application to a first application operator and a second application operator of a first compute cluster and a second compute cluster, respectively, the first application operator including a first control loop that modifies the first compute cluster based on first resources of the first compute cluster, and the second application operator including a second control loop that modifies the second compute cluster based on second resources of the second compute cluster; configuring synchronization of the first resources with the second resources; scaling the first application operator and the second application operator to a non-zero value and zero, respectively; and configuring a load balancer to direct requests to the first compute cluster.In another example implementation, the method further comprises: scaling the first application operator and the second application operator to zero and non-zero, respectively; and reconfiguring the load balancer to direct requests to the second compute cluster. In another example implementation of the method: the first compute cluster includes a first manager node and first worker nodes, the first manager node executing the first control loop and the first application operator further including first pods on the first worker nodes; and the second compute cluster includes a second manager node and second worker nodes, the second manager node executing the second control loop and the second application operator further including second pods on the second worker nodes.In another example implementation of the method, scaling the first application operator and the second application operator comprises: setting a first set of the first pods to greater than zero; and setting a second set of the second pods to zero. In another example implementation of the method, the first resources are stored in a first key-value store of the first compute cluster, the second resources are stored in a second key-value store of the second compute cluster, and configuring synchronization of the first resources and the second resources comprises: configuring synchronization of the first key-value store with the second key-value store. In another example implementation of the method, the first key-value store and the second key-value store are synchronized via a cloud object storage service.In another example implementation of the method, deploying the stateful application comprises: deploying custom resource definitions for the first resources and the second resources on the first computer cluster and the second computer cluster, respectively. In another example implementation, the method further comprises: deploying a first cluster group controller and a second cluster group controller on the first computer cluster and the second computer cluster, respectively, wherein scaling the first application operator and the second application operator comprises sending deployment commands that command the first cluster group controller to scale up the first application operator and command the second cluster group controller to scale down the second application operator.In another example implementation of the method, the stateful application is a single-instance stateful application.

[0046] In an example implementation consistent with the features disclosed herein, a system includes: a plurality of compute clusters, each of the compute clusters including resources; a cluster group manager configured to: deploy a stateful application on each respective compute cluster of the compute clusters, the stateful application including a control loop that modifies the respective compute cluster based on the resources of the respective compute cluster; designate one of the compute clusters as an active compute cluster and another of the compute clusters as a passive compute cluster; scale the stateful application on the active compute cluster to greater than zero; and scale the stateful application on the passive compute cluster to zero.In another example implementation of the system, the cluster group manager is further configured to: configure a load balancer to direct requests to the active compute cluster. In another example implementation of the system, the cluster group manager is further configured to: configure the synchronization of the resources of the active compute cluster and the passive compute cluster. In another example implementation of the system, the resources of the active compute cluster and the passive compute cluster are synchronized through an object storage service. In another example implementation of the system, the cluster group manager is further configured to: reverse the naming of the active compute cluster and the passive compute cluster in response to detecting a failure of the active compute cluster.In another example implementation of the system, the failure of the active compute cluster is detected based on the resources of the active compute cluster.

[0047] In an example implementation consistent with the features disclosed herein, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to: provide a stateful application to a first application operator and a second application operator of a first compute cluster, respectively.a second compute cluster, wherein the first application operator includes a first control loop that modifies the first compute cluster based on first resources of the first compute cluster, wherein the second application operator includes a second control loop that modifies the second compute cluster based on second resources of the second compute cluster, wherein the first compute cluster and the second compute cluster are located in different geographic areas; configure synchronization of the first resources and the second resources via an object storage service; scale the first application operator and the second application operator to a non-zero value and zero, respectively; and configure a load balancer to direct requests to the first compute cluster.In another example implementation, the non-transitory computer-readable medium further stores instructions that, when executed by the processor, cause the processor to: scale the first application operator and the second application operator to zero and non-zero, respectively, in response to detecting a failure of the first compute cluster; and reconfigure the load balancer to direct requests to the second compute cluster.In another example implementation of the non-transitory computer-readable medium: the first compute cluster includes a first manager node and first worker nodes, the first manager node executing the first control loop, the first application operator further including first pods on the first worker nodes; the second compute cluster includes a second manager node and second worker nodes, the second manager node executing the second control loop, the second application operator further including second pods on the second worker nodes; and the instructions to scale the first application operator and the second application operator include instructions to: set a first set of the first pods to one; and set a second set of the second pods to zero.In another example implementation of the non-transitory computer-readable medium, the instructions for deploying the stateful application include instructions for: deploying custom resource definitions for the first resources and the second resources on the first computer cluster and the second computer cluster. In another example implementation of the non-transitory computer-readable medium, the stateful application is a single-instance stateful application.

[0048] The foregoing outlines features of several examples to enable those skilled in the art to better understand aspects of the present disclosure. Various modifications and combinations of the illustrative examples, as well as other examples, will be apparent to those skilled in the art from the description. Therefore, the appended claims are intended to encompass all such modifications.

Claims

[1] A process comprising: Providing a stateful application at a first application operator and a second application operator of a first compute cluster and a second compute cluster, respectively, wherein the first application operator comprises a first control loop that changes the first compute cluster based on first resources of the first compute cluster, and the second application operator comprises a second control loop that changes the second compute cluster based on second resources of the second compute cluster; Configuring the synchronization of the first resources with the second resources; Scaling the first application operator and the second application operator to a non-zero value and zero, respectively; and Configure a load balancer to forward requests to the first computer cluster. [2] The method of claim 1, further comprising: Scaling the first application operator and the second application operator to zero and non-zero, respectively; and Reconfigure the load balancer to direct requests to the second computer cluster. [3] The method of claim 1, wherein: the first compute cluster comprises a first manager node and first worker nodes, wherein the first manager node executes the first control loop and the first application operator further comprises first pods on the first worker nodes; and the second compute cluster comprises a second manager node and second worker nodes, wherein the second manager node executes the second control loop and the second application operator further comprises second pods on the second worker nodes. [4] The method of claim 3, wherein scaling the first application operator and the second application operator comprises: Setting a first quantity of the first sleeves to a value greater than zero; and Setting a second quantity of the second sleeves to zero. [5] The method of claim 1, wherein the first resources are stored in a first key-value store of the first computing cluster, the second resources are stored in a second key-value store of the second computing cluster, and configuring the synchronization of the first resources and the second resources comprises: Configure the synchronization of the first key-value store with the second key-value store. [6] The method of claim 5, wherein the first key-value store and the second key-value store are synchronized via a cloud object storage service. [7] The method of claim 1, wherein providing the stateful application comprises: Deploy custom resource definitions for the first resources and the second resources on the first compute cluster and the second compute cluster, respectively. [8] The method of claim 1, further comprising: Providing a first cluster group controller and a second cluster group controller on the first computing cluster and the second computing cluster, respectively, wherein scaling the first application operator and the second application operator comprises sending provisioning commands that command the first cluster group controller to scale up the first application operator and command the second cluster group controller to scale down the second application operator. [9] The method of claim 1, wherein the stateful application is a single-instance stateful application. [10] A system consisting of: a plurality of computer clusters, each of the computer clusters comprising resources; a cluster group manager configured to: Deploying a stateful application on each respective computer cluster of the computer clusters, the stateful application comprising a control loop that modifies the respective computer cluster based on the resources of the respective computer cluster; designate one of the computer clusters as the active computer cluster and another of the computer clusters as the passive computer cluster; scale the stateful application on the active compute cluster to a value greater than zero; and to scale the stateful application to zero on the passive computing cluster. [11] The system of claim 10, wherein the cluster group manager is further configured to: Configure a load balancer to forward requests to the active computer cluster. [12] The system of claim 10, wherein the cluster group manager is further configured to: Configure the synchronization of resources between the active compute cluster and the passive compute cluster. [13] The system of claim 12, wherein the resources of the active computing cluster and the passive computing cluster are synchronized via an object storage service. [14] The system of claim 10, wherein the cluster group manager is further configured to: Reversing the determination of the active compute cluster and the passive compute cluster in response to the detection of a failure of the active compute cluster. [15] The system of claim 14, wherein the failure of the active compute cluster is detected based on the resources of the active compute cluster. [16] A non-transitory computer-readable medium that stores instructions that, when executed by a processor, cause the processor to: provide a stateful application at a first application operator and a second application operator of a first compute cluster and a second compute cluster, respectively, wherein the first application operator comprises a first control loop that modifies the first compute cluster based on first resources of the first compute cluster, the second application operator comprises a second control loop that modifies the second compute cluster based on second resources of the second compute cluster, wherein the first compute cluster and the second compute cluster are located in different geographical areas; configure the synchronization of the first resources and the second resources via an object storage service; Scaling the first application operator and the second application operator to a non-zero value and zero, respectively; and Configure a load balancer to forward requests to the first computer cluster. [17] The non-transitory computer-readable medium of claim 16, further storing instructions that, when executed by the processor, cause the processor to: Scaling the first application operator and the second application operator to zero and non-zero, respectively, in response to detecting a failure of the first compute cluster; and reconfigure the load balancer to forward requests to the second computer cluster. [18] The non-transitory computer-readable medium of claim 16, wherein: the first compute cluster comprises a first manager node and first worker nodes, wherein the first manager node executes the first control loop and the first application operator further comprises first pods on the first worker nodes; the second compute cluster comprises a second manager node and second worker nodes, wherein the second manager node executes the second control loop and the second application operator further comprises second pods on the second worker nodes; and the scaling instructions of the first application operator and the second application operator include instructions to: place a first quantity of the first pods on one; and set a second amount of the second pods to zero. [19] The non-transitory computer-readable medium of claim 16, wherein the instructions for providing the stateful application comprise instructions to: custom resource definitions for the first resources and the second resources on the first computer cluster and the second computer cluster provide. [20] The non-transitory computer-readable medium of claim 16, wherein the stateful application is a single-instance stateful application.

Images