High availability computing clusters

Abstract

The embodiment of the invention relates to a high-availability computing cluster. In example implementations according to features disclosed herein, high availability of stateful applications is achieved by orchestrating a plurality of computing clusters. A stateful application is deployed at a first application operator and a second application operator of the first computing cluster and the second computing cluster, respectively. The first application operator includes a first control loop that changes the first computing cluster based on a first resource of the first computing cluster. The second application operator includes a second control loop that changes the second computing cluster based on a second resource of the second computing cluster. Synchronization of the first resource with the second resource is configured, such as via an object storage service. The first application operator and the second application operator are scaled to non-zero and zero, respectively. The load balancer is configured to route the request to the first computing cluster.

Description

Background Art

[0001] Container orchestration can be used to automate the deployment, scaling, and management of applications. Container management systems can be used to perform container orchestration. Container management systems can include a set of primitives that are collectively used to orchestrate containers across a computing cluster of computing nodes. A computing cluster includes one or more manager nodes (as part of the control plane) and one or more worker nodes (as part of the data plane). The manager node of a computing cluster can distribute workloads to worker nodes of the computing cluster, manage the transfer of workloads between worker nodes, expand or reduce workloads through application containers orchestrated on worker nodes, and the like. Application containers are a form of operating system virtualization in which containers include minimal operating system resources, memory, and dependencies for running applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures.

[0003] Figure 1A-1B is a schematic diagram of a computing cluster according to some implementations.

[0004] Figure 2 is a schematic diagram of an active-passive computing cluster system according to some implementations.

[0005] Figure 3 is a diagram of a computing cluster group deployment method according to some implementations.

[0006] Figure 4A-4B is a schematic diagram of an active-passive computing cluster system during operation according to some implementations.

[0007] Figure 5 is a diagram of a computing cluster group reconfiguration method according to some implementations.

[0008] Corresponding reference numerals in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the disclosure and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0009] The following disclosure provides many different examples for implementing different features. Specific examples of components and arrangements are described below to simplify the disclosure. Of course, these are only examples and are not intended to be limiting.

[0010] The container management system is suitable for orchestrating cloud computing workloads and can therefore be used in a public cloud (which may be publicly accessible) and / or a private cloud (which may not be publicly accessible). For example, a container management system can be used at a locally deployed data center for an enterprise to orchestrate containers across a cluster of on-site nodes. An enterprise can use such a container management system to orchestrate applications on a private cloud. Applications orchestrated on a computing cluster can be stateless applications or stateful applications. Stateless applications operate without using domain-specific knowledge of the computing cluster. Stateful applications operate using domain-specific knowledge of the computing cluster. An enterprise may desire to achieve high availability (HA) for stateful applications, such as high availability across different sites (e.g., geographic regions or countries), but the use of domain-specific knowledge by stateful applications poses a challenge to achieving high availability. Additionally, some stateful applications are single-instance stateful applications that are limited to one instance per computing cluster. One technique for obtaining high availability of single-instance stateful applications is to use an extended computing cluster, which is a computing cluster orchestrated across multiple sites, but networking limitations may cause the extended computing cluster to have poor performance.

[0011] In an example implementation according to the features disclosed herein, high availability of a single-instance stateful application is achieved by orchestrating multiple computing clusters at different sites. One of the computing clusters is designated as an active computing cluster, and another computing cluster in the computing cluster is designated as a passive computing cluster. A single-instance stateful application is deployed on each computing cluster in the computing cluster. Resources used by the stateful application are synchronized between the computing clusters. The stateful application on the active computing cluster is scaled up, while the stateful application on the passive computing cluster is scaled down. Specifically, the stateful application on the active computing cluster is scaled to be greater than zero so that it is deployed on at least one worker node of the active computing cluster, while the stateful application on the passive computing cluster is scaled to zero so that it is not deployed to the worker node of the passive computing cluster. Therefore, the stateful application runs on the active computing cluster, but not on the passive computing cluster. If the active computing cluster fails (e.g., due to failure, maintenance, etc.), the designation of the active / passive computing cluster is reversed, and the stateful application on the new active computing cluster (e.g., the previous passive computing cluster) is scaled up. Because the resources used by the stateful application are synchronized between the computing clusters, the new active computing cluster will have a copy of the resources used by the stateful application and can therefore start running with low latency. Therefore, high availability (specifically, active-passive high availability) of stateful applications can be achieved across different sites. The operation of the private cloud can therefore be improved.

[0012] In order to enable clients to access stateful applications, a load balancer is deployed and configured. Specifically, the load balancer is configured to route requests to any one of the computing clusters that is currently designated as the active computing cluster. When the designation of the active computing cluster is changed, the load balancer is updated to point to the current active computing cluster.

[0013] A computing cluster can be provided by a container management system. Such a system provides a platform for automatically deploying, scaling, and operating application containers across a computing cluster of worker nodes. Computing cluster resources or components can be created, grouped, updated, and / or deleted, and these operations can be automated. The basic scheduling unit of a container management system is a pod, which adds a higher level of abstraction by grouping containerized components. A container group includes containers that are co-located on the same worker node and can share resources. An abstract service (e.g., a computing cluster service) that is a resource of a computing cluster can be created at a computing cluster. A service includes one or more container groups that work together to provide functionality. A computing cluster may also include one or more controllers. A controller (e.g., a computing cluster controller) includes a control loop that monitors the state of a computing cluster to track the resources of the computing cluster, and based on the state of the resources, automatically makes changes or requests changes to the computing cluster. A controller may include one or more container groups that implement a control loop. A cluster may also include one or more operators. An operator is a software extension of a cluster that uses custom resources to manage applications and their components. The operator includes a controller, wherein a control loop of the controller monitors the state of the cluster to track the customized resources of the cluster, and automatically makes changes or requests changes to the cluster based on the state of the customized resources.

[0014] Figure 1A-1B is a schematic diagram of a computing cluster 100 according to some implementations. Figure 1A is a diagram of the components of computing cluster 100 and Figure 1B is a diagram of computing nodes of computing cluster 100. Figure 1A , the computing cluster 100 includes a manager node 102 responsible for managing the computing cluster 100, and a plurality of worker nodes 104 (including a first worker node 104A and a second worker node 104B), and the components of the computing cluster 100 within the plurality of worker nodes 104 are adapted to perform the requested cluster operation. Examples of such requested cluster operations may include operations for creating an application deployment, deleting an application deployment, updating an application deployment, and the like.

[0015] The manager node 102 is the entry point for management tasks of the computing cluster 100 and is responsible for orchestrating the worker nodes 104, and the components of the computing cluster 100 for generating cluster operations are located within the worker nodes 104. The manager node 102 includes an API server 106, which provides both internal and external interfaces for accessing the computing cluster 100 via the manager node 102. The API server 106 receives commands from the management interface 108. The command can be a representative state transfer (REST) ​​command request. The API server 106 processes the command from the management interface 108, verifies the command, and executes the logic specified by the command.

[0016] The results of commands processed by the API server 106 may be stored in a storage component 110. The storage component 110 may be a distributed key-value storage library, such as an etcd storage component that may be included within the manager node 102. The storage component 110 stores configuration data for the computing cluster 100, which represents the state of the computing cluster 100 (e.g., what container groups exist, what container groups should be running, on which nodes the container groups should be running, etc.). The storage component 110 provides storage for commands received by the API server 106 to perform create-read-update-delete (CRUD) operations, as well as for interfaces to register monitors on specific nodes, thereby providing a reliable way to notify the rest of the computing cluster 100 of configuration changes within the computing cluster 100. For example, the information in the storage component 110 enables the manager node 102 to be notified of configuration changes, such as jobs being scheduled, created, and deployed; container group / service details and status; namespace and replication information, etc.

[0017] The manager node 102 also includes a resource scheduler 112 and a controller manager 114. The resource scheduler 112 is adapted to deploy container groups (and thus applications) onto the worker nodes 104. The resource scheduler 112 includes information about available resources on the computing cluster 100 and resources for running applications. The resource scheduler 112 uses this information to decide where to deploy a particular application. The controller manager 114 manages the controllers of the computing cluster 100. The controller uses the API server 106 to monitor the state of one or more resources of the computing cluster 100 and automatically makes changes to the computing cluster 100 based on the state of the (multiple) resources. For example, the controller can use the API server 106 to make changes to the current state of the computing cluster 100 to change the current state to another state, recreate a failed container group, remove an additional scheduled container group, etc. Additionally, the manager node 102 may include a DNS server 116 that serves DNS records for components (e.g., container groups and services) of the computing cluster 100. The node agent of the worker node 104 may use the DNS server 116 to resolve domain names.

[0018] The container group 118 (including the first container group 118A and the second container group 118B) is co-located in each worker node in the worker node 104. The container 120 (including the first container 120A and the second container 120B) resides in a corresponding container group in the container group 118. The container 120 is co-located on the corresponding worker node running the corresponding container group 118 in the worker node 104, and can share resources. The container group 118 is a group of containerized components that share resources such as storage, namespaces, control groups, IP addresses, etc. Each container group in the container group 118 is assigned an IP address within the computing cluster 100. The container group 118 may include volumes, such as local disk directories or network disks, and the volumes may be exposed to the containers 120 within the container group 118. The container group 118 may be manually managed by the API server 106, or the management of the container group 118 may be automatically performed by the controller (managed by the controller manager 114).

[0019] The container 120 includes minimal operating system resources, memory, and dependencies for running applications. Examples of dependencies include files, environment variables, libraries, etc. The host operating system for the worker node 104 constrains the container 120's access to the worker node's 104 physical resources (such as CPU, storage, memory, etc.). The worker node 104 can use virtualization to run the container 120.

[0020] The container groups 118 included in the worker nodes 104 are created, destroyed, and recreated based on the state of the computing cluster 100. Therefore, the container groups 118 may not be persistent or exist for a long time. Due to the relatively short life of the container groups 118, the IP addresses they serve may change. In order to facilitate communication with the container groups 118, services can be defined for certain container groups 118 even when their IP addresses change. A service is an abstraction of a group of container groups 118 that usually uses a proxy. A virtual IP address can be assigned to a service so that other components communicate with the service via the virtual IP address. Load balancing can be established for at least some of the container groups 118 so that the container groups 118 can be exposed via services. The container group 118 can be recreated and its corresponding IP protocol can be changed without changing the virtual IP address of the service. Therefore, a service can be created with a stable IP address and DNS name, which can be used by other container groups 118 to communicate with the service. For example, consider an image processing backend that runs with three copies. These copies are replaceable because the front end does not care which image processing backend is used. Although the container groups 118 that make up the backend set may change, the front-end client may not be aware of these changes by communicating with the backend via the service, so that the front-end client does not track the list of backend sets. Each service of the container 120 can be assigned a DNS name that identifies the container group 118 that hosts the service.

[0021] Each of the worker nodes 104 includes a node agent 122 (including a first node agent 122A and a second node agent 122B). The node agent 122 communicates with the manager node 102 and receives details of the configuration for the container group 118 from the API server 106. The node agent 122 uses the received details to ensure that the container 120 is constructed and runs as expected. Additionally, the node agent 122 can also receive information about the service from the storage component 110 to obtain information about the service and create details about the newly created service.

[0022] Additionally, each worker node in the worker node 104 includes an agent 124 (including a first agent 124A and a second agent 124B). Each agent 124 serves as a network agent or hub through which requests are transmitted, and serves as a load balancer for services on the worker node 104 to reverse the agent and distribute network traffic across the container 120. The agent 124 is used to increase the capacity and reliability of the application and perform network routing for transmission control protocol (TCP) packets and user data protocol (UDP) packets. The agent 124 routes traffic to the appropriate container 120 so that the service can be accessed based on the virtual IP address of the service. The agent 124 can also perform numbering of incoming requests, and this information can be used to create cluster operations. In this way, the components of the worker node 104 can be grouped together and identified so that when the application is to be deployed, the components for creating and running the application are located in the entire worker node 104. If any of the worker nodes 104 are added or removed, the computing cluster 100 is able to create or deploy an application by combining components from different worker nodes 104 or using a combination of different components within the worker nodes 104 .

[0023] In order to perform cluster operations in the container management system, a deployment configuration providing instructions on how to create and update components for performing cluster operations can be input to the manager node 102 via the management interface 108. Once the instructions on how to create and update components for performing cluster operations are received by the manager node 102, the API server 106 dispatches the cluster operations to the worker nodes 104 to perform the cluster operations using a combination of multiple different components within multiple different containers 120 of multiple different container groups 118. In this way, the cluster operations are performed using a combination of components in multiple containers 120 within one or more container groups 118 within one or more worker nodes in the worker nodes 104.

[0024] Once a cluster operation is scheduled, the manager node 102 monitors the container group 118. If the manager node 102 determines that a resource used for the cluster operation within one of the containers 120 of the container group 118 fails or is deleted, the manager node 102 replaces the deleted or inoperable container group 118 associated with the cluster operation with a different combination of currently available resources within the containers 120 of the container group 118. In this way, the API server 106 monitors the functionality of the container group 118 and recreates the container group 118 when the container group 118 is no longer operating as expected.

[0025] The manager node 102 also includes one or more computing cluster operators, such as application operators 126. The computing cluster operator is a software augmentation for the container management system that allows the behavior of the computing cluster 100 to be augmented, such as by augmenting the API server 106. The operator uses custom resources to manage applications and their components. Custom resources are resources of the computing cluster 100 that are customized for the application and are defined by custom resource definitions (CRDs). Custom resources can be stored in the storage component 110. Configurations and settings for the application can be stored in the custom resources for the operator. The operator includes a dedicated controller (and one or more associated container groups 118) managed by the controller manager 114. The controller of the operator is linked to the dedicated custom resources. Specifically, the controller of the operator includes a control loop that tracks the dedicated custom resources and performs actions based on these resources. The application operator 126 is an operator for an application based on the operator. An application based on the operator is a single instance stateful application that operates using custom resources. Therefore, an application based on the operator can be directly controlled by interacting with the custom resources using the management interface 108. In one example implementation, computing cluster 100 is Cluster, application operator 126 is Operators and operator-based applications are applications for monitoring the computing cluster 100, such as Prometheus.

[0026] The manager node 102 also includes one or more compute cluster controllers, such as a cluster group controller 128 managed by the controller manager 114. The compute cluster controller includes a control loop that tracks the resources of the compute cluster 100 and automatically makes or requests changes to the compute cluster 100 based on the status of the resources. As described in more detail later, multiple compute clusters 100 will be used to achieve high availability of operator-based applications for application operators 126. The cluster group controller 128 of the compute clusters 100 will be used to coordinate the compute clusters 100 so that the operator-based application is deployed on each of the compute clusters 100, but only one instance of the operator-based application is running across the compute clusters 100 at a given time. The operation of the cluster group controller 128 will be described in more detail later.

[0027] The nodes of computing cluster 100 (e.g., manager node 102 and worker node 104) can each include suitable components. Suitable components include processors, application specific integrated circuits, microcontrollers, memory, etc. The nodes of computing cluster 100 can each be a physical device, such as a computer. For example, and with reference to Figure 1B, the manager node 102 and the worker node 104 may each include a processor 132 and a memory 134. The memory 134 may be a non-transitory computer-readable medium storing instructions executed by the processor 132. One or more modules within a node of the computing cluster 100 may be partially or fully implemented as software and / or hardware for performing any of the functions described herein.

[0028] Figure 2 is a schematic diagram of an active-passive computing cluster system 200 according to some implementations. In the active-passive computing cluster system 200, multiple computing clusters 100 (including a first computing cluster 100A and a second computing cluster 100B) are grouped together and used to implement high availability of single-instance stateful applications such as operator-based applications. The first computing cluster 100A can be arranged at a first site (e.g., site A), and the second computing cluster 100B can be arranged at a second site (e.g., site B) different from the first site. Specifically, the first site can be located in a different geographic region from the second site, so that the first computing cluster 100A and the second computing cluster 100B are set in different geographic regions.

[0029] The active-passive computing cluster system 200 includes a cluster group manager 202 and an object storage service 204. The cluster group manager 202 may be a computing cluster controller deployed on the management computing cluster 100C. The cluster group manager 202 controls the computing clusters 100 that have been registered as group members so that they are grouped together in a high-availability cluster group. The cluster group controller 128 (including the first cluster group controller 128A and the second cluster group controller 128B) of the corresponding computing cluster 100 processes the deployment command from the cluster group manager 202. The cluster group manager 202 sends the deployment command to the cluster group controller 128 to coordinate the computing clusters 100 so that the operator-based application is deployed on each computing cluster in the computing cluster 100, but only one instance of the operator-based application runs across the computing cluster 100 at a given time. The storage component 110 (including the first storage component 110A and the second storage component 110B) of the corresponding computing cluster 100 stores customized resources defined for the operator-based application. The object storage service 204 will be used to facilitate synchronization of customized resources for operator-based applications between storage components 110. The object storage service 204 can be a cloud object storage service, such as an Amazon Simple Storage Service (S3) bucket, etc. In an example implementation, the storage components 110 are not synchronized in bulk via the object storage service 204, but only synchronized for customized resources defined by operator-based applications.

[0030] Initially, an operator-based application is deployed in each computing cluster in the computing cluster 100 along with its custom resource definition (CRD). Therefore, the computing cluster 100 may include application operators 126 (including a first application operator 126A and a second application operator 126B) on which the operator-based application is deployed. The cluster group manager 202 designates one site as an active site and other sites as passive sites. In this example, the active-passive computing cluster system 200 includes one passive site, but multiple passive sites can be utilized. The selection of active / passive sites can be performed via set rules or by manual execution (e.g., by an administrator of the active-passive computing cluster system 200 using a management interface for managing the computing cluster 100C). The cluster group controller 128 in the passive site then reduces the application operators 126 of the corresponding computing cluster 100 to zero. Specifically, the application operator 126 includes a container group, and the number of container groups for the application operator 126 at the passive site is set to zero. Additionally, the cluster group controller 128 in the active site then scales up the application operator 126 of the corresponding computing cluster 100 to be greater than zero. Specifically, the number of container groups for the application operator 126 at the active site is set to be greater than 0, such as 1. As a result, only one instance of an operator-based application runs across the computing cluster 100 at a given time.

[0031] The application operator 126 includes a control loop that changes the computing cluster 100 based on the customized resources of the operator-based application stored in the storage component 110. Synchronization of customized resources between the computing clusters 100 is performed using the object storage service 204. The cluster group controller 128 synchronizes the customized resources of the corresponding storage components 110 to and from the object storage service 204. As a result, the customized resources written to the storage component 110 of one computing cluster are synchronized with the storage component 110 of the other computing cluster. During operation, the cluster group controller 128 in the active site writes to its storage component 110, while the cluster group controller 128 in the passive site does not write to its storage component 110 because the cluster group controller 128 is scaled to zero (e.g., not running). As a result, the customized resources at the passive site are non-functional copies of the customized resources at the active site.

[0032] The active-passive computing cluster system 200 also includes a load balancer 206. The load balancer 206 can be a computing cluster service deployed on the management computing cluster 100C. The computing cluster service includes one or more container groups 118 (see Figure 1) that work together to provide functions, such as load balancing in the case of the load balancer 206. The load balancer 206 is configured to route requests from the client 208 to the computing cluster 100 of the active site.

[0033] The request for interaction with the custom resource at the active site causes the data for the custom resource to be synchronized to the passive site. When the active site fails (e.g., due to failure, maintenance, etc.), the cluster group manager 202 designates the previous passive site as the new active site, so that the role of the previous active / passive site is reversed. The cluster group controller 128 in the new active site expands the application operator 126 running in the corresponding computing cluster 100 to be greater than zero, while the cluster group controller 128 in the previous active site reduces the application operator 126 running in the corresponding computing cluster 100 to zero. The load balancer 206 is reconfigured to route requests from the client 208 to the computing cluster 100 of the new active site. As a result, the high availability of the application based on the operator can be realized at the application layer. The availability of the application based on the operator can therefore be improved across sites without changing the site infrastructure.

[0034] Figure 3 is a diagram of a computing cluster group deployment method 300 according to some implementations. Figure 2 The computing cluster group deployment method 300 is described below. The computing cluster group deployment method 300 may be performed by the cluster group manager 202 when the first computing cluster 100A and the second computing cluster 100B are established for high availability.

[0035] In step 302, a stateful application is deployed at a first application operator 126A and a second application operator 126B of a first computing cluster 100A and a second computing cluster 100B, respectively. The first application operator 126A includes a first control loop for changing the first computing cluster 100A based on a first customized resource of the first computing cluster 100A. The second application operator 126B includes a second control loop for changing the second computing cluster 100B based on a second customized resource of the second computing cluster 100B. The stateful application may be an operator-based application. The cluster group manager 202 may deploy the stateful application by sending a deployment command, which instructs the first cluster group controller 128A and the second cluster group controller 128B to deploy a copy of the stateful application to the first application operator 126A and the second application operator 126B, respectively.

[0036] In step 304, synchronization of the first customized resource with the second customized resource is configured. The first customized resource is stored in the first storage component 110A of the first computing cluster 100A. The second customized resource is stored in the second storage component 110B of the second computing cluster 100B. The first storage component 110A and the second storage component 110B can each be configured to synchronize with each other via the object storage service 204. In an example implementation in which the first storage component 110A is a first key-value storage library and the second storage component 110B is a second key-value storage library, the key-value storage libraries will be synchronized with each other. The cluster group manager 202 can configure synchronization of customized resources by sending a deployment command that instructs the first cluster group controller 128A and the second cluster group controller 128B to start synchronizing the customized resources stored in the first computing cluster 100A and the second storage component 110B with the object storage service 204, respectively.

[0037] In step 306, the first application operator 126A and the second application operator 126B are scaled to non-zero and zero, respectively. The first control loop of the first application operator 126A may be executed by a first manager node of the first computing cluster 100A, and the first application operator 126A may include a first container group deployed to a first worker node of the first computing cluster 100A. Scaling the first application operator 126A to non-zero may include setting a first number of the first container group to be greater than zero, such as 1. Similarly, the second control loop of the second application operator 126B may be executed by a second manager node of the second computing cluster 100B, and the second application operator 126B may include a second container group deployed to a second worker node of the second computing cluster 100B. Scaling the second application operator 126B to zero may include setting a second number of the second container group to zero. Therefore, the first number of the first container group is greater than the second number of the second container group. The cluster group manager 202 may scale the first application operator 126A and the second application operator 126B by sending a deployment command instructing the first cluster group controller 128A to scale the first application operator 126A up and instructing the second cluster group controller 128B to scale the second application operator 126B down.

[0038] In step 308, the load balancer 206 is configured to route the request to the first computing cluster 100A. At this point, the first computing cluster 100A is configured as an active computing cluster, and the second computing cluster 100B is configured as a passive computing cluster. In this way, the request from the client 208 is routed to the active computing cluster. The cluster group manager 202 can configure the load balancer 206 by sending a deployment command, and the deployment command instructs the load balancer 206 to change the load balancing weight so that for the load balancing decision, the active computing cluster is given a heavier weight than the passive computing cluster. In one example, the load balancing weight is set to zero for the passive computing cluster and is set to non-zero for the active computing cluster.

[0039] Additional steps may be performed. In one example implementation, before other steps are performed, the cluster group manager 202 deploys the first cluster group controller 128A and the second cluster group controller 128B to the first computing cluster 100A and the second computing cluster 100B, respectively. For example, when the first cluster group controller 128A and the second cluster group controller 128B are computing cluster controllers, they may be orchestrated on the first computing cluster 100A and the second computing cluster 100B via respective API servers for the computing clusters.

[0040] The computing cluster group deployment method 300 is initially performed during the deployment of the active-passive computing cluster system 200. Once completed, the active-passive computing cluster system 200 provides high availability of operator-based applications deployed at the first application operator 126A and the second application operator 126B. During operation, the current active computing cluster may fail (e.g., due to failure, maintenance, etc.). When the current active computing cluster fails, the cluster group manager 202 designates the new computing cluster as the active computing cluster and the new computing cluster as the passive computing cluster. The roles of the previous active / passive computing clusters are thus reversed.

[0041] Figure 4A-4B is a schematic diagram of an active-passive computing cluster system 200 during operation according to some implementations. Application operators 126 for stateful applications are deployed on each of the computing clusters 100 .

[0042] exist Figure 4AIn the first active-passive configuration shown, the first computing cluster 100A is designated as the active computing cluster and the second computing cluster 100B is designated as the passive computing cluster. The first application operator 126A running in the first computing cluster 100A is scaled to be greater than zero, while the second application operator 126B running in the second computing cluster 100B is scaled to zero. The first application operator 126A interacts with the custom resources stored in the first storage component 110A, and changes to those custom resources are synchronized with the second storage component 110B. The load balancer 206 is configured to route requests to the first computing cluster 100A.

[0043] exist Figure 4B In the second active-passive configuration shown, the first computing cluster 100A is designated as a passive computing cluster, and the second computing cluster 100B is designated as an active computing cluster. The first application operator 126A running in the first computing cluster 100A is scaled to zero, while the second application operator 126B running in the second computing cluster 100B is scaled to be greater than zero. The second application operator 126B interacts with the custom resources stored in the second storage component 110B, and changes to those custom resources are synchronized with the first storage component 110A. The load balancer 206 is configured to route requests to the second computing cluster 100B.

[0044] As previously described, the customized resources for the operator-based application are synchronized between the first storage component 110A and the second storage component 110B. As a result, when the designation of the active / passive computing cluster is changed, the application operator 126 of the new active computing cluster will have a copy of the customized resources used by the operator-based application. Therefore, the operator-based application can start running with low latency at any computing cluster where it is expanded. In addition, when the designation of the active / passive computing site is changed, the state of the operator-based application is maintained.

[0045] Figure 5 is a diagram of a computing cluster group reconfiguration method 500 according to some implementations. Figure 4A-4B 5. The computing cluster group reconfiguration method 500 may be performed by the cluster group manager 202 during a failover, for example, when reconfiguring the first computing cluster 100A and the second computing cluster 100B for high availability.

[0046] In step 502, the first computing cluster 100A is designated as the first active computing cluster, and the second computing cluster 100B is designated as the first passive computing cluster. Stateful applications are deployed at the first application operator 126A and the second application operator 126B of the first computing cluster 100A and the second computing cluster 100B, respectively. The stateful applications can be operator-based applications. The first application operator 126A of the first active computing cluster and the second application operator 126B of the first passive computing cluster are scaled to non-zero and zero, respectively. Additionally, the load balancer 206 is configured to route requests to the first active computing cluster (e.g., the first computing cluster 100A). In this way, the active-passive computing cluster system 200 is set to a first active-passive configuration (see Figure 4A ).

[0047] In step 504, an outage of a first active computing cluster (e.g., first computing cluster 100A) is detected by cluster group manager 202. The first active computing cluster may fail due to a malfunction, maintenance, etc. Cluster group manager 202 may detect an outage by receiving an outage message from first cluster group controller 128A, by detecting a loss of connection with first cluster group controller 128A, etc. In one example implementation, cluster group manager 202 detects an outage based on information of a container management system, such as information stored in storage component 110 of the computing cluster. Specifically, when an outage is detected, customized resources of the computing cluster may be considered. Information in storage component 110 may be used to detect a lack of application availability due to a resource shortage or outage.

[0048] In step 506, in response to detecting the outage, the first computing cluster 100A is designated as a second passive computing cluster, and the second computing cluster 100B is designated as a second active computing cluster. The first application operator 126A of the second passive computing cluster and the second application operator 126B of the second active computing cluster are scaled to zero and non-zero, respectively. Additionally, the load balancer 206 is configured to route the request to the second active computing cluster (e.g., the second computing cluster 100B). In this way, the active-passive computing cluster system 200 is configured to be a second active-passive configuration different from the first active-passive configuration (see Figure 4B). At this point, the active / passive computing cluster designation is reversed. Thus, requests from client 208 are routed to the second active computing cluster (formerly the first passive computing cluster) rather than the first active computing cluster (now the second passive computing cluster). When the active / passive computing cluster designation is reversed, the second active computing cluster will have a copy of the custom resources used by the operator-based application and can therefore begin running with low latency. The operator-based application will therefore begin running in the second active computing cluster in the same state as it was previously running at the first active computing cluster.

[0049] Some variations are contemplated. In one example implementation, an operator-based application also uses data stored on a persistent volume (e.g., a persistent volume of a container group 118) for state persistence. In such an implementation, a persistent volume can be migrated from a previous active computing cluster to a new active computing cluster during a failover. The migration can be performed by a volume migration process, rather than by synchronizing the volumes through the object storage service 204. For example, a persistent volume can be copied from a corresponding container group of a first computing cluster 100A to a corresponding container group of a second computing cluster 100B. Performing volume migration may increase failover latency, but may be more efficient than volume synchronization due to networking limitations. In another example implementation, multiple computing clusters may be grouped, and these groups are considered to be single units for active / passive designation. An active group of multiple active computing clusters can run multiple operator-based applications. When an interruption for any application in the active group is detected, the computing cluster in the active group can be reduced, and the corresponding computing cluster in the previous passive group can be expanded. The load balancer 206 can then be reconfigured so that the operator-based application in the new active group is used to process client requests.

[0050] In an example implementation according to features disclosed herein, a method includes: deploying a stateful application at a first application operator and a second application operator at a first computing cluster and a second computing cluster, respectively, the first application operator including changing a first control loop of the first computing cluster based on a first resource of the first computing cluster, the second application operator including changing a second control loop of the second computing cluster based on a second resource of the second computing cluster; configuring synchronization of the first resource with the second resource; scaling the first application operator and the second application operator to non-zero and zero, respectively; and configuring a load balancer to route requests to the first computing cluster. In another example implementation, the method further includes: scaling the first application operator and the second application operator to zero and non-zero, respectively; and reconfiguring the load balancer to route requests to the second computing cluster. In another example implementation of the method: the first computing cluster includes a first manager node and a first worker node, the first manager node executes the first control loop, the first application operator also includes a first container group on the first worker node; and the second computing cluster includes a second manager node and a second worker node, the second manager node executes the second control loop, and the second application operator also includes a second container group on the second worker node. In another example implementation of the method, scaling the first application operator and the second application operator includes: setting a first number of the first container group to be greater than zero; and setting a second number of the second container group to zero. In another example implementation of the method, the first resource is stored in a first key-value store of the first computing cluster, the second resource is stored in a second key-value store of the second computing cluster, and configuring synchronization of the first resource and the second resource includes: configuring synchronization of the first key-value store with the second key-value store. In another example implementation of the method, the first key-value store and the second key-value store are synchronized through a cloud object storage service. In another example implementation of the method, deploying a stateful application includes: deploying customized resource definitions for the first resource and the second resource on the first computing cluster and the second computing cluster, respectively. In another example implementation, the method further includes: deploying a first cluster group controller and a second cluster group controller at the first computing cluster and the second computing cluster, respectively, wherein scaling the first application operator and the second application operator includes sending a deployment command, the deployment command instructing the first cluster group controller to scale up the first application operator and instructing the second cluster group controller to scale down the second application operator. In another example implementation of the method, the stateful application is a single-instance stateful application.

[0051] In an example implementation according to features disclosed herein, a system includes: a plurality of computing clusters, each computing cluster in the computing clusters including resources; a cluster group manager configured to: deploy a stateful application on each respective computing cluster in the computing clusters, the stateful application including a control loop that changes the respective computing cluster based on the resources of the respective computing clusters; designate one computing cluster in the computing clusters as an active computing cluster and designate another computing cluster in the computing clusters as a passive computing cluster; scale the stateful application on the active computing cluster to greater than zero; and scale the stateful application on the passive computing cluster to zero. In another example implementation of the system, the cluster group manager is further configured to: configure a load balancer to route requests to the active computing cluster. In another example implementation of the system, the cluster group manager is further configured to: configure synchronization of resources of the active computing cluster and the passive computing cluster. In another example implementation of the system, the resources of the active computing cluster and the passive computing cluster are synchronized through an object storage service. In another example implementation of the system, the cluster group manager is further configured to: reverse the designation of the active computing cluster and the passive computing cluster in response to detecting an outage of the active computing cluster. In another example implementation of the system, an outage of the active computing cluster is detected based on resources of the active computing cluster.

[0052] In an example implementation according to features disclosed herein, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to: deploy a stateful application at a first application operator and a second application operator at a first computing cluster and a second computing cluster, respectively, the first application operator comprising changing a first control loop of the first computing cluster based on a first resource of the first computing cluster, the second application operator comprising changing a second control loop of the second computing cluster based on a second resource of the second computing cluster, the first computing cluster and the second computing cluster being arranged in different geographic regions; configure synchronization of the first resource and the second resource via an object storage service; scale the first application operator and the second application operator to non-zero and zero, respectively; and configure a load balancer to route requests to the first computing cluster. In another example implementation, the non-transitory computer-readable medium further stores instructions that, when executed by the processor, cause the processor to: in response to detecting an interruption of the first computing cluster, scale the first application operator and the second application operator to zero and non-zero, respectively; and reconfigure the load balancer to route requests to the second computing cluster. In another example implementation of a non-transitory computer-readable medium: a first computing cluster includes a first manager node and a first worker node, the first manager node executes a first control loop, a first application operator also includes a first container group on the first worker node; a second computing cluster includes a second manager node and a second worker node, the second manager node executes a second control loop, the second application operator also includes a second container group on the second worker node; and instructions for scaling the first application operator and the second application operator include instructions for: setting a first number of the first container group to 1; and setting a second number of the second container group to zero. In another example implementation of a non-transitory computer-readable medium, instructions for deploying a stateful application include instructions for: deploying customized resource definitions for the first resource and the second resource on the first computing cluster and the second computing cluster. In another example implementation of a non-transitory computer-readable medium, the stateful application is a single-instance stateful application.

[0053] The foregoing summarizes the features of several examples so that those skilled in the art can better understand the various aspects of the present disclosure. Various modifications and combinations of the illustrative examples and other examples will be apparent to those skilled in the art when referring to the specification. Therefore, the appended claims are intended to cover any such modifications.

Claims

1. A method comprising: Deploying a stateful application at a first application operator and a second application operator at a first computing cluster and a second computing cluster, respectively, the first application operator comprising changing a first control loop of the first computing cluster based on a first resource of the first computing cluster, the second application operator comprising changing a second control loop of the second computing cluster based on a second resource of the second computing cluster; configuring synchronization of the first resource and the second resource; scaling the first application operator and the second application operator to non-zero and zero, respectively; as well as A load balancer is configured to route requests to the first computing cluster.

2. The method according to claim 1, further comprising: scaling the first application operator and the second application operator to zero and non-zero, respectively; as well as The load balancer is reconfigured to route requests to the second computing cluster.

3. The method according to claim 1, wherein: The first computing cluster includes a first manager node and a first worker node, the first manager node executing the first control loop, the first application operator further including a first container group on the first worker node; and The second computing cluster includes a second manager node and a second worker node, the second manager node executing the second control loop, the second application operator also including a second container group on the second worker node.

4. The method of claim 3, wherein scaling the first application operator and the second application operator comprises: setting a first quantity of the first container group to be greater than zero; as well as The second quantity of the second container group is set to zero.

5. The method of claim 1, wherein the first resource is stored in a first key-value storage repository of the first computing cluster, the second resource is stored in a second key-value storage repository of the second computing cluster, and configuring synchronization of the first resource and the second resource comprises: Synchronization of the first key-value store and the second key-value store is configured.

6. The method of claim 5, wherein the first key-value store and the second key-value store are synchronized via a cloud object storage service.

7. The method according to claim 1, wherein deploying the stateful application comprises: Customized resource definitions for the first resource and the second resource are deployed on the first computing cluster and the second computing cluster, respectively.

8. The method according to claim 1, further comprising: deploying a first cluster group controller and a second cluster group controller at the first computing cluster and the second computing cluster respectively, Scaling the first application operator and the second application operator includes sending a deployment command, the deployment command instructing the first cluster group controller to scale up the first application operator and instructing the second cluster group controller to scale down the second application operator.

9. The method of claim 1, wherein the stateful application is a single-instance stateful application.

10. A system comprising: A plurality of computing clusters, each of the computing clusters comprising resources; The cluster group manager is configured to: deploying a stateful application on each respective one of the computing clusters, the stateful application including a control loop for changing the respective computing cluster based on the resources of the respective computing cluster; designating one of the computing clusters as an active computing cluster and designating another of the computing clusters as a passive computing cluster; scaling the stateful application on the active computing cluster to be greater than zero; and The stateful application on the passive computing cluster is scaled to zero.

11. The system of claim 10, wherein the cluster group manager is further configured to: A load balancer is configured to route requests to the active computing cluster.

12. The system of claim 10, wherein the cluster group manager is further configured to: Synchronization of the resources of the active computing cluster and the passive computing cluster is configured.

13. The system of claim 12, wherein the resources of the active computing cluster and the passive computing cluster are synchronized via an object storage service.

14. The system of claim 10, wherein the cluster group manager is further configured to: In response to detecting an outage of the active computing cluster, the designation of the active computing cluster and the passive computing cluster is reversed.

15. The system of claim 14, wherein the outage of the active computing cluster is detected based on the resources of the active computing cluster.

16. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to: Deploy a stateful application at a first application operator and a second application operator at a first computing cluster and a second computing cluster, respectively, the first application operator comprising a first control loop for changing the first computing cluster based on a first resource of the first computing cluster, the second application operator comprising a second control loop for changing the second computing cluster based on a second resource of the second computing cluster, the first computing cluster and the second computing cluster being arranged in different geographic regions; Configuring synchronization of the first resource and the second resource via an object storage service; scaling the first application operator and the second application operator to non-zero and zero, respectively; as well as A load balancer is configured to route requests to the first computing cluster.

17. The non-transitory computer readable medium of claim 16, further storing instructions that, when executed by the processor, cause the processor to: In response to detecting an outage of the first computing cluster, scaling the first application operator and the second application operator to zero and non-zero, respectively; and The load balancer is reconfigured to route requests to the second computing cluster.

18. The non-transitory computer readable medium of claim 16, wherein: The first computing cluster includes a first manager node and a first worker node, the first manager node executing the first control loop, the first application operator further including a first container group on the first worker node; The second computing cluster includes a second manager node and a second worker node, the second manager node executing the second control loop, the second application operator further including a second container group on the second worker node; and The instructions for scaling the first application operator and the second application operator include instructions for: setting a first quantity of the first container group to one; and The second quantity of the second container group is set to zero.

19. The non-transitory computer-readable medium of claim 16, wherein the instructions for deploying the stateful application include instructions for: Custom resource definitions for the first resource and the second resource are deployed on the first computing cluster and the second computing cluster.

20. The non-transitory computer-readable medium of claim 16, wherein the stateful application is a single-instance stateful application.

Images