Resource control method, system, and program in cloud environment

Abstract

A method for controlling resources in a cloud environment is provided, the method comprising: determining whether a quorum of each of the set of clusters is online; determining whether one or more applications of the cluster are running on another cluster of the set of clusters in response to determining that the quorum of each of the set of clusters is online; determining whether the cluster is designated as a highest priority cluster in response to determining that one or more applications of the cluster are not running on another cluster of the set of clusters; determining whether the cluster is designated as a highest priority cluster; determining whether a main cluster of the set of clusters is online in response to determining that the main cluster is online; and scaling a new custom resource to one (1) in response to determining that the main cluster is online.

Description

[Technical field]

[0001] The present invention relates generally to the field of cloud computing, and more particularly to data protection and resource control in distributed hybrid multi-cloud environments using Kubernetes clusters. [Background technology]

[0002] A multi-cloud environment exists when an enterprise uses multiple public cloud services, often from different cloud providers. Because not all cloud providers are created equal, organizations adopt a multi-cloud strategy to deliver the best IT services, to avoid locking in to a single cloud provider, or to take advantage of cloud arbitrage and choose a provider for a particular service based on which provider is offering the lowest price at a given time.

[0003] A hybrid cloud environment exists when an enterprise's cloud environment includes a private cloud infrastructure, such as the enterprise's own data center, along with one or more public cloud services that typically work in conjunction to achieve a business goal. Thus, when an enterprise's cloud environment includes multiple public cloud services and private cloud components, it is considered a hybrid multicloud environment.

[0004] Kubernetes is an open source system for managing containerized applications across multiple hosts. It provides a fundamental mechanism for application deployment, maintenance, and scaling. Enterprises offer Kubernetes-based platforms and infrastructure as a service (IaaS) that deploy Kubernetes. Once Kubernetes is deployed, a cluster is formed. A Kubernetes cluster consists of at least one worker machine (i.e., node) that runs containerized applications. A worker node hosts pods (i.e., a set of running containers in a cluster), which are components of an application workload. A cluster also contains a control plane that manages the worker nodes and pods in the cluster. In production environments, the control plane typically runs across multiple computers, and a cluster typically runs multiple nodes to provide fault tolerance and high availability. Summary of the Invention

[0005] Aspects of an embodiment of the present invention disclose a method, computer program product, and computer system for ensuring data protection and control in a distributed hybrid multi-cloud environment using Kubernetes clusters. A processor determines whether a quorum of each of the set of clusters is online. In response to determining that a quorum of each of the set of clusters is online, the processor determines whether one or more applications of the cluster are running on another cluster of the set of clusters. In response to determining that one or more applications of the cluster are not running on another cluster of the set of clusters, the processor determines whether the cluster is designated as a highest priority cluster. In response to determining that the cluster is designated as a highest priority cluster, the processor determines whether a main cluster of the set of clusters is online. In response to determining that the main cluster is online, the processor scales the new custom resource to one (1). [Brief description of the drawings]

[0006] [Figure 1] 1 is a functional block diagram illustrating a distributed data processing environment in accordance with an embodiment of the present invention. [Diagram 2] 1 is a flowchart illustrating operational steps of a data protection program for controlling application integrity locking using a new custom resource in a distributed hybrid multi-cloud environment using a Kubernetes cluster according to an embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a cloud computing environment according to an embodiment of the present invention. [Figure 4] FIG. 2 illustrates an abstract model layer according to an embodiment of the present invention. [Diagram 5]FIG. 2 is a block diagram of components of a computing node of a cloud computing environment in the distributed data processing environment of FIG. 1 for executing a data protection program in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] With digital transformation and the exponential growth of data, businesses want to extend their computing power beyond the limitations of data centers by storing data and computing on the cloud, often on many different clouds. Many technologies exist for multi-cloud, multi-cluster, multi-geography container application management, but these are entirely focused on stateless application movement between sites. To enable a true hybrid multi-cloud environment, embodiments of the present invention recognize the need to also enable stateful application movement.

[0008] Embodiments of the present invention recognize that when deploying distributed hybrid multicloud environments using Kubernetes clusters (also referred to herein simply as "clusters"), there is an unsolved challenge of asynchronously accessing persistent volumes (also referred to as "PersistentVolumes" or "PVs") across more than two clusters. This creates an issue where PVs can become corrupted when accessed simultaneously against a file system without proper locking methods, which is not possible in asynchronous mode. Thus, embodiments of the present invention recognize the need to maintain consistency across geographies and enable cloud mobility for geographically unaware stateful applications and databases in a multicloud environment.

[0009] Embodiments of the present invention provide systems and methods that work with Kubernetes clusters to ensure persistent volume locks that control access to loosely coupled asynchronous replicas. Even if an application with a database is migrated from one location to another cloud, embodiments of the present invention should prevent access anywhere unless the application and database are fully terminated in all other locations. Embodiments of the present invention ensure that access locks are created, maintained, migrated, and released when needed at the application level. This helps ensure data integrity and secure application mobility even for stateful applications when there are ongoing PV input / output (I / O) operations.

[0010] Embodiments of the present invention enhance Kubernetes cluster capabilities across multiple regions. Embodiments of the present invention provide an underlying asynchronous replicated set across clusters that can spawn persistent volumes across different clouds, including private on-premise clouds. When different cloud vendors offer different Kubernetes cluster types, embodiments of the present invention enable application integrity locking across clouds through the creation of a new custom resource (also called a "Custom Resource" or "CR") on each Kubernetes cloud service that is controlled by embodiments of the present invention. The new custom resource has an operator on each cluster to expose an application programming interface (API) service to communicate state across clusters.

[0011] In Kubernetes, a resource is an endpoint of the Kubernetes API that stores a collection of API objects of some kind. A custom resource is an extension of the Kubernetes API that represents a customization of a particular Kubernetes installation. Custom resources are defined using the Custom Resource Definition (also known as "CRD") API, where a new custom resource with a specified name and schema is created by defining a CRD object. CRDs allow users to create new types of resources without adding a separate API server.

[0012] Implementations of embodiments of the present invention may be in a variety of forms, and details of example implementations are discussed below with reference to the figures.

[0013] FIG. 1 is a functional block diagram illustrating a distributed data processing environment, generally designated 100, in accordance with one embodiment of the present invention. As used herein, the term "distributed" refers to a computer system that includes multiple physically separate devices that operate together as a single computer system. FIG. 1 is merely illustrative of one implementation and does not imply any limitation with respect to the environment in which different embodiments may be implemented. Many modifications to the illustrated environment may be made by one of ordinary skill in the art without departing from the scope of the invention as set forth in the claims.

[0014] The distributed data processing environment 100 includes a hybrid multi-cloud environment 110 and a user computing device 120 interconnected via a network 105. The network 105 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. The network 105 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, or video signals, including multimedia signals including voice, data, and video information. In general, the network 105 may be any combination of connections and protocols that would support communications between the hybrid multi-cloud environment 110, the user computing device 120, and other computing devices (not shown) in the distributed data processing environment 100.

[0015] The hybrid multi-cloud environment 110 is a cloud computing environment having at least one public cloud and at least one private cloud. In an embodiment, the hybrid multi-cloud environment 110 represents a server computing system that utilizes multiple computers as server systems, as in a cloud computing environment, and can communicate with the user computing devices 120 and other computing devices (not shown) in the distributed data processing environment 100 via the network 105. In another embodiment, the hybrid multi-cloud environment 110 represents a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within the distributed data processing environment 100. The hybrid multi-cloud environment 110 includes a cluster 112. The hybrid multi-cloud environment 110 may include internal and external hardware components, as illustrated and described in further detail with respect to FIGS. 3-5.

[0016] The clusters 112 operate as Kubernetes clusters, each containing a copy of the data protection program 114 and an application protected by the data protection program 114. A Kubernetes cluster is a set of computing nodes that run containerized applications. In an embodiment of the invention, the clusters 112 include at least three clusters. The nodes of the clusters 112 may include internal and external hardware components, as shown and described in further detail with respect to FIG. 5.

[0017] The data protection program 114 operates as a method for ensuring data protection and control in a distributed hybrid multi-cloud environment using Kubernetes clusters. In other words, the data protection program 114 configures and controls entities that protect applications running on clusters controlled by StatefulSet or Deployment entities. StatefulSet or Deployment entities are standard entities on clusters that are used to protect applications on clusters, such as cluster 112, that utilize PV. The present invention creates a new layer on top of StatefulSet or Deployment that prevents corruption in a hybrid multi-cloud environment. In another embodiment, the data protection program 114 may reside on another computing device, server, cloud server, or may be spread across multiple devices elsewhere (not shown) in the distributed data processing environment 100, provided that the data protection program 114 has access to the network 105. In the illustrated embodiment, the data protection program 114 is a standalone program with a copy running on each cluster of the cluster 112. In another embodiment, the data protection program 114 may be integrated into another software product, such as a cloud service offering. The main operations of the data protection program 114 are shown and described in further detail with respect to Figure 2. Before the main operations of the data protection program 114 can occur, an initial setup is completed by a user, for example, through a user interface 122 of a user computing device 120.

[0018] In an embodiment, the data protection program 114 allows a user to complete the initial setup of a new CR defined using a CRD for use in a distributed hybrid multi-cloud environment with Kubernetes clusters, such as hybrid multi-cloud environment 110. As used herein, a "new CR" and a "new CRD" refer to a new custom resource created and controlled by the data protection program 114. The initial setup includes allowing a user (e.g., an administrator) to set up the following configurations for all clusters: In an embodiment, the data protection program 114 allows a user to create a new CR. In an embodiment, the data protection program 114 allows an administrator to be configured with the authority to retrieve, list, edit, and create new CRs on all clusters in the hybrid multi-cloud environment. In an embodiment, the data protection program 114 allows a user to create (i.e., configure) a Kubernetes operator (or simply "operator"). A Kubernetes operator is a software extension to Kubernetes that utilizes custom resources to manage applications and their components to control the operation of the new CR. The Kubernetes operator ensures that the data protection program 114 is running properly in each cluster of the clusters 112.

[0019] In an embodiment, the data protection program 114 allows an administrator to set cluster priority settings and a main cluster for the new CR. The cluster priority settings tell the operator the initial priority (i.e., which cluster) on which the application will run during normal operation. In an embodiment, the data protection program 114 has two operating modes: (1) normal and (2) disaster recovery. When operating in normal mode, the data protection program 114 runs the new CR on the highest priority cluster as defined by the administrator and disaster recovery is not declared. By setting a main cluster for the new CR, the data protection program 114 allows the administrator to set which cluster has the source of the data, considering that this is based on asynchronous replication due to the extended distances between different clouds. For data consistency, there needs to be one cluster that is the "main owner" of the data source that is always the main point for asynchronous replication.

[0020] In an embodiment, the data protection program 114 allows an administrator to configure to the operator which clusters are part of the new CR, which is a list of Kubernetes clusters part of the CR, which includes all Kubernetes cluster API endpoints. An embodiment of the invention assumes that there are at least three clusters, and the quorum (majority of clusters) is automatically calculated by the number of clusters on the list.

[0021] A CR declaration has at least two definitions. In an embodiment, the data protection program 114 allows an administrator to define (1) one or more persistent volumes (PVs) and persistent volume claims (PVCs), and (2) one or more stateful sets or deployment entities in a Kubernetes cluster. Both (1) and (2) are Kubernetes entities that define running applications and are controlled by the data protection program 114 in an embodiment herein. These definitions can also be imported at the time of the initial creation of a CRD. The initial number of application replicas (i.e., pods) expected for a deployment or stateful set when a new CR is scaled to one (1) must be defined. If a new CR is scaled to zero (0), the data protection program 114 sets the replication of the deployment or stateful set to zero (0).

[0022] In an embodiment, the data protection program 114 allows an administrator to configure application migration, which causes an application migration command to redefine cluster priorities across all Kubernetes clusters (i.e., determine a new highest priority cluster defined by the administrator in the application migration), whereby the data protection program 114 automatically enforces the correct application migration to the new highest priority cluster. For example, the data protection program 114 allows an administrator to configure the migration of an application from cloud A to cloud B in the hybrid multi-cloud environment 110. By allowing an administrator to configure application migration, the data protection program 114 allows an administrator to change the application location.

[0023] The user computing device 120 operates as a computing device associated with a user through which the user can interact with the data protection program 114 through an application user interface. In the illustrated embodiment, the user computing device 120 includes an instance of a user interface 122. In an embodiment, the user computing device 120 may be a laptop computer, a tablet computer, a smartphone, a smartwatch, an e-reader, smart glasses, a wearable computer, or any programmable electronic device capable of communicating with various components and devices in the distributed data processing environment 100 over the network 105. In general, the user computing device 120 represents one or more programmable electronic devices or a combination of programmable electronic devices capable of executing machine-readable program instructions and communicating with other computing devices (not shown) in the distributed data processing environment 100 over a network, such as the network 105.

[0024] The user interface 122 provides an interface between the data protection program 114 on the hybrid multi-cloud environment 110 and a user of the user computing device 120. In one embodiment, the user interface 122 is mobile application software. Mobile application software, or "apps," are computer programs designed to run on smartphones, tablet computers, and other mobile computing devices. In one embodiment, the user interface 122 may be a graphical user interface (GUI) or web user interface (WUI) that can display text, documents, web browser windows, user options, application interfaces, and instructions for operation and can include information (such as graphics, text, and sounds) that the program presents to the user, as well as control sequences that the user employs to control the program. The user interface 122 enables the user of the user computing device 120 to complete an initial setup of the data protection program 114 and / or act as an administrator of the data protection program 114. The user interface 122 allows a user of the user computing device 120 to view or manage the functionality and / or output of the data protection program 114 as desired, for example allowing the user to declare disaster recovery if necessary.

[0025] 2 is a flowchart 200 illustrating the operational steps of the data protection program 114 for controlling application integrity locking using new CRs in a distributed hybrid multi-cloud environment with Kubernetes clusters, in accordance with an embodiment of the present invention. It should be understood that the process illustrated in FIG. 2 illustrates one possible iteration of the data protection program 114 running on each of the clusters 112, and that the process is from the perspective of one cluster at a time.

[0026] In decision 205, data protection program 114 determines whether there is a quorum of clusters. In an embodiment, data protection program 114 determines whether there are quorum of "healthy" (i.e., able to communicate with each other) or online clusters among clusters 112. A quorum of clusters is defined as a majority of clusters 112, i.e., 50% or more of clusters 112. For example, if there are three clusters, two clusters must be online to have quorum. In an embodiment, data protection program 114 receives an indication from a Kubernetes operator on the cluster of whether there is a quorum among the online clusters.

[0027] If data protection program 114 determines that there is no quorum (the No branch of decision 205), data protection program 114 proceeds to step 210. In step 210, data protection program 114 scales the CR to zero (0) on the clusters that it did not determine had quorum to ensure the integrity of the given data, and then proceeds to step 215. In step 215, data protection program 114 sends a communication to all reachable clusters (i.e., clusters that are online and therefore responsive to the communication) that the CR is down for this cluster, and then returns to waiting until a quorum is reached.

[0028] If the data protection program 114 determines that there is a quorum (yes branch at decision 205), the data protection program 114 proceeds to decision 220. At decision 220, the data protection program 114 determines whether one or more applications are running on any other reachable clusters. If the data protection program 114 determines that one or more applications are running on any other reachable clusters (yes branch at decision 220), the data protection program 114 proceeds to step 210 (as described above) to scale the CR to zero (0) to ensure consistency and ensure that the protected application or applications run in only one cluster. Thus, by setting the CR to zero (0) for the cluster, the data protection program 114 has protected the one or more applications from corruption.

[0029] If the data protection program 114 determines that the application is not running on any other reachable cluster (decision 220, no branch), the data protection program 114 proceeds to decision 225. In decision 225, the data protection program 114 determines whether this is the highest priority cluster in the quorum. In an embodiment, the data protection program 114 determines whether this is the highest priority cluster in the quorum by checking which cluster was set as the highest priority by an administrator. For example, a user is set as the administrator via a user interface 122 on a user computing device 120. If the data protection program 114 determines that this is not the highest priority cluster in the quorum (decision 225, no branch), the data protection program 114 proceeds to step 210 (as described above) to scale the CR to zero (0) because the highest priority cluster in the quorum should be the only cluster for starting, i.e., running, one or more applications.

[0030] If the data protection program 114 determines that this is the highest priority cluster in the quorum (yes branch of decision 225), the data protection program 114 proceeds to decision 230. In decision 230, the data protection program 114 determines whether the main cluster is up. In an embodiment, the data protection program 114 determines whether the main cluster is up, i.e., online and ready to run one or more applications. The main cluster is the cluster where the source of data is stored and defined during initial setup by the administrator.

[0031] If the data protection program 114 determines that the main cluster is not up (the no branch of decision 230), the data protection program 114 proceeds to decision 235. In decision 235, the data protection program 114 determines whether disaster recovery (DR) has been declared, for example, by an administrator setting an operating mode for the data protection program 114. If the administrator places the data protection program 114 in a normal operating mode, DR is not declared. If the administrator places the data protection program 114 in a DR operating mode, DR is declared, which essentially means that the administrator does not believe that the main cluster is online or will be online. If the data protection program 114 determines that DR is not declared (the no branch of decision 235), the data protection program 114 proceeds to step 210 (as described above) and scales the CR to zero (0).

[0032] If the data protection program 114 determines that the main cluster is up (yes branch at decision 230) or if the data protection program 114 determines that DR is declared (yes branch at decision 235), the data protection program 114 proceeds to step 240. In step 240, the data protection program 114 scales the CR to one (1), and then proceeds to step 245. In step 245, the data protection program 114 sends a communication to all reachable clusters that the application is scaled to one (1), and then proceeds to step 250. In step 250, the data protection program 114 configures the deployment or stateful set entity (whatever entity the cluster has) to the desired replication (i.e., the number of pods the user configures during configuration, depending on the needs of the application being protected) to ensure that only one of the Kubernetes clusters in the hybrid multicloud environment accesses the persistent volume claim (PVC) to ensure the integrity of one or more applications. In an embodiment, the data protection program 114 allows the application to start up by setting the stateful set or deployment entity to the desired replica set during configuration. In other words, the CR is active on the cluster scaled to one (1) and the application is running on that cluster that owns the data, while the other clusters are set to zero (0). This causes the application to remain down as the CR sets the stateful set or deployment entity to zero (0). Once step 250 is complete, the data protection program 114 loops back to the beginning to continue checking the clusters for the conditions as described above.

[0033] Although this disclosure includes a detailed description of cloud computing, it should be understood that implementation of the teachings presented herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0034] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computational resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) and for rapidly provisioning and releasing these resources with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0035] Its features are as follows:

[0036] On-Demand Self-Service: Cloud customers can unilaterally provision computing capacity, such as server time and network storage, as needed automatically, without the need for human interaction with the service provider.

[0037] Broad network access: Cloud capabilities are available over the network and can be accessed using standard mechanisms, facilitating usage by heterogeneous thin- or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0038] Resource Pool: The provider's computing resources are pooled and offered to multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have a sense of location independence in that they have no control or knowledge regarding the exact location of the resources offered to them, although at a higher level of abstraction they may be able to specify a location (e.g., country, state, or data center).

[0039] Rapid Elasticity: Capacity is provisioned quickly and elastically, in some cases automatically, so it can be quickly scaled out and quickly released to quickly scale in. Capacity available for provisioning often appears to the consumer as unlimited, available for purchase in any quantity at any time.

[0040] Metered Services: Cloud systems leverage metering to automatically control and optimize resource usage at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0041] The service model is as follows:

[0042] SaaS (Software as a Service): The capability provided to the consumer is the use of the provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for the possibility of limited user-specific application configuration settings.

[0043] PaaS (Platform as a Service): The capability provided to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the application hosting environment.

[0044] Infrastructure as a Service (IaaS): The capability provided to a consumer is the provisioning of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases, limited control over selected network components (e.g., host firewalls).

[0045] The deployment model is as follows:

[0046] Private Cloud: This cloud infrastructure is operated solely for the organization. It is managed by the organization or a third party and may reside on-premise or off-premise.

[0047] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may reside on-premise or off-premise.

[0048] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0049] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0050] A cloud computing environment is a service-oriented environment with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0051] Referring now to FIG. 3, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as previously described (not shown). This allows the cloud computing environment 50 to provide an infrastructure, platform, and / or SaaS that does not require the cloud consumer to maintain resources on a local computing device. It will be understood that the types of computing devices 54A-54N illustrated in FIG. 1 are intended to be exemplary only, and that computing node 10 and cloud computing environment 50 may communicate with any type of computer-controlled device via any type of network and / or network-addressable connection (e.g., a connection using a web browser).

[0052] Referring now to Figure 4, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 3) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 4 are intended to be exemplary only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0053] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, Reduced Instruction Set Computer (RISC) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0054] The virtualization layer 70 comprises an abstraction layer at which virtual entities such as virtual servers 71 , virtual storage 72 , virtual networks including virtual private networks 73 , virtual applications and operating systems 74 , and virtual clients 75 are provided.

[0055] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computational and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the utilization of those resources. In one example, those resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computational resources to meet required service levels. Service level agreement (SLA) planning and execution 85 provides pre-provisioning and procurement of cloud computational resources in anticipation of upcoming demands in accordance with SLAs.

[0056] The workload layer 90 provides examples of functions for which cloud computing environments are utilized. Examples of workloads and functions provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and data protection processing 96.

[0057] 5 illustrates a block diagram of one component of a cloud computing node 10 suitable for running the data protection program 114 in the distributed data processing environment 100 of FIG. 1 in accordance with an embodiment of the present invention. It should be appreciated that FIG. 5 is merely provided as an example of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0058] Computing device 500 includes a communications fabric 502 that provides communications between cache 516, memory 506, persistent storage 508, communications unit 510, and input / output (I / O) interfaces 512. Communications fabric 502 may be implemented using any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communications and network processor, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 502 may be implemented with one or more buses or crossbar switches.

[0059] Memory 506 and persistent storage 508 are computer-readable storage media. In this embodiment, memory 506 includes random access memory (RAM). In general, memory 506 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 516 is a high-speed memory that enhances the performance of computer processor 504 by holding recently accessed data and data near the accessed data from memory 506.

[0060] The programs may be stored in persistent storage 508 and memory 506 for execution and / or access via cache 516 by one or more of the respective computer processors 504. In an embodiment, persistent storage 508 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 508 may include a solid state hard drive, a semiconductor storage device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0061] The media used by persistent storage 508 may also be removable. For example, a removable hard drive may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive to transfer data to another computer-readable storage medium that is also part of persistent storage 508.

[0062] Communications unit 510, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 510 includes one or more network interface cards. Communications unit 510 may provide communications using either or both physical and wireless communications links. Programs may be downloaded to persistent storage 508 through communications unit 510.

[0063] The I / O interface 512 allows for input and output of data with other devices connected to one of the cloud computing nodes 10. For example, the I / O interface 512 may provide a connection to an external device 518, such as a keyboard, a keypad, a touch screen, or any other suitable input device, or a combination thereof. The external device 518 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention may be stored on such portable computer-readable storage media and loaded into the persistent storage 508 via the I / O interface 512. The I / O interface 512 also connects to a display 520.

[0064] Display 520 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0065] The programs described herein are identified based on the application for which they are implemented in particular embodiments of the invention, but it should be understood that any particular program nomenclature herein is used merely as a matter of convenience, and thus the invention should not be limited to use in any particular application identified and / or suggested by such nomenclature.

[0066] The present invention may be a system, method, and / or computer program product, which may include a computer readable storage medium (or media) having computer readable program instructions for causing a processor to carry out aspects of the present invention.

[0067] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0068] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0069] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language, or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, or partially on the user's computer, as a stand-alone software package, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0070] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0071] These computer readable program instructions may be provided to a general purpose computer, special purpose computer, or other programmable data processing apparatus processor to produce a machine, the instructions of which execute via the processor of the computer, or other programmable data processing apparatus processor, to produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, such that the computer readable storage medium having stored thereon an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0072] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operable steps to create a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0073] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may be performed out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, are implemented by a dedicated hardware-based system that performs the specified functions or actions or performs a combination of dedicated hardware and computer instructions.

[0074] Although the description of various embodiments of the present invention has been presented for illustrative purposes, the description is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the present invention. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method running on a cluster of a set of clusters, comprising: determining, by one or more processors, whether a quorum of each of the set of clusters is online; determining, by the one or more processors, in response to determining that a quorum of each of the set of clusters is online, whether one or more applications of the cluster are running on another cluster of the set of clusters; determining, by the one or more processors, whether the cluster has been designated as a highest priority cluster in response to determining that the one or more applications of the cluster are not running on another cluster of the set of clusters; determining, by the one or more processors, whether a main cluster of the set of clusters is online in response to determining that the cluster has been designated as the highest priority cluster; and in response to determining that the main cluster is online, scaling, by the one or more processors, a new custom resource to one (1).

2. sending, by the one or more processors, a communication to each cluster of the set of clusters that is online stating that the new custom resource has been scaled to one (1); The method of claim 1 , further comprising: setting, by the one or more processors, a deployment entity or a statefulset entity to a desired replica.

3. in response to determining that a quorum of each of the set of clusters is not online, scaling the new custom resource to zero (0) on the cluster by the one or more processors; 2. The method of claim 1, comprising: sending, by the one or more processors, a communication to each cluster in the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

4. in response to determining that the one or more applications of the cluster are running on another cluster of the set of clusters, scaling new custom resources on the cluster by the one or more processors to zero (0); 2. The method of claim 1, further comprising: sending, by the one or more processors, a communication to each cluster in the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

5. in response to determining that the cluster is not designated as the highest priority cluster, scaling the new custom resource to zero (0) on the cluster by the one or more processors; 2. The method of claim 1, further comprising: sending, by the one or more processors, a communication to each cluster in the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

6. 2. The method of claim 1, further comprising determining, by the one or more processors, whether a disaster recovery mode is active in response to determining that the main cluster is not online.

7. in response to determining that the disaster recovery mode is not active, scaling the new custom resource to zero (0) on the cluster by the one or more processors; 7. The method of claim 6, further comprising: sending, by the one or more processors, a communication to each cluster in the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

8. A computer program product running on a cluster of a set of clusters, said computer program product comprising: one or more computer readable storage media; and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising: program instructions for determining whether a quorum of each of the set of clusters is online; in response to determining that a quorum of each of the set of clusters is online, program instructions for determining whether one or more applications of the cluster are running on another cluster of the set of clusters; in response to determining that the one or more applications of the cluster are not running on another cluster of the set of clusters, program instructions for determining whether the cluster is designated as a highest priority cluster; in response to determining that the cluster is designated as the highest priority cluster, program instructions for determining whether a main cluster of the set of clusters is online; and program instructions for scaling a new custom resource to one (1) in response to determining that the main cluster is online.

9. program instructions for sending a communication to each cluster of the set of clusters that is online stating that the new custom resource is scaled to one (1); 9. The computer program product of claim 8, further comprising: program instructions for setting the deployment entity or the statefulset entity to a desired replica.

10. in response to determining that a quorum of each of the set of clusters is not online, program instructions for scaling the new custom resource to zero (0) on the cluster; 9. The computer program product of claim 8, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

11. in response to determining that the one or more applications of the cluster are running on another cluster of the set of clusters, program instructions for scaling the new custom resource to zero (0) on the cluster; 9. The computer program product of claim 8, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

12. in response to determining that the cluster is not designated as the highest priority cluster, program instructions for scaling the new custom resource to zero (0) on the cluster; 9. The computer program product of claim 8, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

13. 9. The computer program product of claim 8, further comprising program instructions for determining whether a disaster recovery mode is active in response to determining that the main cluster is not online.

14. in response to determining that the disaster recovery mode is not active, program instructions for scaling the new custom resource to zero (0) on the cluster; 14. The computer program product of claim 13, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

15. 1. A computer system comprising: one or more computer processors; one or more computer readable storage media; program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors on a cluster of a set of clusters, the program instructions comprising: program instructions for determining whether a quorum of each of the set of clusters is online; in response to determining that a quorum of each of the set of clusters is online, program instructions for determining whether one or more applications of the cluster are running on another cluster of the set of clusters; in response to determining that the one or more applications of the cluster are not running on another cluster of the set of clusters, program instructions for determining whether the cluster is designated as a highest priority cluster; in response to determining that the cluster is designated as the highest priority cluster, program instructions for determining whether a main cluster of the set of clusters is online; and in response to determining that the main cluster is online, scaling a new custom resource to one (1).

16. program instructions for sending a communication to each cluster of the set of clusters that is online stating that the new custom resource is scaled to one (1); 16. The computer system of claim 15, further comprising: program instructions for setting a deployment entity or a statefulset entity to a desired replica.

17. in response to determining that a quorum of each of the set of clusters is not online, program instructions for scaling the new custom resource to zero (0) on the cluster; 16. The computer system of claim 15, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

18. in response to determining that the one or more applications of the cluster are running on another cluster of the set of clusters, program instructions for scaling the new custom resource to zero (0) on the cluster; 16. The computer system of claim 15, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

19. in response to determining that the cluster is not designated as the highest priority cluster, program instructions for scaling the new custom resource to zero (0) on the cluster; 16. The computer system of claim 15, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

20. in response to determining that the main cluster is not online, program instructions for determining whether a disaster recovery mode is active; in response to determining that the disaster recovery mode is not active, program instructions for scaling the new custom resource to zero (0) on the cluster; 16. The computer system of claim 15, further comprising: program instructions for sending a communication to each cluster of the set of clusters that are online, the communication including that the new custom resource has been scaled to zero (0).

Images