High-availability methods and systems for Kubernetes federated management control planes across data centers

Abstract

This invention discloses a high-availability method for a cross-datacenter Kubernetes federated management control plane, belonging to the field of computer technology. The method includes: constructing management clusters and service clusters in multiple data centers; using the management cluster in one data center as the master cluster and the management clusters in the remaining data centers as slave master clusters; the master cluster acquiring federated resources and sending them to the service clusters in each data center; and the master cluster acquiring resource change information from user metadata and synchronizing it to the slave clusters. This invention also provides a high-availability system for a cross-datacenter Kubernetes federated management control plane. The federated management implemented by this invention truly achieves high availability across multiple data centers from the control plane to the service plane, ensuring that federated management can continue to provide services even when a single data center fails.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a high-availability method and system for a Kubernetes federated management control plane across data centers. Background Technology

[0002] As individual Kubernetes clusters grow larger, single clusters may encounter bottlenecks in terms of the number of managed nodes and containers. Simultaneously, many enterprises are migrating their core businesses to cloud-native environments, leading to a corresponding increase in the priority of cluster disaster recovery needs, such as dual-site three-center and active-active disaster recovery requirements.

[0003] Federation management, also known as a multi-cluster management framework, is a multi-cluster management technology in Kubernetes that enables multiple single Kubernetes clusters to be managed as a single cluster, thereby scaling up a single cluster.

[0004] In addition, federated management can also achieve cross-cluster scheduling and cross-cluster access capabilities across multiple monolithic clusters, assisting users in implementing cloud-native two-site three-center, active-active disaster recovery architectures, such as... Figure 1 As shown.

[0005] Existing federated management technologies are similar in model, such as Kubefed, Karmada, and OpenClusterManagement, and share the following architectural characteristics:

[0006] (1) It is divided into a management plane and a business plane. Communication between the management plane and the business plane is carried out by direct connection (Push) or proxy pull-down method;

[0007] (2) The resource data structure is divided into three modules: the resource template describes the general resource, the specialization process describes the differences between the general resource in different clusters, and the scheduling policy describes which clusters the general resource should be distributed to.

[0008] Using the federated management technology mentioned in the background, users can achieve their needs for expanding cluster size and multi-datacenter active-active disaster recovery in the following ways. For users using federated technology, the behavior is as follows:

[0009] 1) Users publish federated resources in the management cluster, which include scheduling policies from the federation model;

[0010] 2) The federation component monitors federation resources and scheduling policies, and converts federation resources into resources and publishes them to various business clusters according to the scheduling policies.

[0011] Most federated management technologies lack autonomy, and for users, the management cluster is a centralized control plane. Obviously, in a multi-datacenter environment, when the management cluster itself has problems, federated management technologies cannot provide services normally. Summary of the Invention

[0012] The purpose of this invention is to provide a high-availability method and system for a cross-datacenter Kubernetes federated management plane, so that federated management can still provide services normally when the federated management control plane of a single datacenter fails.

[0013] To address the aforementioned technical problems, this invention provides a high-availability method for a cross-datacenter Kubernetes federated management control plane, comprising the following steps:

[0014] Build management clusters and business clusters in multiple data centers respectively;

[0015] The management cluster of one of the data centers is designated as the master cluster, and the management clusters of the other data centers are designated as slave master clusters.

[0016] The main cluster connects to the business clusters in each data center;

[0017] The main cluster acquires federated resources and sends them to the business clusters in each data center;

[0018] The master cluster obtains the user's metadata information and synchronizes it to the slave clusters.

[0019] Preferably, the master cluster obtains the user's metadata information and synchronizes it to the slave cluster, specifically including the following steps:

[0020] Install the synchronization middleware on both the master and slave clusters;

[0021] The master cluster obtains and stores the metadata information sent by the user, and sends it to the synchronization middleware of the slave cluster through the synchronization middleware of the master cluster.

[0022] The synchronization middleware of the slave cluster synchronizes the metadata information stored in the slave cluster based on the received metadata information.

[0023] Preferably, the synchronization middleware of the master cluster sends its stored metadata information to the synchronization middleware of the slave cluster. The synchronization middleware of the slave cluster then synchronizes the metadata information stored in the slave cluster based on the received metadata information, specifically including the following steps:

[0024] The main cluster generates resource change information based on its stored metadata.

[0025] The synchronization middleware of the master cluster sends resource change information to the synchronization middleware of the slave cluster;

[0026] The synchronization middleware of the slave cluster synchronizes the metadata information stored in the slave cluster based on the received resource change information.

[0027] Preferably, the main cluster acquires federated resources and sends them to the business clusters in each data center, specifically including the following steps:

[0028] The main cluster sends federated resources to the business clusters in each data center through federated components.

[0029] Preferably, the synchronization middleware includes etcd synchronization middleware and mysql synchronization middleware.

[0030] Preferably, a GRPC connection is established between the synchronization middleware of the master cluster and the slave cluster.

[0031] Preferably, the method further includes the following steps:

[0032] When a failover event is detected, the master and slave clusters switch over.

[0033] Preferably, the number of data centers is two.

[0034] This invention also provides a high-availability system for a cross-datacenter Kubernetes federated management control plane, comprising:

[0035] The cluster building module is used to build management clusters and business clusters in multiple data centers respectively;

[0036] The cluster configuration module is used to designate the management cluster of one of the data centers as the master cluster and the management clusters of the other data centers as slave master clusters.

[0037] The cluster connection module is used to connect the main cluster with the business clusters in each data center;

[0038] The resource configuration module is used by the main cluster to obtain federated resources and send them to the business clusters in each data center;

[0039] The synchronization module is used by the master cluster to obtain user metadata information and synchronize it to the slave clusters.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] The federated management implemented in this invention truly achieves high availability across multiple data centers from the control plane to the business plane, ensuring that federated management can continue to provide services even when a single data center fails. Attached Figure Description

[0042] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic diagram of the community federation management technology architecture;

[0044] Figure 2 This is a schematic diagram of the high availability technology architecture for the multi-data center control plane of the present invention;

[0045] Figure 3 This is a schematic diagram of the multi-data center high availability architecture of the present invention. Detailed Implementation

[0046] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0047] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0049] The present invention will now be described in further detail with reference to the accompanying drawings:

[0050] This invention provides a high-availability method for a cross-datacenter Kubernetes federated management control plane, comprising the following steps:

[0051] Build management clusters and business clusters in multiple data centers respectively;

[0052] The management cluster of one of the data centers is designated as the master cluster, and the management clusters of the other data centers are designated as slave master clusters.

[0053] The main cluster connects to the business clusters in each data center;

[0054] The main cluster acquires federated resources and sends them to the business clusters in each data center;

[0055] The master cluster obtains the user's metadata information and synchronizes it to the slave clusters.

[0056] Preferably, the master cluster obtains the user's metadata information and synchronizes it to the slave cluster, specifically including the following steps:

[0057] Install the synchronization middleware on both the master and slave clusters;

[0058] The master cluster obtains and stores the metadata information sent by the user, and sends it to the synchronization middleware of the slave cluster through the synchronization middleware of the master cluster.

[0059] The synchronization middleware of the slave cluster synchronizes the metadata information stored in the slave cluster based on the received metadata information.

[0060] Preferably, the synchronization middleware of the master cluster sends its stored metadata information to the synchronization middleware of the slave cluster. The synchronization middleware of the slave cluster then synchronizes the metadata information stored in the slave cluster based on the received metadata information, specifically including the following steps:

[0061] The main cluster generates resource change information based on its stored metadata.

[0062] The synchronization middleware of the master cluster sends resource change information to the synchronization middleware of the slave cluster;

[0063] The synchronization middleware of the slave cluster synchronizes the metadata information stored in the slave cluster based on the received resource change information.

[0064] Preferably, the main cluster acquires federated resources and sends them to the business clusters in each data center, specifically including the following steps:

[0065] The main cluster sends federated resources to the business clusters in each data center through federated components.

[0066] Preferably, the synchronization middleware includes etcd synchronization middleware and mysql synchronization middleware.

[0067] Preferably, a GRPC connection is established between the synchronization middleware of the master cluster and the slave cluster.

[0068] Preferably, the method further includes the following steps:

[0069] When a failover event is detected, the master and slave clusters switch over.

[0070] Preferably, the number of data centers is two.

[0071] This invention also provides a high-availability system for a cross-datacenter Kubernetes federated management control plane, comprising:

[0072] The cluster building module is used to build management clusters and business clusters in multiple data centers respectively;

[0073] The cluster configuration module is used to designate the management cluster of one of the data centers as the master cluster and the management clusters of the other data centers as slave master clusters.

[0074] The cluster connection module is used to connect the main cluster with the business clusters in each data center;

[0075] The resource configuration module is used by the main cluster to obtain federated resources and send them to the business clusters in each data center;

[0076] The synchronization module is used by the master cluster to obtain user metadata information and synchronize it to the slave clusters.

[0077] In this invention, generally, in cross-data center scenarios, it is not possible to extend a single management cluster to other data centers. Therefore, it is necessary to start two management clusters and two federated components in the two data centers respectively to provide services.

[0078] Metadata information for federated components is typically stored in the cluster ETCD database. In this case, the two management clusters need to be marked as master / slave clusters, and data synchronization middleware needs to be started on the two management clusters.

[0079] The master cluster synchronization middleware establishes a GRPC connection with the slave cluster synchronization middleware; the master cluster synchronization middleware listens to the ETCD database of its cluster and subscribes to change events; when a change event is triggered, the master cluster synchronization middleware sends an incremental change request to the slave cluster synchronization middleware; the slave cluster synchronization middleware receives the request and reproduces the change event to ensure the consistency of federated management data between the master and slave clusters.

[0080] In addition, since the federated components operate in a subscription-listen-operate mode, even if the data is the same, repeated operations of the same behavior may cause some unpredictable problems. Therefore, in the cluster, the synchronization middleware will listen to the implementation of the federated components in the local cluster and adjust the number of service capabilities provided by the federated components to 0, thereby limiting the federated components to provide services.

[0081] The method and system technical architecture provided by this invention, such as Figure 2 As shown.

[0082] This invention provides a high-availability method and system for the Kubernetes federated management control plane across data centers. The federated management implemented using the technical architecture of this invention truly achieves high availability across multiple data centers from the control plane to the business plane, enabling federated management to continue providing services even when a single data center fails.

[0083] To better illustrate the technical effects of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process:

[0084] Example 1: A high-availability method for a cross-datacenter Kubernetes federated management control plane, as shown in Figure 3, has the following key points:

[0085] 1. In the first data center, launch the management cluster, start the federation component, and install the synchronization middleware (i.e., the etcd synchronization middleware in the diagram) to initialize the primary cluster; that is, to build the management cluster and the business cluster in the first data center.

[0086] 2. In the second data center, launch the management cluster, start the federated components, and install the synchronization middleware (i.e., the etcd synchronization middleware in the diagram), and initialize it as a slave cluster; that is, to build the management cluster and the business cluster in the second data center.

[0087] 3. The synchronization middleware monitors the master / slave status of the cluster it is in. If the cluster is the master cluster, it will synchronize the federated resources of this cluster to the peer cluster. If the cluster is the slave cluster, it will control the federated components of this cluster to suspend service to prevent split-brain.

[0088] 4. At most one master cluster can provide federated management services at any given time;

[0089] 5. The Federation Management Service connects all business clusters in the business plane, implements unified management, and distributes federated resources in the control plane to the two business clusters according to certain rules; it distributes the same application unit (resource) in multiple locations to achieve application disaster recovery or dual-active operation.

[0090] The rules can adopt the multi-cluster rules recommended by KubernetesSIG. The metadata information consists of: template + override + location. The template determines the baseline of the resource, the override determines the different performance of the resource in different clusters, and the location is the dispatch result and determines the deployment location of the resource.

[0091] The startup process of the synchronization middleware is as follows:

[0092] 1. The synchronization middleware obtains the current cluster configuration information and, based on the resource type described in the configuration list, listens for the corresponding resource in the cluster;

[0093] 2. The synchronization middleware identifies the current cluster identity. If it is the master cluster, it sends resource change requests to the slave clusters; if it is a slave cluster, it only accepts and processes the requests.

[0094] 3. The synchronization middleware in the master cluster will periodically send heartbeats to the slave clusters;

[0095] 4. Control the federated components of this cluster to pause service from the synchronization middleware in the cluster.

[0096] The switching process for the synchronization middleware is as follows:

[0097] 1. Change roles on the slave cluster, switch the slave cluster to the master cluster, after switching, open the send request module, synchronize the data of this cluster, and at the same time restore the federation components in the cluster to provide services;

[0098] 2. After receiving the switch event, the synchronization middleware synchronizes the event to the peer. If it fails, it keeps retrying.

[0099] 3. After receiving the role change request, the peer updates the original master cluster role, switches to slave cluster, and shuts down the request sending module, only accepting and processing requests, and controls the federated components in the cluster to suspend service.

[0100] This example uses custom rules to synchronize control plane data of the main cluster, improving scalability and adapting to different federated management implementation schemes.

[0101] The cluster synchronization middleware will suspend the service of the federated components in the cluster. Since these components are stateful, even if the metadata information of multiple data centers has been synchronized, responding to a certain event multiple times may produce unpredictable consequences.

[0102] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules, units, or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0103] The units may or may not be physically separate. The components shown as units can be one or more physical units, meaning they can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A high-availability method for a cross-datacenter Kubernetes federated management control plane, characterized in that, Includes the following steps: Build management clusters and business clusters in multiple data centers respectively; The management cluster of one of the data centers is designated as the master cluster, and the management clusters of the other data centers are designated as slave clusters. The main cluster connects to the business clusters in each data center; The main cluster acquires federated resources and sends them to the business clusters in each data center; Install the synchronization middleware on both the master and slave clusters; The main cluster generates resource change information based on its stored metadata. The synchronization middleware of the master cluster sends resource change information to the synchronization middleware of the slave cluster; The synchronization middleware of the slave cluster synchronizes the metadata information stored in the slave cluster based on the received resource change information; In the cluster, the synchronization middleware will monitor the implementation of the federated components in the local cluster and adjust the number of instances providing services to 0, thereby limiting the federated components to provide services.

2. The high availability method for a cross-datacenter Kubernetes federated management control plane according to claim 1, characterized in that, The primary cluster acquires federated resources and sends them to the business clusters in each data center, specifically including the following steps: The main cluster sends federated resources to the business clusters in each data center through federated components.

3. The high availability method for a cross-datacenter Kubernetes federated management control plane according to claim 1, characterized in that: The synchronization middleware includes etcd synchronization middleware and mysql synchronization middleware.

4. The high availability method for a cross-datacenter Kubernetes federated management control plane according to claim 1, characterized in that: A GRPC connection is established between the synchronization middleware of the master cluster and the slave cluster.

5. The high availability method for a cross-datacenter Kubernetes federated management control plane according to claim 1, characterized in that, It also includes the following steps: When a failover event is detected, the master and slave clusters switch over.

6. The high availability method for a cross-datacenter Kubernetes federated management control plane according to claim 1, characterized in that: There are two data centers.

7. A high-availability system for a cross-datacenter Kubernetes federated management control plane, used to implement the high-availability method for a cross-datacenter Kubernetes federated management control plane as described in any one of claims 1-6, characterized in that, include: The cluster building module is used to build management clusters and business clusters in multiple data centers respectively; The cluster configuration module is used to designate the management cluster of one of the data centers as the master cluster and the management clusters of the other data centers as slave clusters. The cluster connection module is used to connect the main cluster with the business clusters in each data center; The resource configuration module is used by the main cluster to obtain federated resources and send them to the business clusters in each data center; The synchronization module is used by the master cluster to obtain user metadata information and synchronize it to the slave clusters.

Images