Multi-tenant-supporting multi-cluster application fault migration method and system

By supporting tenant-level resource isolation and automated failure migration solutions in multi-cluster environments, the problems of flexibility, reusability and low resource utilization in multi-cluster application failure migration technology are solved, and efficient application migration and resources are achieved manage.

CN119088623AActive Publication Date: 2024-12-06CHINA ACADEMY OF RAILWAY SCI CORP LTD +2

Patent Information

Application Number
CN202411132256.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The existing multi-cluster application failure migration technology has problems such as inflexible point-to-point migration, does not support per-tenant migration, poor reusability, complex configuration and low resource utilization, making it difficult to effectively migrate applications when a single kubernetes cluster fails.

Method used

By creating a multi-cluster environment, including management clusters and work clusters, it supports tenant-level resource isolation and migration, and uses failure migration policies and resource quota information table to generate automated migration solutions, combining multi-cluster state collection modules, failure migration scheduling modules and tenant management Modules to enable flexible migration of tenants, applications and components.

Benefits of technology

It improves the flexibility and resource utilization of failure migration, reduces configuration complexity, supports migration and reuse by tenant, minimizes service stop time, and improves the overall utilization efficiency of resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088623A_ABST
    Figure CN119088623A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-tenant supporting multi-cluster application fault migration method and system, and relates to the technical field of cluster application fault migration, the method comprises the following steps: utilizing kubernetes application software to create a multi-cluster environment; for each working cluster in the multi-cluster environment, creating a tenant in the working cluster; for each tenant, deploying an application in the tenant, and configuring a fault migration strategy associated with the application; generating a resource quota information table according to the fault migration strategy and the state information of the working cluster; according to the resource quota information table, a fault migration scheme is generated and used for providing technical guidance after a certain work cluster breaks down, so that migratable tenants, applications and components in the work cluster with the fault are migrated to other work clusters without the fault and redeployment is carried out. According to the method, migration and reuse according to tenants are supported, the adjustment flexibility is improved, the configuration complexity is reduced, and the resource utilization rate is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of cluster application failure migration, and in particular to a multi-cluster application failure migration method and system supporting multi-tenants. Background Art

[0002] As containerized applications become more and more popular, most enterprises use kubernetes (a container orchestration engine open sourced by Google) as a container orchestration tool to carry out container operations. Although the high availability of the kubernetes cluster can guarantee the reliability of applications on the cluster to a certain extent, it still cannot avoid single cluster failures caused by network failures or host failures. Therefore, in order to enable enterprise applications to provide stable external services under any circumstances, it is necessary to solve the problem of application failure migration under the premise of a single kubernetes cluster failure and improve application reliability.

[0003] Currently, multiple Kubernetes clusters are usually used to form a multi-cluster. When a cluster fails, the applications running on the failed cluster can be migrated to other healthy clusters. However, this method has the following problems:

[0004] (1) Point-to-point migration. It only supports migration from cluster A to cluster B. It cannot be adjusted flexibly. Each adjustment requires reconfiguration of the policy.

[0005] (2) Migration by tenant is not supported. The current cluster fault migration solutions are all for cluster migration or single application fault migration, which do not involve the concept of tenants and do not use tenant resource metering.

[0006] (3) Each case is discussed separately, and reuse is not supported. After a cluster failure occurs, the cluster failure migration configuration needs to readjust the strategy to adapt to the new cluster environment as the cluster environment changes, which is not flexible enough.

[0007] (4) Complex configuration. In order to control the fault migration process, users need to configure more parameters, which increases the difficulty and threshold for users to use and is not conducive to the implementation of the overall strategy.

[0008] (5) A lot of resource waste. Many of the existing Kubernetes clusters in the enterprise are running at relatively full capacity, and each cluster has scattered resources remaining but cannot be effectively utilized. In order to meet the overall migration of application failures, a new backup cluster needs to be built to accommodate the applications to be migrated, resulting in a lot of resource waste and increasing enterprise costs.

[0009] In summary, how to provide a multi-cluster application fault migration method that can be flexibly adjusted, supports migration and reuse by tenant, has simple configuration, and has high resource utilization is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0010] The purpose of this application is to provide a multi-cluster application fault migration method and system that supports multi-tenants, which not only supports migration and reuse by tenants, but also improves adjustment flexibility, reduces configuration complexity, and improves resource utilization.

[0011] To achieve the above objectives, this application provides the following solutions:

[0012] In a first aspect, the present application provides a multi-cluster application failure migration method supporting multiple tenants, the multi-cluster application failure migration method supporting multiple tenants comprising:

[0013] Utilize kubernetes application software to create a multi-cluster environment; the multi-cluster environment includes a management cluster and several working clusters, the management cluster is used to manage each of the working clusters, and the working clusters are used to deploy tenants.

[0014] For each working cluster in the multi-cluster environment, a tenant is created in the working cluster. Each working cluster includes a plurality of tenants, and the tenants are used for resource isolation in a single working cluster scenario and cross-cluster resource isolation in a multi-working cluster scenario.

[0015] For each of the tenants, an application is deployed within the tenant, and a fault migration strategy associated with the application is configured; the application includes multiple components, the fault migration strategy includes tenant information and application information, the application information includes application weight and application topology, the application weight is used to indicate the priority of the application during fault migration, and the application topology includes the maximum number of replicas and the minimum number of replicas of each component in the application after fault migration.

[0016] A resource quota information table is generated according to the fault migration strategy and the status information of the working cluster; the status information of the working cluster includes the application deployment information, application resource occupancy information and the overall resource occupancy information of the working cluster for each tenant in the working cluster; the resource quota information table includes the total resource information of each working cluster, the resource occupancy information of each tenant, the maximum resource usage, the average resource occupancy value and the working cluster status.

[0017] A fault migration plan is generated according to the resource quota information table; the fault migration plan is used to provide technical guidance after a certain working cluster fails, so as to migrate the tenants, applications and components that can be migrated in the failed working cluster to other working clusters that have not failed and redeploy them.

[0018] Optionally, a multi-cluster status collection module, a fault migration scheduling module and a tenant management module are provided in the management cluster.

[0019] The multi-cluster status collection module is used to configure the access authorization certificate of each working cluster in the management cluster, and to obtain the status information and resource information of each working cluster through the API Server of each working cluster.

[0020] The fault migration scheduling module is used to obtain the status information and resource information of each working cluster from the multi-cluster status collection module, and generate the fault migration plan in real time according to the fault migration strategy, and execute the fault migration of the application.

[0021] The tenant management module is used to manage tenant information in each of the working clusters.

[0022] Optionally, for each of the working clusters in the multi-cluster environment, creating a tenant in the working cluster specifically includes:

[0023] According to the tenant configuration information, the corresponding tenant is created in each of the working clusters; wherein the tenant configuration information includes the tenant name, the total resource quota and the allocated quota, the total resource quota represents the total CPU quota, total memory quota and total storage quota occupied by the tenant in the entire multi-cluster environment, and the allocated quota represents the CPU quota, memory quota and storage quota occupied by the tenant in each of the working clusters.

[0024] An identification code is generated for each of the tenants, where the identification code is used to identify the identity information of the tenant.

[0025] Optionally, the identification code is a UUID code.

[0026] Optionally, for each of the tenants, deploying an application in the tenant and configuring a fault migration strategy associated with the application specifically includes:

[0027] An automatic deployment tool is used to deploy applications to each tenant, and tenant information and application information corresponding to each tenant are obtained.

[0028] The fault migration strategy is determined according to the tenant information and application information corresponding to each tenant.

[0029] Optionally, the automatic deployment tool is Argocd.

[0030] Optionally, generating a fault migration plan according to the resource quota information table specifically includes:

[0031] A preliminary fault migration plan is generated according to the resource quota information table.

[0032] The preliminary fault migration plan is optimized to obtain an optimized fault migration plan; the optimized fault migration plan is used as the finally generated fault migration plan.

[0033] Optionally, the preliminary fault migration plan is optimized to obtain an optimized fault migration plan, which specifically includes:

[0034] According to the application weights corresponding to the respective applications, an application with the smallest application weight is determined from among all the applications of the tenant that have satisfied the migration condition.

[0035] For each component in the application with the smallest application weight, when the maximum number of copies of a component is greater than 1, the number of copies of the component in the preliminary fault migration plan is reduced by 1, and the number of reducible copies of all components in all applications is reduced in this way, until the components that do not meet the migration conditions are included in the preliminary fault migration plan.

[0036] When all reducible copies have been reduced and there are still applications and their components that do not meet the migration conditions, the applications and their components that do not meet the migration conditions are marked as types that cannot be migrated, and other components of the application that already meet the migration conditions are deleted from the preliminary fault migration plan to obtain the optimized fault migration plan.

[0037] Optionally, after the step of optimizing the preliminary fault migration solution to obtain an optimized fault migration solution, the multi-cluster application fault migration method supporting multiple tenants further includes:

[0038] After a failure occurs in a certain working cluster, the tenants, applications and components that can be migrated in the failed working cluster are failed over according to the optimized failure migration plan, and a failure migration report is generated; the failure migration report includes the migration-in working cluster information and the migration-out working cluster information of the successfully migrated applications in each tenant in the failed working cluster, as well as the names and components of the applications that cannot be migrated.

[0039] In a second aspect, the present application provides a multi-cluster application fault migration system supporting multiple tenants, wherein the multi-cluster application fault migration system supporting multiple tenants comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-cluster application fault migration method supporting multiple tenants described in the first aspect.

[0040] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0041] The present application provides a multi-cluster application fault migration method and system supporting multiple tenants. By creating a multi-cluster environment, deploying applications for tenants, and configuring fault migration strategies associated with applications, a resource quota information table can be generated according to the fault migration strategy and the status information of the working cluster. Further, a fault migration plan can be automatically generated according to the resource quota information table. When a certain working cluster fails, the tenants, applications, and components that can be migrated in the failed working cluster can be migrated and redeployed to other normal working clusters according to the fault migration plan. Not only does it support migration and reuse by tenants, but it can also flexibly adjust the resource quota information table and the fault migration plan according to the fault migration strategy and the status information of the working cluster, effectively improving the flexibility of adjustment and reducing the complexity of configuration. In addition, tenants, applications, and their components can automatically implement fault migration in a multi-cluster environment, which can minimize the service downtime and effectively improve the utilization of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 The present invention is a flowchart of a multi-cluster application failure migration method supporting multiple tenants in one embodiment of the present application.

[0044] Figure 2 A diagram of a multi-cluster architecture provided in accordance with an embodiment of the present application.

[0045] Figure 3 A diagram showing the relationship between tenants and multiple clusters provided in an embodiment of the present application.

[0046] Figure 4 A diagram of application migration strategy configuration parameters in a tenant provided in an embodiment of the present application.

[0047] Figure 5A diagram showing the relationship between tenants and the total amount of application resources provided in an embodiment of the present application.

[0048] Figure 6 A schematic diagram of overall tenant failure migration provided in an embodiment of the present application.

[0049] Figure 7 A relationship diagram between applications and the total amount of component resources provided in an embodiment of the present application.

[0050] Figure 8 A schematic diagram of overall migration of applications within a tenant provided in an embodiment of the present application.

[0051] Fig. 9 A schematic diagram of completing the migration of application subcomponents provided in an embodiment of the present application.

[0052] Fig.10 A schematic diagram of the structure of a multi-cluster application failure migration system supporting multiple tenants provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] like Figure 1 As shown, this embodiment proposes a multi-cluster application failure migration method supporting multiple tenants, and the multi-cluster application failure migration method supporting multiple tenants includes the following steps:

[0056] Step S1: Use kubernetes application software to create a multi-cluster environment.

[0057] In this embodiment, the multi-cluster environment includes a management cluster and several working clusters. The management cluster is used to manage each of the working clusters, and the working clusters are used to deploy tenants.

[0058] In this embodiment, a multi-cluster status collection module, a fault migration scheduling module and a tenant management module are provided in the management cluster.

[0059] The multi-cluster status collection module is mainly used to configure the access authorization certificate of each working cluster in the management cluster, and to obtain the status information and resource information of each working cluster through the API Server of each working cluster.

[0060] The fault migration scheduling module is mainly used to obtain the status information and resource information of each working cluster from the multi-cluster status collection module, and generate the fault migration plan in real time according to the fault migration strategy, and execute the fault migration of the application.

[0061] The tenant management module is mainly used to manage tenant information in each of the working clusters.

[0062] Step S2: for each of the working clusters in the multi-cluster environment, create a tenant in the working cluster.

[0063] In this embodiment, each of the working clusters includes a plurality of tenants, and the tenants are used for resource isolation in a single working cluster scenario and cross-cluster resource isolation in a multi-working cluster scenario.

[0064] In this embodiment, step S2 creates a tenant in each working cluster in the multi-cluster environment, specifically including:

[0065] Step S21. Create the corresponding tenant in each of the working clusters according to the tenant configuration information; wherein the tenant configuration information includes the tenant name, the total resource quota and the allocated quota, the total resource quota represents the total CPU quota, total memory quota and total storage quota occupied by the tenant in the entire multi-cluster environment, and the allocated quota represents the CPU quota, memory quota and storage quota occupied by the tenant in each of the working clusters.

[0066] Step S22: After the tenants are constructed, an identification code may be generated for each tenant, and the identification code is used to identify the identity information of the tenant. The identification code may be a UUID (Universally Unique Identifier) ​​or other types of identification codes.

[0067] Step S3: for each of the tenants, deploy an application in the tenant and configure a fault migration policy associated with the application.

[0068] In this embodiment, the application includes multiple components, the fault migration strategy includes tenant information and application information, the application information includes application weight and application topology, the application weight is used to indicate the priority of the application during fault migration, and the application topology includes the maximum number of copies and the minimum number of copies of each component in the application after fault migration.

[0069] In this embodiment, step S3 deploys an application in each tenant and configures a fault migration strategy associated with the application, specifically including:

[0070] Step S31: Use an automatic deployment tool to deploy applications to each tenant, and obtain tenant information and application information corresponding to each tenant. The automatic deployment tool may be Argocd or other automatic deployment tools.

[0071] Step S32: Determine the fault migration strategy according to the tenant information and application information corresponding to each tenant.

[0072] Step S4: Generate a resource quota information table according to the fault migration strategy and the status information of the working cluster.

[0073] In this embodiment, the status information of the working cluster includes the application deployment information, application resource occupancy information and the overall resource occupancy information of the working cluster of each tenant in the working cluster, and the resource quota information table includes the total resource information of each working cluster, the resource occupancy information of each tenant, the maximum resource usage, the average resource occupancy value and the working cluster status.

[0074] Step S5: Generate a fault migration plan according to the resource quota information table.

[0075] In this embodiment, the fault migration solution can be used to provide technical guidance after a certain working cluster fails, so as to migrate the tenants, applications and components that can be migrated in the failed working cluster to other working clusters that have not failed and redeploy them.

[0076] In this embodiment, step S5 generates a fault migration plan according to the resource quota information table, which specifically includes:

[0077] Step S51: Generate a preliminary fault migration plan according to the resource quota information table.

[0078] Step S52: Due to the limitation of tenant resource quota, the created preliminary fault migration plan may not satisfy the fault migration of all applications and components. Therefore, this embodiment considers the need to optimize the preliminary fault migration plan, so that it can satisfy the fault migration of as many applications or components as possible, thereby greatly improving resource utilization. Specifically, the preliminary fault migration plan is optimized to obtain an optimized fault migration plan. The optimized fault migration plan is used as the final generated fault migration plan.

[0079] In this embodiment, step S52 optimizes the preliminary fault migration plan to obtain an optimized fault migration plan, which specifically includes:

[0080] Step S521: According to the application weights corresponding to the respective applications, determine the application with the smallest application weight from among all the applications of the tenant that meet the migration condition.

[0081] Step S522: For each component in the application with the smallest application weight, when the maximum number of copies of a component is greater than 1, the number of copies of the component in the preliminary fault migration plan is reduced by 1, and the number of reducible copies of all components in all applications is reduced in this manner until the components that do not meet the migration conditions are included in the preliminary fault migration plan.

[0082] Step S523: When all reducible copies have been reduced and there are still applications and their components that do not meet the migration conditions, the applications and their components that do not meet the migration conditions are marked as types that cannot be migrated, and other components of the application that already meet the migration conditions are deleted from the preliminary fault migration plan to obtain the optimized fault migration plan.

[0083] In this embodiment, after the step of optimizing the preliminary fault migration solution in step S52 to obtain an optimized fault migration solution, the following steps are also included:

[0084] Step S53: After a certain working cluster fails, perform failover on the tenants, applications and components that can be migrated in the working cluster that fails according to the optimized failover solution, and generate a failover report. The failover report includes the migration-in working cluster information and the migration-out working cluster information of the successfully migrated applications in each tenant in the working cluster that fails, as well as the names and components of the applications that cannot be migrated.

[0085] This embodiment proposes a multi-cluster application failure migration method that supports multi-tenants. In actual application, the specific operation process is as follows:

[0086] Step (1): Create a multi-cluster environment.

[0087] In this embodiment, first, in a multi-cluster environment consisting of multiple Kubernetes environments, a multi-cluster environment refers to an environment with multiple clusters. The clusters here are divided into two categories: management cluster and working cluster. There is usually one management cluster and several working clusters, such as Figure 2 As shown. Select one of the clusters (3 master nodes and 1-2 node nodes) as the management cluster, and the other clusters as the working clusters. Kubernetes is abbreviated as k8s, which is an abbreviation that uses the number "8" to replace the 8 characters "ubernete" in the middle of the name. It is an open source containerized application for managing multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating and maintenance.

[0088] In this embodiment, the following modules need to be deployed in the management cluster: multi-cluster status collection module, fault migration scheduling module and tenant management module. The above three modules constitute the multi-cluster management plane as a whole. Figure 2 As shown in the figure, the main functions of each module are as follows:

[0089] 1) Multi-cluster status collection module: mainly used to configure access authorization certificates for each working cluster in the management cluster. This module can obtain the working cluster status and resource information through each working cluster. This module runs in the management cluster as a containerized application. Among them, APIServer is the central API coordinator in the k8s cluster, acting as the brain and entry point of the cluster. It is responsible for receiving and processing API requests from users, controllers and other components, and executing these requests in the cluster.

[0090] 2) Fault migration scheduling module: mainly used to obtain the status information and resource information of each working cluster from the multi-cluster status collection module, generate scheduling strategies in real time according to the fault migration strategy configured by the user, and execute the fault migration of the application; this module runs in the management cluster as a containerized application.

[0091] 3) Tenant management module: mainly used to manage tenant information of each work cluster.

[0092] In this embodiment, the method of incorporating the working cluster into the management cluster requires the following two operations: importing the kubeconfig file of each working cluster into the specified file directory of the management cluster as the identity basis for the multi-cluster status collection module to read the working cluster information. And configuring the IP address and port number of the APIServer of each working cluster in the multi-cluster status collection module of the management cluster, and the multi-cluster status collection module obtains the working cluster information through the IP address and port.

[0093] Step (2): Create cross-cluster tenants in a multi-cluster environment.

[0094] In this embodiment, in a single-cluster scenario, tenants are resource divisions for working clusters, which are used to isolate resources within the cluster. In a multi-cluster scenario, tenants are used for cross-cluster resource isolation, that is, a tenant can occupy resources on multiple clusters, and the sum of the resources occupied by the tenant on each working cluster is the tenant's overall resource quota. Several users can be created in a tenant, and users with tenant resource management capabilities are called tenant administrators. In the management cluster, the kubernetes administrator submits tenant configuration information in a multi-cluster environment to the tenant management module, and the tenant configuration information is used to create tenants. The tenant configuration information includes the tenant name, total resource quota, and allocation quota. Among them, the total resource quota indicates the total CPU quota, total memory quota, and total storage quota occupied by the tenant in the entire multi-cluster environment. The allocation quota indicates the CPU quota, memory quota, and storage quota occupied by the tenant on each working cluster (which can be all working clusters or some working clusters). The sum of the allocations of each working cluster should be less than or equal to the total quota. The relationship between tenants and multi-clusters is as follows Figure 3 shown.

[0095] In this embodiment, after the tenant is created, the tenant management module will generate a UUID to represent the tenant, and create a cluster user in all the working clusters involved in the tenant, and associate the UUID to represent the tenant administrator in the working cluster. Among them, UUID is an industrial standard used in computer systems to ensure that information is highly unique. Usually, platforms and programming languages ​​provide corresponding functions or methods to generate UUIDs, with the purpose of enabling all elements in a distributed system to have unique identification information without the need to specify identification information through a central control terminal.

[0096] Step (3): Deploy the application within the tenant.

[0097] In this embodiment, the application is a logical integration of multiple components, each of which is a container image that can be deployed on Kubernetes. A containerized application consists of one or more components, each of which is a containerized image. The application can be deployed in any cluster of the working cluster, and the deployment method and script are consistent with the deployment of services in Kubernetes. It is necessary to write scripts such as Kubernetes deployment, service, and configmap for each application. The script must include the resource size required by the application component. The tenant administrator of the working cluster created by the above step (2) uses an automatic deployment tool (such as Argocd) to implement the deployment of the application in a single working cluster. After successful deployment, the application will occupy a certain amount of resources. At this time, the remaining available resource quota of the tenant should be equal to the current available resource quota minus the resource quota occupied by the application.

[0098] Step (4): Configure the failover policy associated with the application within the tenant.

[0099] In this embodiment, after the application is deployed, the Kubernetes administrator who manages the cluster submits the basic failure migration strategy of all applications under each tenant to the failure migration module. The failure migration strategy mainly includes tenant information and application information. The policy configuration parameter file is as follows: Figure 4 As shown in the figure, tenant information includes tenant name and tenant ID. Application information includes application weight and application topology. Application weight is the priority of the application during failover. Applications not listed are configured by default with the lowest priority of 1. Application topology is the maximum and minimum number of replicas of each component in the application after failover. Components not listed are configured by default with the maximum and minimum number of replicas of 1.

[0100] It should be noted that the Kubernetes administrator and the tenant administrator are different. The Kubernetes administrator is the system administrator, which refers to the user with the highest authority in the Kubernetes cluster and is responsible for the management of the Kubernetes cluster itself; while the tenant administrator refers to a user with management authority in the tenant and is responsible for the management of resources within the tenant.

[0101] Step (5): The management cluster generates and maintains the resource quota information table in real time based on the status information and fault migration strategy of the current working cluster.

[0102] In this embodiment, there can be multiple tenants in a multi-cluster environment, and the resources of each tenant can span multiple clusters. The multi-cluster status collection module in the multi-cluster management plane will refresh the working cluster status regularly. For example, every 15 seconds, the multi-cluster status collection module reads the status information of the working cluster through the working cluster APIServer interface. The status information includes the deployment status of the application under each tenant of the current working cluster, the application resource occupancy information, and the overall occupancy of the current cluster resources. Therefore, the multi-cluster status collection module will generate a resource quota information table based on the status of each working cluster and the tenant resource quota and other information, as shown in Table 1. Table 1 takes the two working clusters of working cluster 1 and working cluster 2 as examples for illustration, and can actually include more working clusters.

[0103] Table 1 Resource quota information table

[0104]

[0105] Step (6): Once the management cluster senses a working cluster failure, it generates an application failure migration plan based on the resource quota information table.

[0106] In this embodiment, when a working cluster fails to communicate with the multi-cluster management plane for two consecutive refresh cycles, the working cluster status is marked as "abnormal". When a working cluster is marked as "abnormal", the multi-cluster management plane will start the failover of the application in the working cluster. In this embodiment, the refresh cycle can be set to 30s.

[0107] In this embodiment, when the multi-cluster management plane of the management cluster senses that a certain working cluster is in an abnormal state, such as a connection interruption, inability to obtain working cluster information, etc., the fault migration scheduling module will automatically create a fault migration plan based on the current "resource quota information table" (i.e., Table 1) and the deployment information of the applications in each working cluster. The creation process of the fault migration plan includes the following:

[0108] 1) The working cluster that fails can be referred to as the failure cluster. All tenants in the failure cluster are regarded as independent entities. The actual total amount of resources occupied by tenant i in the failure cluster R(i) is equal to the sum of the resource usage quotas of each application in the tenant, such as Figure 5 shown.

[0109] 2) Among all working clusters in normal working status, find a working cluster that meets the following conditions: the available resources RA(i) of tenant i in the working cluster exceeds the total resources R(i) occupied by tenant i in the faulty cluster; RA(i)-R(i) is the largest among all working clusters that meet the conditions; this working cluster is used as the target cluster for the overall migration of all applications in tenant i in the faulty cluster, that is, it is ultimately necessary to migrate to this working cluster, which can also be called the migration-in working cluster. Correspondingly, the working cluster that has a fault is called the migration-out working cluster. Figure 6 shown.

[0110] 3) If no working cluster can be found that satisfies RA(i)≥R(i), it means that all working clusters are insufficient to accommodate all applications of tenant i as a whole; then a failure migration plan is planned separately for each application in tenant i in the failure cluster.

[0111] 4) Sort tenant i's applications from high to low according to their application priority. For applications with the same priority configuration, sort them according to the principle of minimum total resource required. Applications with less resource consumption have higher priority, and applications with more resource consumption have lower priority. The resource consumption size is compared in the order of storage, CPU, and memory. For applications with the same storage requirements, applications with less CPU requirements have smaller total resources. For applications with the same storage and CPU requirements, applications with less memory usage have smaller total resources.

[0112] 5) From the sorted queue, calculate the total resource usage Rapp(i, j) for each application j in turn, as Figure 7 As shown in the figure, a working cluster that meets the following conditions is searched among all working clusters in normal working status: the available resources RA(i) of tenant i in the working cluster exceeds the total resources occupied by application j of tenant i in the faulty cluster Rapp(i, j). RA(i)-Rapp(i, j) is the largest among all clusters that meet the conditions; the working cluster is used as the target cluster for the migration of application j of tenant i in the faulty cluster, and application j is deleted from the queue. Figure 8 shown.

[0113] 6) If the tenant i space of all working clusters in normal working status cannot completely accommodate all components of application j of tenant i of faulty cluster, the fault migration of application j is subdivided into different components of application j being migrated to different working clusters.

[0114] 7) For component k of application j in tenant i of the faulty cluster, calculate its total resource usage RappComp(i, j, k), and search for a cluster that meets the following conditions in all working clusters in normal working status: the available resources RA(i) of tenant i in the working cluster exceed the resource usage RappComp(i, j, k) of component k of application j of tenant i in the faulty cluster; RA(i)-RappComp(i, j, k) is the largest among all working clusters that meet the conditions; this working cluster serves as the target cluster for migration of component k of application j of tenant i in the faulty cluster.

[0115] 8) If all components in application j in tenant i meet the migration conditions, then delete application j from the queue; select the next application from the queue and jump to step 5). Fig. 9 shown.

[0116] Step (7): The management cluster optimizes the generated preliminary fault migration plan.

[0117] In this embodiment, the optimization and adjustment principles of the fault migration scheduling module for the preliminary fault migration plan are as follows:

[0118] 1) When all components of all applications in the tenant have met the migration conditions (the sorting queue has been cleared), there is no need for solution optimization and go directly to step (8).

[0119] 2) If there are still applications or application components in the sorting queue that do not meet the migration conditions, then select the application j with the lowest priority from the applications of the current tenant that meet the migration conditions. For its component k, if the maxReplicas (maximum number of copies) of component k is greater than 1, then reduce its number of copies by 1 from the preliminary fault migration plan to save some resources, but the value after reduction is not less than minReplicas (minimum number of copies), thereby reducing the number of resources occupied by the component after migration, and jump to step (6) of step 6). Among them, the preliminary fault migration plan is a fault migration plan created according to the default configuration policy, for example, a certain application component maintains 3 copies after migration to the new working cluster. maxReplicas and minReplicas represent the maximum number of copies and the minimum number of copies, respectively.

[0120] 3) Continue to try to reduce the number of removable replicas of all applications until the components that do not meet the migration conditions are included in the fault migration plan.

[0121] 4) If all reducible replicas have been reduced and there are still application components in the queue that do not meet the conditions, mark the application as unable to be failed over (providing information for generating a migration report), and remove other components of the application that meet the migration conditions from the plan (all components of the application are either migrated or not migrated).

[0122] In this embodiment, if some application components cannot be migrated in the preliminary fault migration plan, which means that resources are insufficient, the number of copies of application components with lower priority is reduced in the preliminary fault migration plan, thereby saving some resources; then check whether the remaining resources can include the components that could not be migrated due to insufficient resources into the migration process; if the resources are still insufficient, delete some copies in turn, and so on; if all copies that can be reduced have been reduced, and the saved resources still cannot accommodate all components, then these components cannot be fault migrated.

[0123] Step (8): The management cluster implements failover according to the final failover plan.

[0124] In this embodiment, the fault migration scheduling module implements fault migration according to the final optimized fault migration plan generated by the above steps, and redeploys all applications and components that can be migrated in the fault cluster in the target cluster, thereby completing the fault migration.

[0125] Step (9): After the failover is completed, the management cluster generates a failover report.

[0126] In this embodiment, after the migration of tenants, applications and components is completed according to the fault migration plan, the fault migration scheduling module will generate a fault migration report. Taking tenant A as an example, the form and specific content of the fault migration report are as follows:

[0127] Tenant A:

[0128] Successfully migrated applications: the working cluster where the application is located after the failover (i.e. the working cluster to which the application is migrated), or the working cluster where each component is located.

[0129] Applications that cannot be migrated: Names and components of applications that cannot be migrated.

[0130] In this embodiment, in a multi-cluster environment constructed by multiple kubernetes clusters, the management cluster manages other working clusters and senses the health status of the working clusters in real time; the tenants created by the management cluster occupy a part of the resource quota in each working cluster. When the management cluster senses the failure of a working cluster, it will automatically generate a fault migration plan and implement the migration based on the tenant application migration configuration information and the current status information of each working cluster, so that the application in a multi-cluster, multi-tenant environment can minimize the service downtime and effectively improve the utilization of resources.

[0131] This embodiment proposes a multi-cluster application fault migration method and system supporting multiple tenants, and constructs a multi-cluster environment consisting of a kubernetes management cluster and multiple kubernetes working clusters. In the multi-cluster environment, user applications belong to tenants of the working cluster; when any working cluster fails, the management cluster automatically generates a fault migration plan according to the working cluster state, and according to the remaining status of the tenant's resources, the application is migrated from the failed working cluster to other working clusters that have not failed with tenants, applications, and components as granularity, thereby achieving high application reliability and improving resource utilization. The system is a software system including a multi-cluster state collection module, a fault migration scheduling module, and a tenant management module. It not only supports the migration of applications from a fault cluster to multiple normal clusters, but also supports the application to perform fault migration according to tenants, so as to facilitate resource metering for fault migration. In addition, there is no need to specify the target cluster to be migrated in advance, and the fault migration plan is automatically generated according to the brief configuration and cluster resource information, which is highly flexible; at the same time, the method also supports the idempotent principle. The method and system support both the failure of a working cluster and the simultaneous or sequential failure of multiple working clusters. The processing plan is the same and no special treatment is required. In addition, it also supports migrating application components to different working clusters separately, making full use of the fragmented resources in each working cluster, thereby avoiding the increase in resource application due to overall disaster recovery of the application.

[0132] In an exemplary embodiment, a multi-cluster application failover system supporting multi-tenants is provided. The system may be a computer device, which may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to multi-cluster application fault migration supporting multiple tenants. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-cluster application fault migration method supporting multiple tenants is implemented.

[0133] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0135] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-cluster application failure migration method supporting multi-tenants, characterized in that: The multi-cluster application failure migration method supporting multi-tenants includes: Using kubernetes application software, a multi-cluster environment is created; the multi-cluster environment includes a management cluster and several working clusters, the management cluster is used to manage each of the working clusters, and the working clusters are used to deploy tenants; For each of the working clusters in the multi-cluster environment, a tenant is created in the working cluster; each of the working clusters includes a plurality of tenants, and the tenants are used for resource isolation in a single working cluster scenario and cross-cluster resource isolation in a multi-working cluster scenario; For each of the tenants, deploy an application in the tenant and configure a fault migration policy associated with the application; the application includes multiple components, the fault migration policy includes tenant information and application information, the application information includes application weight and application topology, the application weight is used to indicate the priority of the application during fault migration, and the application topology includes the maximum number of replicas and the minimum number of replicas of each component in the application after fault migration; Generate a resource quota information table according to the fault migration strategy and the status information of the working cluster; the status information of the working cluster includes the application deployment information, application resource occupancy information and the overall occupancy information of the working cluster resources of each tenant in the working cluster, and the resource quota information table includes the total resource information of each working cluster, the resource occupancy information of each tenant, the maximum resource usage, the average resource occupancy value and the working cluster status; A fault migration plan is generated according to the resource quota information table; the fault migration plan is used to provide technical guidance after a certain working cluster fails, so as to migrate the tenants, applications and components that can be migrated in the failed working cluster to other working clusters that have not failed and redeploy them.

2. The multi-cluster application failure migration method supporting multi-tenants according to claim 1, characterized in that: The management cluster is provided with a multi-cluster status collection module, a fault migration scheduling module and a tenant management module; The multi-cluster status collection module is used to configure the access authorization certificate of each working cluster in the management cluster, and obtain the status information and resource information of each working cluster through the API Server of each working cluster; The fault migration scheduling module is used to obtain the status information and resource information of each working cluster from the multi-cluster status collection module, and generate the fault migration plan in real time according to the fault migration strategy, and execute the fault migration of the application; The tenant management module is used to manage tenant information in each of the working clusters.

3. The multi-cluster application failure migration method supporting multi-tenants according to claim 1, characterized in that: For each of the working clusters in the multi-cluster environment, creating a tenant in the working cluster specifically includes: According to the tenant configuration information, the corresponding tenant is created in each of the working clusters; wherein the tenant configuration information includes the tenant name, the total resource quota and the allocation quota, the total resource quota represents the total CPU quota, total memory quota and total storage quota occupied by the tenant in the entire multi-cluster environment, and the allocation quota represents the CPU quota, memory quota and storage quota occupied by the tenant in each of the working clusters; An identification code is generated for each of the tenants, where the identification code is used to identify the identity information of the tenant.

4. The multi-cluster application failure migration method supporting multi-tenants according to claim 3 is characterized in that: The identification code is a UUID code.

5. The multi-cluster application failure migration method supporting multi-tenants according to claim 1, characterized in that: For each of the tenants, deploy an application in the tenant and configure a fault migration strategy associated with the application, specifically including: Using an automatic deployment tool, deploy applications to each tenant respectively to obtain tenant information and application information corresponding to each tenant; The fault migration strategy is determined according to the tenant information and application information corresponding to each tenant.

6. The multi-cluster application failure migration method supporting multi-tenants according to claim 5, characterized in that: The automatic deployment tool is Argocd.

7. The multi-cluster application failure migration method supporting multi-tenants according to claim 1, characterized in that: Generate a fault migration plan based on the resource quota information table, specifically including: Generate a preliminary fault migration plan according to the resource quota information table; The preliminary fault migration plan is optimized to obtain an optimized fault migration plan; the optimized fault migration plan is used as the finally generated fault migration plan.

8. The multi-cluster application failure migration method supporting multi-tenants according to claim 7, characterized in that: The preliminary fault migration plan is optimized to obtain an optimized fault migration plan, which specifically includes: According to the application weights corresponding to the respective applications, determining the application with the smallest application weight from among all the applications of the tenant that have satisfied the migration condition; For each component in the application with the smallest application weight, when the maximum number of copies of a component is greater than 1, the number of copies of the component in the preliminary fault migration plan is reduced by 1, and the number of reducible copies of all components in all applications is reduced in this manner until the components that do not meet the migration conditions are included in the preliminary fault migration plan; When all reducible copies have been reduced and there are still applications and their components that do not meet the migration conditions, the applications and their components that do not meet the migration conditions are marked as types that cannot be migrated, and other components of the application that already meet the migration conditions are deleted from the preliminary fault migration plan to obtain the optimized fault migration plan.

9. The multi-cluster application failure migration method supporting multi-tenants according to claim 7, characterized in that: After the step of optimizing the preliminary fault migration scheme to obtain an optimized fault migration scheme, the multi-cluster application fault migration method supporting multiple tenants further includes: After a failure occurs in a certain working cluster, the tenants, applications and components that can be migrated in the failed working cluster are failed over according to the optimized failure migration plan, and a failure migration report is generated; the failure migration report includes the migration-in working cluster information and the migration-out working cluster information of the successfully migrated applications in each tenant in the failed working cluster, as well as the names and components of the applications that cannot be migrated.

10. A multi-cluster application fault migration system supporting multiple tenants, the multi-cluster application fault migration system supporting multiple tenants comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the multi-cluster application failure migration method supporting multiple tenants according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Control method for cloud computing resource scheduling problem based on multi-tenant theory

    CN114666333A

  • Data migration method and device for distributed cache system

    CN115292293A

  • Full-link implementation method for cross-cluster application migration

    CN116866159A

  • Cross-cluster application migration method and device, cross-cluster application recovery method and device and cluster system

    CN116954870A

  • Managing user data in a multitenant deployment

    US20190095241A1

Cited By

  • Multi-cluster application failover method and system supporting multiple tenants

    WO2026040817A1