Method, device and system for scheduling container resources in multi-container cluster

Abstract

The application relates to a container resource scheduling method, device and system in a multi-container cluster, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a resource scheduling request, determining a resource scheduling strategy, wherein the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster and a third identifier of a container resource to be scheduled; generating a resource scheduling instruction according to the resource scheduling strategy; and sending the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier. The method uses a container management platform to realize the scheduling of the container resource in the multi-container cluster, and can improve the stability of the container resource scheduling in the multi-container cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource scheduling, and in particular to a container resource scheduling method, device and system in a multi-container cluster, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] With the popularity of containers and the increasing maturity of container clusters (such as Kubernetes), more and more organizations are migrating their infrastructure to container clusters. Due to the needs of network isolation, scalability, multi-cloud and other scenarios, it is quite common for enterprises to run multiple container clusters internally. Resource allocation and scheduling in a multi-container cluster can significantly improve service availability, and containers can not only be run across regions and across regions, but also can continue to run in other normal regions when a fault occurs in a certain region.

[0003] In the prior art, a container resource scheduling method in a multi-container cluster needs to first deploy a control component in a container cluster to upgrade it to a control cluster, and then use a command to incorporate other clusters, and the control cluster is used as an entry to uniformly complete the scheduling of container resources in the multi-container cluster.

[0004] However, in the prior art, the container cluster is converted into a control cluster, and the original functions in the container cluster and the resource scheduling function may not be fully compatible, thereby causing adverse effects and leading to low stability of container resource scheduling in a multi-container cluster. SUMMARY

[0005] Therefore, it is necessary to provide a container resource scheduling method, device and system in a multi-container cluster, a computer device, a computer readable storage medium and a computer program product, which can improve the stability of container resource scheduling in a multi-container cluster.

[0006] In a first aspect, the present application provides a container resource scheduling method in a multi-container cluster. The method is applied to a container management platform, and the method comprises:

[0007] In response to a resource scheduling request, a resource scheduling strategy is determined, wherein the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster and a third identifier of a container resource to be scheduled;

[0008] According to the resource scheduling strategy, a resource scheduling instruction is generated;

[0009] The resource scheduling instruction is sent to the resource sending container cluster corresponding to the first identifier, to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0010] In one of the embodiments, the generating of the resource scheduling instruction according to the resource scheduling strategy comprises:

[0011] obtaining a template YAML configuration file, wherein the template YAML configuration file contains standard resource scheduling strategy data;

[0012] comparing the resource scheduling strategy with the standard resource scheduling strategy data in the template YAML configuration file to determine policy differentiation data;

[0013] adjusting the standard resource scheduling strategy data in the template YAML configuration file according to the policy differentiation data to obtain a cluster YAML configuration file;

[0014] generating a resource scheduling instruction carrying the cluster YAML configuration file.

[0015] In one of the embodiments, the determining of the resource scheduling strategy in response to the resource scheduling request comprises:

[0016] obtaining resource requirement information and container resource usage information of a container cluster in response to a resource scheduling request;

[0017] determining a resource scheduling strategy according to the resource requirement information and the container resource usage information of the container cluster.

[0018] In one of the embodiments, the method further comprises:

[0019] controlling the container cluster managed by the container management platform through an API Server component in the container cluster.

[0020] In one of the embodiments, the controlling of the container cluster managed by the container management platform through the API Server component in the container cluster comprises:

[0021] sending a state acquisition request to the container cluster managed by the container management platform through the API Server component in the container cluster;

[0022] receiving state data returned by the managed container cluster in response to the state acquisition request;

[0023] The controlling of the container cluster managed by the container management platform through the API Server component in the container cluster further comprises:

[0024] allocating an authentication token to the container cluster managed by the container management platform through the API Server component in the container cluster, wherein the authentication token is used to indicate the permission of the container cluster and the container resource.

[0025] The controlling, by the API Server component in the container cluster, the container cluster managed by the container management platform further includes:

[0026] The API Server component in the container cluster sends a container debugging command to the container cluster managed by the container management platform, wherein the container debugging command is used to debug the container in the managed container cluster.

[0027] In one of the embodiments, the container cluster managed by the container management platform adopts a Mac VLAN mode for networking.

[0028] The method further includes:

[0029] In response to the container IP processing instruction, the IPAM Server component processes the container IP to be processed.

[0030] In a second aspect, the application further provides a container resource scheduling device in a multi-container cluster. The device is used in a container management platform, and the device includes:

[0031] A policy determination module is configured to determine a resource scheduling policy in response to a resource scheduling request, wherein the resource scheduling policy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled.

[0032] An instruction generation module is configured to generate a resource scheduling instruction according to the resource scheduling policy.

[0033] An instruction sending module is configured to send the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0034] In a third aspect, the application further provides a container resource scheduling system in a multi-container cluster. The system includes a container management platform, a resource sending container cluster, and a resource receiving container cluster, wherein:

[0035] The container management platform is configured to determine a resource scheduling policy in response to a resource scheduling request, wherein the resource scheduling policy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled.

[0036] The container management platform is further configured to generate a resource scheduling instruction according to the resource scheduling strategy, and send the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier.

[0037] The resource sending container cluster is configured to read the container resource to be scheduled corresponding to the third identifier according to the resource scheduling instruction, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0038] In a fourth aspect, the present application further provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0039] In response to a resource scheduling request, a resource scheduling strategy is determined, wherein the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled;

[0040] A resource scheduling instruction is generated according to the resource scheduling strategy;

[0041] The resource scheduling instruction is sent to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0042] In a fifth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0043] In response to a resource scheduling request, a resource scheduling strategy is determined, wherein the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled;

[0044] A resource scheduling instruction is generated according to the resource scheduling strategy;

[0045] The resource scheduling instruction is sent to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0046] In a sixth aspect, the present application further provides a computer program product. The computer program product comprises a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0047] determining a resource scheduling strategy in response to the resource scheduling request, wherein the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled;

[0048] generating a resource scheduling instruction according to the resource scheduling strategy;

[0049] sending the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0050] The container resource scheduling method, device, system, computer device, storage medium, and computer program product provided in the above multiple container cluster can process a resource scheduling request by using a container management platform, determine a resource scheduling strategy, and then generate a resource scheduling instruction and send it to a resource sending container cluster, to instruct the resource sending container cluster to send a container resource to be scheduled to a resource receiving container cluster. It can be understood that the present application uses a container management platform to implement container resource scheduling in a multiple container cluster, without converting the container cluster into a control cluster. Therefore, in the container resource scheduling process, there is no adverse effect caused by the incompatibility between the original functions in the container cluster and the resource scheduling function, which is beneficial to improving the stability of container resource scheduling in a multiple container cluster. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 An application environment diagram of the container resource scheduling method in a multiple container cluster in one embodiment;

[0052] Figure 2 A flowchart of the container resource scheduling method in a multiple container cluster in one embodiment;

[0053] Figure 3 A structure block diagram of the container resource scheduling device in one embodiment;

[0054] Figure 4 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0056] The container resource scheduling method in a multiple container cluster provided in the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the container management platform 102 determines the resource scheduling strategy in response to the resource scheduling request, wherein the resource scheduling strategy includes the first identification of the resource sending container cluster 104, the second identification of the resource receiving container cluster 106 and the third identification of the container resource to be scheduled, generates the resource scheduling instruction according to the resource scheduling strategy, and sends the resource scheduling instruction to the resource sending container cluster 104 corresponding to the first identification to instruct the resource sending container cluster 104 to read the container resource to be scheduled corresponding to the third identification and send the container resource to be scheduled to the resource receiving container cluster 106 corresponding to the second identification. Wherein, the container management platform 102 can be a cloud platform, a server or a server cluster.

[0057] In one embodiment, as shown in Figure 2 , a container resource scheduling method in a multi-container cluster is provided. The method is applied to the container management platform 102 in Figure 1 for example, which includes the following steps:

[0058] Step S202, in response to the resource scheduling request, determine the resource scheduling strategy.

[0059] Among them, the resource scheduling strategy includes the first identification of the resource sending container cluster, the second identification of the resource receiving container cluster and the third identification of the container resource to be scheduled. The resource sending container cluster refers to the container cluster for sending the scheduled container resource. The resource receiving container cluster refers to the container cluster for receiving the scheduled container resource. Usually, the container resource to be scheduled is stored in the resource sending container cluster.

[0060] Specifically, the resource scheduling request can be a request issued by the terminal, or a resource scheduling request automatically generated by the container management platform when the preset condition is met. The container management platform determines the resource scheduling strategy based on the resource scheduling request.

[0061] Step S204, according to the resource scheduling strategy, generate the resource scheduling instruction.

[0062] Specifically, the container management platform generates the resource scheduling instruction carrying the resource scheduling strategy.

[0063] Step S206, send the resource scheduling instruction to the resource sending container cluster corresponding to the first identification to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identification and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identification.

[0064] Specifically, the container management platform sends a resource scheduling instruction to the resource sending container cluster corresponding to the first identifier. The resource sending container cluster parses the resource scheduling instruction to obtain a resource scheduling strategy, and sends the container resource to be scheduled to the resource receiving container cluster according to the resource scheduling strategy.

[0065] In the method for scheduling container resources in the multi-container cluster, the container management platform processes the resource scheduling request, determines the resource scheduling strategy, and then generates and sends a resource scheduling instruction to the resource sending container cluster to instruct the resource sending container cluster to send the container resource to be scheduled to the resource receiving container cluster. It can be understood that the method uses the container management platform to realize the scheduling of container resources in the multi-container cluster, without converting the container cluster into a control cluster. Therefore, in the container resource scheduling process, there is no adverse effect caused by the incompatibility between the original functions in the container cluster and the resource scheduling function, which is beneficial to improve the stability of the container resource scheduling in the multi-container cluster.

[0066] Optionally, in an embodiment, the resource scheduling strategy includes the identifier of the container cluster where the application is deployed and the number of replicas of each container cluster.

[0067] In an embodiment, a possible implementation of the step S202 "determining a resource scheduling strategy in response to a resource scheduling request" is involved. Based on the above embodiment, the step S202 can be implemented by the following steps:

[0068] In step S2022, resource requirement information and container resource usage information of the container cluster are obtained in response to the resource scheduling request.

[0069] In step S2024, the resource scheduling strategy is determined according to the resource requirement information and the container resource usage information of the container cluster.

[0070] The resource requirement information is provided by the business party, and can be specific high-SSD requirement information, CPU-intensive computing requirement information, specific GPU requirement information, and off-site dual-active requirement information, etc. The container resource usage information can be the resource usage in a certain time period, such as the average CPU memory usage in a week, the peak CPU memory usage in a week, etc.

[0071] Specifically, the multi-cluster container scheduling module in the container management platform obtains resource requirement information and container resource usage information of the container cluster in response to a resource scheduling request, and then determines a resource scheduling strategy according to the resource requirement information and the container resource usage information of the container cluster. Optionally, the multi-cluster container scheduling module determines the resource scheduling strategy according to the resource requirement information, the container resource usage information of the container cluster, and the application profile.

[0072] In one embodiment, the application portrait construction process is: the container management platform first acquires resource requirement information submitted by the business party and container resource usage information of the container cluster in a certain time period for different types of business containers, and obtains an application portrait through data analysis on these data by a modeling algorithm. In a specific example, the container management platform collects CPU, memory and other indicators of the application program in each time period of each day and each week, as well as business party information provided by each system (including system attributes, whether it is an A-class system; business information, whether it is a customer system; disaster recovery information, whether it is a multi-active architecture in different places, etc.). By assigning corresponding weights to different conditions, the above multi-dimensional data is iteratively calculated using a weighted calculation method, and the entire process of the output layer can be considered as a model. Finally, the application portrait, i.e., the transaction characteristics and business trends of the application, is obtained.

[0073] In one embodiment, a possible implementation of the above step S204 "generating a resource scheduling instruction according to the resource scheduling strategy" is involved. On the basis of the above embodiment, step S204 can be implemented through the following steps:

[0074] Step S2042, acquiring a template YAML configuration file;

[0075] Step S2044, comparing the resource scheduling strategy with standard resource scheduling strategy data in the template YAML configuration file to determine policy differentiation data;

[0076] Step S2046, adjusting the standard resource scheduling strategy data in the template YAML configuration file according to the policy differentiation data to obtain a cluster YAML configuration file;

[0077] Step S2048, generating a resource scheduling instruction carrying the cluster YAML configuration file.

[0078] The template YAML configuration file contains standard resource scheduling strategy data. The standard resource scheduling strategy data is a collection of data configured in different deployment clusters

[0079] Specifically, for the containers with the default resource scheduling strategy, the multi-cluster split and remainder scheduling strategy is adopted, and the containers are deployed in the multi-container cluster according to the average value of the number of replicas, so that each container cluster can provide equal capability of external services. For the containers with the defined custom resource scheduling strategy, the scheduling strategy based on the CPU and memory usage of the cluster, the custom resource scheduling strategy based on the application profile and the cluster profile can be supported. After the user selects the custom resource scheduling strategy, the extraClusterConfig file is generated and stored in the database. When the actual application is deployed and scaled, before the conversion from the template YAML configuration file to the cluster YAML configuration file, the policy differentiation data is determined based on the extraClusterConfig file and the policy differentiation data is overlaid (for example, adding the policy data in the resource scheduling strategy but not in the template YAML configuration file, and deleting the policy data in the template YAML configuration file but not in the resource scheduling strategy), so as to make the custom resource scheduling strategy effective. In addition, the process will perform related checks, so as to ensure that the related fields that are not allowed to be published are intercepted, and the container image path is determined to be available.

[0080] In the embodiment, by developing scheduling outside the multi-container cluster, the multi-cluster container scheduling module of the container management platform can more flexibly and more strategically perform cross-cluster scheduling of container resources. Moreover, when a container cluster in a certain region is down, the cluster information can be synchronized to the container management platform by the application synchronization module, and the multi-cluster container scheduling module can schedule the down container resources to other normal container clusters, thereby improving the availability of the containers.

[0081] In one embodiment, the method further comprises the following steps:

[0082] In step S212, the container cluster managed by the container management platform is controlled through the API Server component in the container cluster.

[0083] Specifically, the container management platform comprises a multi-cluster management module. The multi-cluster management module can perform management operation on the container clusters in the Overlay environment on the cloud and the Underlay environment off the cloud. The container management platform can control the container cluster managed by the container management platform through the API Server component in the container cluster.

[0084] In this embodiment, since the container management platform uses the API Server component for control operation, the corresponding component agent does not need to be deployed invasively to complete the related deployment and scheduling operation, thereby reducing the complexity of container resource scheduling. Moreover, through the compatibility of the overall API Server component, the container management platform can improve the aggregation and use of the container cluster interface.

[0085] In one embodiment, a possible implementation of the above step S212 "controlling the container cluster managed by the container management platform through the API Server component in the container cluster" is involved. On the basis of the above embodiment, step S212 can be implemented through the following steps:

[0086] Step S2122, sending a state acquisition request to the container cluster managed by the container management platform through the API Server component in the container cluster;

[0087] Step S2124, receiving state data returned by the managed container cluster in response to the state acquisition request.

[0088] The API Server component is used to provide interfaces for adding, deleting, modifying and inquiring various resource objects of the container cluster. That is, the API Server component provides monitoring of the state of each resource object.

[0089] Specifically, the container management platform includes an application synchronization module and a database. The application synchronization module is implemented based on the Informer mechanism. The application synchronization module establishes a TCP long connection with the API Server component. The application synchronization module sends a state acquisition request to the API Server component in the container cluster managed by the container management platform. The API Server component in the managed container cluster returns state data (for example, an Event used to show the situation occurring in the container cluster) to the container management platform in response to the state acquisition request, for example, the change state information of each resource object in the container cluster. After processing the state data (including adding, updating or discarding the Event data, and updating or saving all or part of the data in the Event), the container management platform stores it in the database (for example, an ETCD database) to provide real-time updated state data.

[0090] In this embodiment, the ETCD load pressure caused by directly calling the API Server component to query in the case of large concurrency can be effectively reduced, and the availability of the API Server component is improved.

[0091] In another embodiment, a possible implementation of the above step S212 "controlling the container cluster managed by the container management platform through the API Server component in the container cluster" is involved. Based on the above embodiment, step S212 can be implemented through the following steps:

[0092] Step S2126, assigning an authentication token to the container cluster managed by the container management platform through the API Server component in the container cluster.

[0093] The authentication token (Token) is used to indicate the permission of the container cluster and the container resource. Optionally, the authentication token is generated by a ServiceAccount.

[0094] Specifically, the multi-cluster management module in the container management platform generates authentication tokens with different permissions in the container cluster through the API Server component in the container cluster, thereby providing accurate permission control granularity for different container clusters and achieving the purpose of differentiated permission control for different container clusters and different container resources.

[0095] In another embodiment, a possible implementation of the above step S212 "controlling the container cluster managed by the container management platform through the API Server component in the container cluster" is involved. Based on the above embodiment, step S212 can be implemented through the following steps:

[0096] Step S2128, sending a container debugging command to the container cluster managed by the container management platform through the API Server component in the container cluster.

[0097] The container debugging command is used to debug the container in the managed container cluster.

[0098] In one embodiment, the container cluster managed by the container management platform adopts a Mac VLAN (Virtual Local Area Network) networking mode. Optionally, the Underlay cluster under the container management platform adopts the Mac VLAN networking mode. With this networking mode, the network accessed by the container and the network accessed by the cluster host can be in the same plane. Based on this, in one embodiment, the method further includes the following steps:

[0099] Step S222, processing the container IP to be processed through the IPAM Server component in response to the container IP processing instruction.

[0100] The IPAM (IP Address Management) Server component can be installed in the container management platform or as a network plug-in of the container cluster, and is mainly used to implement IP allocation and recycling of macvaln mode containers.

[0101] Specifically, the container management platform includes an IP management module. The container management platform supports interface operation and can provide functions such as adding, deleting, and binding IP address pools of Deployment of available IP addresses in the container cluster address pool. The user operates (for example, clicks) the interface elements (for example, container IP adding controls, container IP deleting buttons, etc.) displayed in the container management platform, the IP management module generates corresponding container IP processing instructions, and processes the container IP to be processed through the IPAM Server component according to the container IP processing instructions. The use of the highly available IPAM Server component enables the container management platform to have stable container IP management functions. In addition, a variety of IP affinity policies are also configured in the IP management module.

[0102] One embodiment of the present application will be described below in conjunction with a specific Kubernetes cluster scenario. Specifically, the multi-Kubernetes cluster container resource scheduling device is centered on a container cloud management platform, which includes a multi-cluster management module, a multi-cluster container scheduling module, an application deployment module, an application synchronization module, and an IP management module. In order to facilitate user use and external system integration, a front-end UI portal and an API gateway are provided for external users. Among them:

[0103] The multi-cluster management module is the core running component of the container cloud management platform, and is used to implement abstract management of multi-cluster container resources.

[0104] Specifically, the multi-cluster management module can manage Kubernetes clusters in Overlay environments on the cloud and Underlay environments off the cloud. The multi-cluster management module can provide precise permission control granularity to different clusters by generating different permission tokens in the cluster. Moreover, the container cloud management platform completely uses the Kubernetes ApiServer component for control operation, without the need for invasive deployment and corresponding component agents to complete related deployment and scheduling operations.

[0105] The multi-cluster container scheduling module provides a scheduling mode in which containers are scheduled to multiple clusters according to multiple strategies, and supports custom policy definition.

[0106] Specifically, the multi-cluster container scheduling module first uses an external Java program to implement more functions from the perspectives of scalability and custom policies, instead of using the Scheduler of Kuubernetes. For different types of business containers, the application portrait is obtained through data analysis by modeling algorithms based on the information submitted by the business side (such as specific high-SSD requirements, CPU-intensive computing requirements, specific GPU requirements, and off-site dual-active requirements) and the actual resource usage in a certain period obtained from the cluster (such as weekly average CPU memory usage, weekly peak CPU memory usage, etc.). On this basis, the default policy container adopts a multi-cluster division and remainder scheduling policy to ensure that the application multi-Kubernetes cluster is homogeneous to provide services externally. For containers with defined custom scheduling policies, scheduling policies based on cluster CPU and memory usage, custom scheduling policies based on application portrait and cluster portrait can be supported. After the user selects the custom policy, the extraClusterConfig file is generated and stored in the database. When the actual application is deployed, scaled, or shrunk, the custom policy is covered based on the extraClusterConfig file before the template Yaml is converted to the cluster Yaml, so that the custom policy takes effect. The process will also perform related checks to ensure that related fields that are not allowed to be published are intercepted, and the container image path is definitely available, etc.

[0107] The application deployment module processes the input submitted by the user and processes the user's request to publish to the corresponding Kubernetes cluster.

[0108] Specifically, the application deployment module is directly provided to the user and interfaces with each DevOps pipeline to manage the full life cycle of container deployment, issuance, start-stop, and deletion. The user deploys resources by uploading a standardized resource object YAML file, and the platform provides YAML standardization checks, YAML historical version archiving, and YAML configuration transcription templates. After the application submits the YAML file, the container is scheduled to different clusters for startup by the multi-cluster scheduling module. The platform also provides elastic scaling, image updating, and application job calling for application class resources.

[0109] More specifically, the elastic scaling capability, i.e., based on certain rules such as CPU, memory, TPS, API interface call, etc. to elastically expand the container instance, when the traffic increases, the number of backend container replicas can be automatically increased. The elastic scaling capability of Kubernetes itself can only be based on CPU and memory. The container cloud management platform extends the native capability of Kubernetes, can call external rule modules for related parameter transmission, and after improvement, can use elastic scaling capability based on TPS, API interface, JVM parameter, fixed time frequency, etc. Indicators can support more scenarios.

[0110] The application job is called because the container itself does not necessarily provide the ability of ssh remote login, etc. The debugging of the container has always been very inconvenient. The container cloud management platform encapsulates the commonly used debugging commands externally, and allows users to customize the commands to be executed in the container. After the user creates an application job, the platform provides operations on the container by calling the corresponding interface of the API server component, and returns the corresponding execution result to the user by the container cloud management platform.

[0111] The application synchronization module synchronizes all managed cluster data to the platform database, and provides real-time data information storage and query functions.

[0112] Specifically, the application synchronization module is implemented based on the Informer mechanism of Kubernetes underlying, and the long chain is established with the API server to obtain the change state of each resource object in the cluster. The event is sent from the cluster side to the platform side, and after the platform data processing, it is stored in the database to provide real-time update state data. Directly calling the API server query in the case of large concurrency effectively solves the ETCD pressure, and improves the availability of the API server.

[0113] The IP management module provides IP reservation, IP binding and other functions for multi-cluster containers, which is convenient for traditional application container migration work.

[0114] Specifically, the Underlay cluster under the jurisdiction of the container cloud management platform adopts Mac VLAN mode networking, which makes the network accessed by the container and the network accessed by the cluster host in the same plane. The platform IP management platform supports interface operation, provides functions such as adding and deleting available IP addresses in the cluster address pool, and Deployment binding IP address pool. And supports multiple IP affinity strategies. The highly available IPAM Server end makes the platform have stable Kubernetes IP management function.

[0115] In the embodiments of the present application, taking multi-Kubernetes cluster management as the starting point, a multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) is constructed by using a management plane high-availability scheme and a non-invasive way to Kubernetes cluster, the design of the platform adopts a new functional distribution design. The multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) provides a container multi-cluster resource scheduling scheme, containers can be scheduled to the most idle cluster according to the cluster CPU and memory usage, and supports custom container scheduling strategies in special situations. The multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) provides container elasticity expansion and contraction capability, can be elastically scaled as needed, and on the basis of only supporting container CPU and memory indicators by default, provides custom indicators based on SQL, MQ, TPS, Metrics, etc. The container scheduling management function of the multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) is non-invasive to the Kubernetes cluster, and does not require the installation of a separate agent. The multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) adopts a real-time dynamic updating mechanism, based on the Informer mechanism of Kubernetes, synchronizes data to the management plane in real time, ensures that the API of each Kubernetes cluster will not be called abnormally, and ensures the independent operation of the Kubernetes cluster.

[0116] The multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) adopts a high-availability, non-invasive multi-Kubernetes cluster management method, and the availability is more stable than the control plane single cluster of open source schemes Kubefed and Karmada. The non-invasive control also reduces the component call chain and improves performance. The platform provides container resource scheduling capability and custom elasticity scaling capability, which can improve the container resource usage. At the same time, the platform provides a unified container monitoring, alarm, log, and gateway system, which can manage the container life cycle on the platform uniformly, and is convenient for interfacing with other related systems, and has the characteristics of high cohesion and loose coupling. And from the perspective of the user, the multi-Kubernetes cluster container scheduling device (enterprise-level container cloud management platform) provides a user-friendly, simple, and easy-to-use front-end portal.

[0117] The container cloud management platform discards the mainstream Kube On Kube mode, adopts an external management plane mode, retains a traditional multi-active database, and thus guarantees the availability of the management plane. In a multi-active scenario, any Kubernetes cluster outage will not affect other Kubernetes clusters. The single control plane exception can also be guaranteed by the multi-active control plane to ensure that the service will not be interrupted. The container cloud management platform currently provides 7x24 hours of uninterrupted service, with a time availability of 99.95%. The RPO of the platform is 2 hours, and the RTO is 2 hours.

[0118] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiments of the present application also provide a container resource scheduling device in a multi-container cluster for implementing the above-mentioned container resource scheduling method in a multi-container cluster. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more container resource scheduling device embodiments in a multi-container cluster provided below can refer to the limitations of the container resource scheduling method in a multi-container cluster in the above text, which will not be repeated here.

[0120] In one embodiment, as shown in Figure 3 A container resource scheduling device in a multi-container cluster is provided, which is used in a container management platform and includes a policy determination module 302, an instruction generation module 304, and an instruction sending module 306, wherein:

[0121] The policy determination module 302 is configured to determine a resource scheduling policy in response to a resource scheduling request, wherein the resource scheduling policy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled.

[0122] The instruction generation module 304 is configured to generate a resource scheduling instruction according to the resource scheduling policy.

[0123] The instruction sending module 306 is configured to send the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the to-be-scheduled container resource corresponding to the third identifier, and send the to-be-scheduled container resource to the resource receiving container cluster corresponding to the second identifier.

[0124] In the above device for scheduling container resources in a multi-container cluster, the container management platform is used to process a resource scheduling request, determine a resource scheduling strategy, and then generate a resource scheduling instruction and send the resource scheduling instruction to a resource sending container cluster, so as to instruct the resource sending container cluster to send a to-be-scheduled container resource to a resource receiving container cluster. It can be understood that the device uses the container management platform to implement the scheduling of container resources in the multi-container cluster, without the need to convert the container cluster into a control cluster. Therefore, in the process of scheduling the container resources, no adverse effects caused by the incompatibility between the original functions in the container cluster and the resource scheduling function can occur, and the stability of the scheduling of the container resources in the multi-container cluster can be improved.

[0125] In one embodiment, the instruction generation module 304 is specifically configured to obtain a template YAML configuration file, wherein the template YAML configuration file contains standard resource scheduling strategy data; compare the resource scheduling strategy with the standard resource scheduling strategy data in the template YAML configuration file to determine strategy differentiation data; adjust the standard resource scheduling strategy data in the template YAML configuration file according to the strategy differentiation data to obtain a cluster YAML configuration file; and generate a resource scheduling instruction carrying the cluster YAML configuration file.

[0126] In one embodiment, the strategy determination module 302 is specifically configured to, in response to a resource scheduling request, obtain resource requirement information and container resource usage information of a container cluster; and determine a resource scheduling strategy according to the resource requirement information and the container resource usage information of the container cluster.

[0127] In one embodiment, the device further includes a cluster control module configured to send a state acquisition request to a container cluster managed by the container management platform through an API Server component in the container cluster; receive state data returned by the managed container cluster in response to the state acquisition request; assign an authentication token to the container cluster managed by the container management platform through the API Server component in the container cluster, wherein the authentication token is used to indicate the permissions of the container cluster and the container resources; and send a container debugging command to the container cluster managed by the container management platform through the API Server component in the container cluster, wherein the container debugging command is used to debug the container in the managed container cluster.

[0128] The modules in the container resource scheduling apparatus in the multi-container cluster can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform operations corresponding to the modules.

[0129] In one embodiment, a computer device, which can be a server, has an internal structure as shown in Figure 4 The computer device includes a processor, a memory, and a network interface connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a method for scheduling container resources in a multi-container cluster.

[0130] Those skilled in the art can understand that Figure 4 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not limit the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps:

[0132] In response to the resource scheduling request, a resource scheduling strategy is determined. The resource scheduling strategy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled.

[0133] According to the resource scheduling strategy, a resource scheduling instruction is generated.

[0134] The resource scheduling instruction is sent to the resource sending container cluster corresponding to the first identifier, to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

[0135] In one embodiment, the processor executes the computer program to further implement the following steps:

[0136] Obtaining a template YAML configuration file, wherein the template YAML configuration file contains standard resource scheduling policy data; comparing the resource scheduling policy with the standard resource scheduling policy data in the template YAML configuration file to determine policy differentiation data; adjusting the standard resource scheduling policy data in the template YAML configuration file according to the policy differentiation data to obtain a cluster YAML configuration file; and generating a resource scheduling instruction carrying the cluster YAML configuration file.

[0137] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0138] In response to the resource scheduling request, obtaining resource requirement information and container resource usage information of the container cluster; and determining a resource scheduling policy according to the resource requirement information and the container resource usage information of the container cluster.

[0139] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0140] Controlling the container cluster managed by the container management platform through an API Server component in the container cluster.

[0141] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0142] In response to the container IP processing instruction, processing the container IP to be processed through an IPAM Server component.

[0143] In one embodiment, a computer readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0144] In one embodiment, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0145] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0147] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0148] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for scheduling container resources in a multi-container cluster, characterized in that, The method is applied to a container management platform, and the method comprises: In response to a resource scheduling request, obtaining resource requirement information and container resource usage information of a container cluster; According to the resource requirement information, the container resource usage information and an application profile, determining a resource scheduling strategy; wherein the application profile is determined according to CPU and memory occupied by an application program in each time period and business party information corresponding to the application program; the business party information comprises system attributes, business information and disaster recovery information; the resource scheduling strategy comprises a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster and a third identifier of a container resource to be scheduled; According to the resource scheduling strategy, generating a resource scheduling instruction; Sending the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

2. The method of claim 1, wherein, The method further comprises: Obtaining a template YAML configuration file, wherein the template YAML configuration file contains standard resource scheduling strategy data; Comparing the resource scheduling strategy with the standard resource scheduling strategy data in the template YAML configuration file to determine strategy differentiation data; According to the strategy differentiation data, adjusting the standard resource scheduling strategy data in the template YAML configuration file to obtain a cluster YAML configuration file; Generating a resource scheduling instruction carrying the cluster YAML configuration file.

3. The method of claim 1, wherein, The method further comprises: Controlling the container cluster managed by the container management platform through an API Server component in the container cluster.

4. The method of claim 3, wherein, The method further comprises: Sending a state acquisition request to the container cluster managed by the container management platform through the API Server component in the container cluster; and receiving state data returned by the managed container cluster in response to the state acquisition request; Alternatively, Allocating an authentication token to the container cluster managed by the container management platform through the API Server component in the container cluster, wherein the authentication token is used to indicate the permissions of the container cluster and the container resource; Alternatively, Sending a container debugging command to the container cluster managed by the container management platform through the API Server component in the container cluster, wherein the container debugging command is used to debug the container in the managed container cluster.

5. The method of claim 1, wherein, The container cluster managed by the container management platform adopts a Mac VLAN mode for networking. The method further comprises: In response to a container IP processing instruction, processing a container IP to be processed through an IPAM Server component.

6. A device for scheduling container resources in a multi-container cluster, characterized in that, The device is used in a container management platform, and the device comprises: The policy determination module is configured to, in response to a resource scheduling request, acquire resource requirement information and container resource usage information of a container cluster, and determine a resource scheduling policy according to the resource requirement information, the container resource usage information, and an application profile; the application profile is determined according to CPU and memory occupied by an application program in each time period and business party information corresponding to the application program; the business party information includes system attributes, business information, and disaster recovery information; the resource scheduling policy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled. The instruction generation module is configured to generate a resource scheduling instruction according to the resource scheduling policy. The instruction sending module is configured to send the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier, so as to instruct the resource sending container cluster to read the container resource to be scheduled corresponding to the third identifier, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier. 7.A system for scheduling container resources in a multi-container cluster, characterized in that, The system includes a container management platform, a resource sending container cluster, and a resource receiving container cluster. The container management platform is configured to, in response to a resource scheduling request, acquire resource requirement information and container resource usage information of a container cluster, and determine a resource scheduling policy according to the resource requirement information, the container resource usage information, and an application profile; the application profile is determined according to CPU and memory occupied by an application program in each time period and business party information corresponding to the application program; the business party information includes system attributes, business information, and disaster recovery information; the resource scheduling policy includes a first identifier of a resource sending container cluster, a second identifier of a resource receiving container cluster, and a third identifier of a container resource to be scheduled. The container management platform is further configured to generate a resource scheduling instruction according to the resource scheduling policy, and send the resource scheduling instruction to the resource sending container cluster corresponding to the first identifier. The resource sending container cluster is configured to read the container resource to be scheduled corresponding to the third identifier according to the resource scheduling instruction, and send the container resource to be scheduled to the resource receiving container cluster corresponding to the second identifier.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images