Resource management method and system
By deploying the master resource controller and slave resource controller in the control cluster and business cluster, the resource scheduling is dispersed, the problem of high pressure on the control cluster scheduling is solved, and the resource scheduling efficiency is improved.
Patent Information
- Application Number
- CN202211084818.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-06
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2042-09-06
AI Technical Summary
In the prior art, the control cluster is under great scheduling pressure due to the management of multiple service clusters and a large number of container applications, and the resource scheduling efficiency is low, and it may even cause the control cluster to go down.
A resource management system is proposed. By deploying the main resource controller in the control cluster and the slave resource controller in the service cluster, the main resource controller is responsible for listening to resource declaration information and sending deployment instructions to the target service cluster, and the target slave resource controller listens to resource description information and performs resource scheduling.
It effectively reduces the scheduling pressure of the resource controller on the control cluster, distributes the pressure of resource scheduling to the resource controller on each business cluster, improves the resource scheduling efficiency, and reduces the monitoring work pressure of the main resource controller.
Smart Images

Figure CN115567523B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of container cluster technology, and in particular to a resource management system. The present application also relates to a resource management method and a resource management device, a computing device, and a computer-readable storage medium. Background Art
[0002] With the rapid development of cloud computing technology, container technology has gradually become a hot spot in the industry. All mainstream cloud computing platforms (cloud container platforms) provide container servers. At present, cloud computing platforms need to support multiple services, and different services need to be implemented by different business clusters, that is, cloud computing platforms need to deploy multiple business clusters. In actual applications, a control cluster will be deployed to manage all business clusters, and the business cluster only needs to focus on business applications. However, with the increase in the number of business clusters and the increase in the number of containers and applications in the business clusters, the scale of business clusters that the control cluster needs to manage continues to expand, and the number of resources that need to be scheduled continues to increase, which greatly increases the scheduling pressure of the control cluster, resulting in low resource scheduling efficiency and causing the control cluster to crash. Therefore, how to reduce the scheduling pressure of the control cluster and improve the efficiency of resource scheduling is an urgent problem to be solved. Summary of the invention
[0003] In view of this, an embodiment of the present application provides a resource management system. The present application also relates to a resource management method, a resource management device, a computing device, and a computer-readable storage medium to solve the problem in the prior art that the controller on the control cluster has a large scheduling pressure on the control cluster due to the expansion of the service cluster and its resource scale, resulting in low resource scheduling efficiency.
[0004] According to a first aspect of an embodiment of the present application, a resource management system is provided, the resource management system comprising a control cluster and a service cluster, wherein a master resource controller is configured in the control cluster;
[0005] The master resource controller is used to receive resource declaration information, determine a target service cluster in the service cluster according to the resource declaration information, and send a deployment instruction corresponding to the target slave resource controller to the target service cluster;
[0006] When the target slave resource controller is deployed in the target business cluster according to the deployment instruction, the target slave resource controller is used to listen to the resource description information of the target business cluster and schedule the resources in the target business cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
[0007] According to a second aspect of an embodiment of the present application, a resource management method is provided, which is applied to a main resource controller deployed in a control cluster, comprising:
[0008] Receive resource declaration information;
[0009] Determine a service cluster in a resource management system to which the control cluster belongs, and determine a target service cluster in the service cluster according to the resource declaration information;
[0010] Send a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0011] According to a third aspect of an embodiment of the present application, a resource management method is provided, which is applied to a target slave resource controller deployed in a target service cluster, including:
[0012] Determining a control cluster in a resource management system to which the target service cluster belongs;
[0013] Monitoring resource description information sent by a master resource controller in the control cluster to the target service cluster;
[0014] The resources in the target service cluster are scheduled based on the resource description information.
[0015] According to a fourth aspect of an embodiment of the present application, a resource management device is provided, which is applied to a main resource controller deployed in a control cluster, including:
[0016] A receiving module, configured to receive resource declaration information;
[0017] a determination module configured to determine a service cluster in a resource management system to which the control cluster belongs, and to determine a target service cluster in the service cluster according to the resource declaration information;
[0018] The sending module is configured to send a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0019] According to a fifth aspect of an embodiment of the present application, a resource management device is provided, which is applied to a main resource controller deployed in a control cluster, including:
[0020] A determination module, configured to determine a control cluster in a resource management system to which the target service cluster belongs;
[0021] A monitoring module, configured to monitor resource description information sent by a main resource controller in the control cluster to the target service cluster;
[0022] The scheduling module is configured to schedule the resources in the target service cluster based on the resource description information.
[0023] According to the sixth aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the resource management method when executing the computer instructions.
[0024] According to a seventh aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the computer instructions are executed by a processor, the steps of the resource management method are implemented.
[0025] The resource management system provided by the present application includes a control cluster and a business cluster, wherein a master resource controller is configured in the control cluster; the master resource controller is used to receive resource declaration information, determine a target business cluster in the business cluster according to the resource declaration information, and send a deployment instruction corresponding to a target slave resource controller to the target business cluster; when the target slave resource controller is deployed in the target business cluster according to the deployment instruction, the target slave resource controller is used to listen to the resource description information of the target business cluster, and schedule the resources in the target business cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
[0026] An embodiment of the present application implements a master resource controller on a control cluster that is responsible for monitoring resource declaration information of a business cluster, and deploys a target slave resource controller on a target business cluster based on the resource declaration information, so that the target slave resource controller determines the scheduling of resources of the target business cluster, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and dispersing the pressure of resource scheduling to the control cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a scenario diagram of a resource management system provided by an embodiment of the present application;
[0028] Figure 2 It is a structural diagram of a resource management system provided by an embodiment of the present application;
[0029] Figure 3 It is a structural diagram of a resource management system provided by an embodiment of the present application;
[0030] Figure 4 is a flow chart of a resource management method provided by an embodiment of the present application;
[0031] Figure 5 It is a processing flow chart of a resource management system applied to a business update scenario provided by an embodiment of the present application;
[0032] Figure 6is a flow chart of a resource management method applied to a master resource controller provided by an embodiment of the present application;
[0033] Figure 7 is a flow chart of a resource management method applied to a slave resource controller provided by an embodiment of the present application;
[0034] Figure 8 It is a structural diagram of a resource management device applied to a main resource controller provided by an embodiment of the present application;
[0035] Fig. 9 It is a structural diagram of a resource management device applied to a slave resource controller provided by an embodiment of the present application;
[0036] Fig.10 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0037] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0038] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.
[0039] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0040] First, the terms involved in one or more embodiments of the present application are explained.
[0041] Kubernetes: Kubernetes is a container cluster management system. It is an open source system for automatically deploying, expanding, and managing containerized applications. It can manage the life cycle of containers in the cluster, and combine its own health check and error recovery mechanisms to achieve high availability of applications within the cluster. It can realize automated deployment of container clusters, automatic expansion and reduction, operation and maintenance monitoring, and other functions. Therefore, Kubernetes container clusters are usually deployed on cloud computing platforms.
[0042] With the continuous development of cloud native technology, users' demand for Kubernetes clusters has also increased, and new requirements have been put forward for cloud native technology. Since Kubernetes clusters can only schedule resources within the cluster and cannot achieve cross-cluster management, when the number of clusters expands to a certain number, the difficulty of management also increases sharply, and manual intervention management may also cause errors. In order to solve this problem, the concepts of control cluster and business cluster were born. The control cluster is responsible for managing all business clusters. Its main function is to dispatch the resources required by users to the target business cluster, while the business cluster is only responsible for the business applications within the cluster, and all resources are dispatched by the control cluster. In this way, all operations are concentrated on the control cluster, so that operation and maintenance managers can manage Kubernetes clusters more effectively based on the control cluster. However, as the number of business clusters that the control cluster needs to manage continues to expand, and the number of resources that need to be dispatched continues to increase, the control cluster may be exhausted from scheduling, the scheduling efficiency will be greatly reduced, and even memory overflow may occur due to too many cluster resources that need to be managed.
[0043] Based on this, in the present application, a resource management system is provided. The present application also involves a resource management method, a resource management apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0044] In actual applications, resource scheduling is usually done by deploying different resource controllers in the control cluster. Each resource controller is responsible for managing the business clusters of the same type of resources. The resource controllers on the control cluster monitor resource changes and schedule or deploy the resources that users ultimately need to the business cluster. Figure 1 , Figure 1 Schematic diagram of a resource management method provided in an embodiment of the present application, wherein a resource controller is deployed in the control cluster, and the resource controller monitors the application definition and resource declaration on each business cluster, such as Figure 1 The resource controller monitors the application definitions on cluster A and cluster B. When a business application in a business cluster requires an application definition, such as application creation or update events, the resource controller dispatches the resources corresponding to the business application to the business cluster.
[0045] Taking application scheduling as an example, the resource controller monitors the application definition events of all clusters on the control cluster and schedules them to different business clusters. At the same time, it also monitors whether the application scheduling of the business cluster is successful and whether the running status is normal, so as to write the monitored data back to the application status for user feedback. Therefore, when there are more and more business clusters and applications, all scheduling pressure is concentrated on the resource controller of the control cluster, resulting in a decrease in the scheduling efficiency of the resource controller and even a risk of crash, which will affect the resource scheduling of all business clusters.
[0046] It can be seen that the above-mentioned asset management scheduling method may have the following problems during operation:
[0047] 1. When too many resources need to be scheduled at the same time, resource scheduling may cause scheduling delays because the controller cannot schedule resources at high concurrency.
[0048] 2. When a business cluster is abnormal, such as network fluctuations or abnormal API server, the control cluster may not be able to detect it in time. While waiting for the business cluster to respond, it will block the scheduling of other business cluster resources, thereby affecting the overall scheduling efficiency of the control cluster.
[0049] 3. When the business cluster reaches a certain scale, the resources that the resource controller on the control cluster needs to cache will also increase, which may cause the resource controller to exceed the memory quota and crash. In this case, Kubernetes automatically restarting the resource controller may not solve the problem.
[0050] 4. When the resource controller is restarted or updated, the resource controller needs to resynchronize and confirm the resource status of all business clusters. The confirmation process may affect the resource scheduling of other business clusters.
[0051] In view of this, in order to reduce the impact of the controller on the control cluster on the control cluster resources due to the expansion of the business cluster and its resource scale, and solve the problem that may cause slow scheduling or even crash, the present application provides a resource management system, which divides the resource controller into a main resource controller deployed on the control cluster and a slave resource controller deployed on the business cluster, so that the main resource controller is responsible for monitoring the resource creation, update and deletion events of the business cluster. When the resources declared in a cluster managed by the user on the control cluster change, such as creation and update, the main resource controller will actively send a deployment instruction from the resource controller to the target business cluster, deploy the slave resource controller on the target business cluster, and let the slave resource controller ensure the scheduling of the target business cluster resources and write the resource status back to the control cluster. It realizes the reduction of the scheduling pressure of the resource controller on the control cluster, distributes the resource scheduling pressure to the resource controllers on the respective business clusters, and the slave resource controllers on the business cluster monitor the resource declaration of the cluster, reducing the monitoring work pressure of the main resource controller. In actual applications, when a business cluster fails, it will not affect the main resource controller, avoiding the impact on the resource scheduling of other business clusters. At the same time, when the master resource controller is restarted or updated, there is no need to confirm the scheduling of resources of all business clusters. It is only necessary to confirm that the slave resource controller has been deployed on the target business cluster, and the slave resource controller confirms the resource scheduling.
[0052] Figure 2 It is a structural diagram of a resource management system provided according to an embodiment of the present application. The resource management system 200 includes a control cluster and a service cluster. A main resource controller 210 is configured in the control cluster.
[0053] The master resource controller 210 is used to receive resource declaration information, determine a target service cluster in the service cluster according to the resource declaration information, and send a deployment instruction corresponding to the target slave resource controller 220 to the target service cluster;
[0054] When the target slave resource controller is deployed in the target business cluster according to the deployment instruction, the target slave resource controller 220 is used to listen to the resource description information of the target business cluster and schedule the resources in the target business cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
[0055] Specifically, the resource management system 200 can be a Kubernetes container cluster management system, which includes a control cluster and multiple business clusters, and there is a communication channel between the control cluster and the business cluster. In actual applications, the control cluster monitors the resource creation, update and deletion events of the business cluster through the deployed master resource controller. When the resources declared by the user in a business cluster managed by the control cluster change, the master resource controller will receive the resource declaration information, which is the information corresponding to the operation request issued by the user for the application. If the user wants to create an application in the business cluster or update the resources in the application, the master resource controller will determine the target business cluster in all business clusters according to the resource declaration information, and send the target slave resource controller 220 to the target business cluster. The deployment instruction corresponding to the target is sent to the target business cluster, that is, a slave resource controller is actively sent to the target business cluster. It should be noted that each business cluster corresponds to its own slave resource controller, and the slave resource controller of each business cluster is used to be responsible for resource scheduling within the cluster.
[0056] Furthermore, after the target business cluster receives the target slave resource controller deployment instruction issued by the main resource controller, the corresponding target slave resource controller can be deployed in the target business cluster. The target slave resource controller is used to listen to the resource description information of this business cluster. The resource description information is the resource declaration of all business applications in the business cluster. In actual applications, the target slave resource controller can obtain the corresponding resource description information from the work queue. When the resource description information is the resource declaration information of this time, the corresponding resource scheduling steps can be executed, and the resource status can be written back to the main resource controller.
[0057] Based on this, by dividing the resource controller into a main resource controller deployed on the control cluster and a slave resource controller deployed on the business cluster, the scheduling pressure of the resource controller on the control cluster is effectively reduced, and the resource scheduling pressure is distributed to the resource controllers on each business cluster. The slave resource controller on the business cluster listens to the resource declaration of this cluster, reducing the monitoring work pressure of the main resource controller.
[0058] See also Figure 3 , Figure 3 is a schematic diagram of a resource management system according to an embodiment of the present application, which includes a control cluster and a service cluster. A main resource controller is deployed in the control cluster. Figure 3The cluster resource controller in the control cluster manages business cluster A and business cluster B. The control cluster is responsible for monitoring the resource declarations of the business clusters. When the resources declared in business cluster A change, the master resource controller will send the controller to business cluster A, that is, a resource controller will be deployed in business cluster A. This resource controller is the slave resource controller corresponding to business cluster A. After the master resource controller of the control cluster sends the resource description information of the monitoring event to the slave resource controller, the slave resource controller starts to schedule the resources of the business applications in the business cluster, monitors the resource status information of each business application, and writes the obtained resource status information back to the control cluster. Figure 3 The business cluster B without business applications is also included. Since there is no resource declaration of business applications in business cluster B, the control cluster will not deploy the corresponding slave resource controller in business cluster B.
[0059] For example, the user is going to create a business application in business cluster A. At this time, the master resource controller in the control cluster will receive a resource declaration information for the business application. The resource declaration information includes relevant information of the business application, such as business application ID identification information, business cluster, business node and other information, and relevant information of resources required by the business application. Therefore, the master resource controller can determine business cluster A as the target business cluster among all business clusters according to the resource declaration information, and the slave resource controller of business cluster A is required to realize the creation of business application A and the scheduling of related resources. Therefore, the master resource controller will send the deployment instruction corresponding to the target slave resource controller to business cluster A. After the slave resource controller is deployed in business cluster A based on the deployment instruction, the slave resource controller in business cluster A starts and starts working, and starts to listen to the resource declaration of all business applications in the business cluster. When the resource declaration information sent by the user this time is listened to by the master resource controller, the slave resource controller creates the corresponding business application in business cluster A based on the resource declaration information and schedules the corresponding resources.
[0060] In summary, the main resource controller in the control cluster monitors the resource declaration in the business cluster, and sends the deployment instructions from the resource controller to the target business cluster based on the resource declaration information. The slave resource controller in the target business cluster implements the resource scheduling corresponding to the resource declaration, so that the scheduling pressure of the main resource controller of the control cluster is dispersed to the slave resource controller in the business cluster, and the problem of high scheduling pressure on the control cluster due to the large number of business clusters will not occur.
[0061] Furthermore, in order to avoid the situation where the master resource controller repeatedly sends deployment instructions to the business system where the slave resource controller has been deployed, when it is monitored that the resources declared in the business system have changed and the slave resource controller needs to be deployed in the business system, it can first be detected whether the slave resource controller has been deployed in the business cluster. Specifically, the master resource controller is also used to determine whether the target slave resource controller has been deployed in the target business cluster. If so, the resource declaration information is sent to the target slave resource controller. If not, the deployment instruction corresponding to the target slave resource controller is sent to the target business cluster.
[0062] In actual applications, the master resource controller can determine whether the corresponding slave resource controller has been deployed in the business cluster by obtaining the deployment information of the business cluster. If the target slave resource controller currently exists in the target business cluster, the master resource controller will send the resource declaration information corresponding to the user operation request to the target slave resource controller; if the target slave resource controller does not exist in the target business cluster, the slave resource controller will send the corresponding slave resource controller deployment instruction to the target business cluster to deploy the target slave resource controller in the target business cluster.
[0063] In a specific embodiment of the present application, following the above example, the master resource controller monitors the resource declaration information in business cluster A, determines that the target business cluster is business cluster A, and determines whether business cluster A currently has a corresponding target slave resource controller deployed. If the judgment result is that business cluster A currently does not have a corresponding target slave resource controller deployed, the deployment instruction corresponding to the target slave resource controller is sent to business cluster A to deploy the target slave resource controller in business cluster A.
[0064] In another specific embodiment of the present application, following the above example, the main resource controller monitors the resource declaration information in business cluster A, determines that the target business cluster is business cluster A, and determines whether business cluster A currently has a corresponding target slave resource controller deployed. If the judgment result is that business cluster A currently has a corresponding target slave resource controller deployed, the resource declaration information is sent to the message queue, and the target slave resource controller in business cluster A obtains the resource declaration information from the message queue and executes resource scheduling corresponding to the resource declaration information.
[0065] In summary, when the master resource controller monitors changes in resources declared by the business cluster, it determines in advance whether the slave resource controller has been deployed in the business cluster, avoiding repeated sending of deployment instructions to the business cluster, thereby reducing unnecessary resource waste.
[0066] Furthermore, after the master resource controller sends resource declaration information to the target slave resource controller, the target slave resource controller can receive the resource declaration information and perform corresponding resource scheduling based on the resource declaration information. Specifically, the target slave resource controller is also used to schedule the target resource associated with the resource declaration information in the target business cluster based on the resource declaration information when the current processing event is associated with the resource declaration information in the resource description information.
[0067] Among them, the current processing event can be understood as the event corresponding to the resource declaration information currently being executed by the resource controller. For example, when the resource declaration information is a resource creation event, after receiving the resource declaration information from the resource controller, the current processing event is the resource creation event associated with the resource declaration information.
[0068] In actual applications, there may be changes in multiple declared resources in a business cluster, so the master resource controller will send multiple resource declaration messages to the slave resource controller accordingly. These multiple resource declaration messages and information from other applications in the business cluster constitute resource description information. The slave resource controller will obtain and process them one by one from the message queue. When the current processing event is the resource declaration information sent by the master resource controller, the associated target resources will be scheduled based on the resource declaration information.
[0069] In a specific embodiment of the present application, following the above example, the main resource controller passes the resource declaration information to the message queue. The resource declaration information is a resource creation event. When the target of business cluster A obtains the resource declaration information from the message queue of the resource controller, the current processing event is the corresponding resource creation event. At this time, the target resource controller schedules the target resources associated with the resource declaration information.
[0070] Furthermore, when the control cluster does not listen to the declaration of resources in the business cluster, it means that there are no resources in the business cluster and there is no need to deploy a slave resource controller. At this time, the slave resource controller deployed in the business cluster can be released to achieve the purpose of saving resources. Specifically, the master resource controller is also used to send a release instruction corresponding to the target slave resource controller to the target business cluster when the resource description information is not received within a preset detection time interval, wherein the release instruction is used to release the target slave resource controller in the target business cluster.
[0071] Among them, the preset detection time interval can be understood as the time preset for the main resource controller to provide detection. When the main resource controller does not listen to the resource declaration information in the target business cluster within the time interval, it means that there are no application resources in the target business cluster at this time. Then, the slave resource controller deployed in the target business cluster can be released, thereby saving resource consumption.
[0072] In a specific embodiment of the present application, following the above example, after business cluster A executes resource scheduling based on resource declaration information, the main resource controller continues to listen to application resource declarations in business cluster A. When the preset detection time interval is 1 minute, if the main resource controller does not listen to the resource declaration of business cluster A within 1 minute, it means that there are no resources in business cluster A. At this time, the corresponding release instruction of the target slave resource controller is sent to business cluster A, so that the target in business cluster A is released from the resource controller to save resource loss.
[0073] Further, after scheduling the synchronization resources based on the resource declaration information, the slave resource controller in the service cluster will start to monitor the status of the scheduled resources to write back the resource status information to the master resource controller. Specifically, the target slave resource controller is also used to monitor the resource status information of the associated service application in the target service cluster and send the resource status information to the master resource controller;
[0074] The master resource controller is further configured to receive the resource status information and feed back the resource status information to the user terminal.
[0075] Among them, resource status information can be understood as status information of business applications, such as application resources and whether business applications are running normally. The resource controller will continuously monitor the resource status information and write the resource status information back to the control cluster so that the control cluster can feedback to the user terminal based on the resource status information. The user terminal can be understood as the front-end device used by the user, and the user can understand the status of the resources based on the user terminal.
[0076] In actual applications, since the business cluster may have abnormal problems, such as network fluctuations, API server problems, etc., the resource scheduling corresponding to the business application may not be executed normally. When the slave resource controller in the business cluster monitors the resource status information of the business application, it will find that the corresponding resources have not completed the scheduling. After sending the resource status information to the main resource controller, the main resource controller will feedback the resource status information to the user terminal so that the user can understand the execution status of the business application, and can choose to resend the operation request for the business application to execute the resource scheduling of the business application again.
[0077] In a specific embodiment of the present application, following the above example, the target in business cluster A monitors the resource status information of the business application just created in business cluster A from the resource controller, and monitors that the resource status information is that the business application is running normally. The resource status information is sent to the main resource controller. After receiving the resource status information, the main resource controller can feedback to the user terminal based on the resource status information to enable the user to understand the running status of the business application.
[0078] Furthermore, when the main resource controller is restarted or updated, there is no need to confirm the scheduling of all resources in all clusters. It is only necessary to ensure that the slave resource controller has been scheduled on the target business cluster, thereby reducing the operating pressure of the main resource controller. Specifically, the main resource controller is also used to obtain the controller deployment information of the target business cluster when the controller status meets the preset configuration conditions, and send feedback information associated with the controller deployment information to the target business cluster based on the controller deployment information.
[0079] Among them, the controller state can be understood as the current operating state of the main resource controller, such as the restart completion state or the update completion state of the main resource controller. The preset configuration condition can be understood as the preset condition for obtaining the business cluster configuration information. When the main resource controller is in the restart completion state or the update completion state, the main resource controller needs to confirm whether the slave resource controller has been deployed on the target business cluster. It is set that when the controller state of the main resource controller is the restart completion state or the update completion state, the preset configuration condition is met. When the preset configuration condition is met, it indicates that the master resource controller needs to obtain the configuration information of each target business cluster, that is, the controller deployment information, so as to confirm whether the slave resource controller has been deployed on the target business cluster.
[0080] In actual applications, after the main resource controller obtains the controller deployment information of the target service cluster, it sends corresponding feedback information to the target service cluster based on the controller deployment information.
[0081] In a specific embodiment of the present application, after the master resource controller completes the restart in response to the restart instruction, it no longer schedules and confirms the resources on each business cluster, but instead obtains the controller deployment information of the target business cluster, and sends corresponding feedback information to the target business cluster based on whether the slave resource controller is deployed on the target business cluster.
[0082] Correspondingly, based on the controller deployment information of the business cluster, it can be determined whether the business cluster currently deploys a slave resource controller, so that the master resource controller can generate corresponding feedback information to the target business cluster based on the controller deployment information. Specifically, the master resource controller is also used to send the resource description information to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is deployed; and send the deployment instructions corresponding to the target slave resource controller to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is not deployed.
[0083] When the target service cluster has deployed the slave resource controller, the feedback information is the resource description information; when the target service cluster has not deployed the slave resource controller, the feedback information is the deployment instruction of the slave resource controller.
[0084] In a specific embodiment of the present application, the master resource controller determines that the service cluster A has currently deployed a slave resource controller according to the controller deployment information of the service cluster A, and then sends the corresponding resource description information to the target service cluster.
[0085] In summary, the main resource controller can obtain whether a slave resource controller is deployed on the business cluster based on the controller deployment information of the business cluster, thereby eliminating the need for the main resource controller to schedule and confirm the resources of all clusters. Instead, the slave resource controller can schedule and confirm the resources in its own cluster, effectively reducing the workload of the main resource controller.
[0086] In actual applications, when the master resource controller is preparing to issue deployment instructions to the target business cluster, it can first determine a target business node in the target business cluster and use the target business node as the carrier of the target slave resource controller. Specifically, the master resource controller is also used to determine the target business node in the target business cluster and generate deployment instructions corresponding to the target slave resource controller based on the target business node.
[0087] Among them, the target business node is a business node determined from multiple business nodes in the business cluster. The selection of the target business node can be set in advance. For example, there are 3 business nodes in business cluster A, namely business node 1, business node 2, and business node 3. It is preset that when business cluster A deploys a slave resource controller, business node 1 is used as the carrier of the slave resource controller, and the target business node is business node 1; the target business node can also be determined based on the load situation of the business node, and the load situation of each business node is compared to select the business node with less load pressure as the target business node.
[0088] In a specific embodiment of the present application, the business nodes include three business nodes, namely business node 1, business node 2, and business node 3. When the master resource controller receives resource declaration information and determines based on the information that the target business cluster is business cluster A, the target business node is selected as the slave resource controller to be deployed according to the load situation of each business node in business cluster A. According to the selection situation, business node 1 with less load is selected as the target business node, and the corresponding deployment instruction is generated based on the node attribute information of business node 1 and sent to the target business cluster.
[0089] The present application provides a resource management system, including a control cluster and a service cluster, wherein the control cluster is configured with a master resource controller; the master resource controller is used to receive resource declaration information, determine a target service cluster in the service cluster according to the resource declaration information, and send a deployment instruction corresponding to a target slave resource controller to the target service cluster; when the target slave resource controller is deployed in the target service cluster according to the deployment instruction, the target slave resource controller is used to monitor the resource description information of the target service cluster, and schedule the resources in the target service cluster based on the resource description information, wherein the resource description information includes the resource declaration information. The master resource controller on the control cluster is responsible for monitoring the resource declaration information of the service cluster, and the target slave resource controller is deployed on the target service cluster according to the resource declaration information, so that the target slave resource controller determines the scheduling of the target service cluster resources, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and distributing the pressure of resource scheduling to the control cluster.
[0090] It should be noted that this embodiment provides multiple implementation methods, and reference can be made to other parts that are not described in detail.
[0091] Corresponding to the above system embodiment, this specification also provides a resource management method embodiment. Figure 4 FIG. 1 is a flowchart of a resource management method provided by an embodiment of the present application. Figure 4 As shown, the resource management method includes:
[0092] Step 402: The main resource controller receives resource declaration information, and determines a target service cluster in the service cluster according to the resource declaration information.
[0093] Step 404: The master resource controller sends a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0094] Step 406: The target slave resource controller monitors the resource description information of the target service cluster.
[0095] Step 408: The target slave resource controller schedules the resources in the target service cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
[0096] In an optional embodiment, the master resource controller is also used to determine whether the target slave resource controller has been deployed in the target business cluster. If so, the resource declaration information is sent to the target slave resource controller. If not, the deployment instructions corresponding to the target slave resource controller are sent to the target business cluster.
[0097] In an optional embodiment, the target slave resource controller is further used to schedule the target resources associated with the resource declaration information in the target business cluster based on the resource declaration information when the current processing event is associated with the resource declaration information in the resource description information.
[0098] In an optional embodiment, the master resource controller is also used to send a release instruction corresponding to the target slave resource controller to the target business cluster when the resource description information is not received within a preset detection time interval, wherein the release instruction is used to release the target slave resource controller in the target business cluster.
[0099] In an optional embodiment, the target slave resource controller is also used to monitor the resource status information of the associated business applications in the target business cluster and send the resource status information to the master resource controller; the master resource controller is also used to receive the resource status information and feed back the resource status information to the user terminal.
[0100] In an optional embodiment, the master resource controller is further used to obtain controller deployment information of the target business cluster when the controller state meets preset configuration conditions, and send feedback information associated with the controller deployment information to the target business cluster based on the controller deployment information.
[0101] In an optional embodiment, the master resource controller is also used to send the resource description information to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is deployed; and to send the deployment instructions corresponding to the target slave resource controller to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is not deployed.
[0102] In an optional embodiment, the master resource controller is further configured to determine a target service node in the target service cluster, and generate a deployment instruction corresponding to the target slave resource controller based on the target service node.
[0103] The resource management method provided in this specification uses the master resource controller on the control cluster to monitor the resource declaration information of the business cluster, and deploys the target slave resource controller on the target business cluster according to the resource declaration information, so that the target slave resource controller determines the scheduling of the target business cluster resources, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and distributing the pressure of resource scheduling to the control cluster. The impact of the controller on the control cluster on the control cluster resources due to the expansion of the business cluster and its resource scale is reduced, solving the problem that may cause slow scheduling or even crash.
[0104] The following combination Figure 5 , taking the application of the resource management system provided by this application in the business update scenario as an example, the resource management system is further explained. Figure 5 A processing flow chart of a resource management system applied to a business update scenario provided by an embodiment of the present application is shown, which specifically includes the following steps:
[0105] Step 502: The main resource controller receives resource declaration information of the updated service application, and determines a target service cluster in the service cluster according to the resource declaration information.
[0106] Step 504: the master resource controller determines that the target slave resource controller is not deployed in the target service cluster, and sends a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0107] Step 506: The target slave resource controller monitors resource declaration information of all service applications of the target service cluster.
[0108] Step 508: When the current processing event is updating a service application, the target slave resource controller schedules the target resources in the target service cluster based on the resource declaration information corresponding to the updated service application.
[0109] Step 510: The target slave resource controller monitors the resource status information of the updated service application in the target service cluster, and sends the resource status information to the master resource controller.
[0110] Step 512: The main resource controller receives the resource status information and feeds back the resource status information to the user terminal.
[0111] Step 514: If the master resource controller does not receive resource declaration information of the target service cluster within a preset detection time interval, the master resource controller sends a release instruction corresponding to the target slave resource controller to the target service cluster.
[0112] To summarize, the present application provides a resource management system in which the master resource controller on the control cluster is responsible for listening to the resource declaration information of the business cluster, and deploys the target slave resource controller on the target business cluster based on the resource declaration information, so that the target slave resource controller determines the scheduling of the target business cluster resources, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and dispersing the pressure of resource scheduling to the control cluster.
[0113] This specification also provides an embodiment of a resource management method applied to a master resource controller. Figure 6 FIG. 1 is a flowchart of a resource management method applied to a master resource controller provided by an embodiment of the present application. Figure 6 As shown, the resource management method includes:
[0114] Step 602: Receive resource declaration information.
[0115] Step 604: Determine a service cluster in the resource management system to which the control cluster belongs, and determine a target service cluster in the service cluster according to the resource declaration information.
[0116] Step 606: Send a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0117] Optionally, the method further includes: determining whether a target slave resource controller has been deployed in the target business cluster; if so, sending the resource declaration information to the target slave resource controller; if not, sending a deployment instruction corresponding to the target slave resource controller to the target business cluster.
[0118] Optionally, the method further includes: when the resource description information is not received within a preset detection time interval, sending a release instruction corresponding to the target slave resource controller to the target business cluster, wherein the release instruction is used to release the target slave resource controller in the target business cluster.
[0119] Optionally, the method further includes: when the controller state meets a preset configuration condition, acquiring controller deployment information of the target business cluster, and sending feedback information associated with the controller deployment information to the target business cluster based on the controller deployment information.
[0120] Optionally, the method also includes: when it is determined according to the controller deployment information that the target slave resource controller is deployed, sending the resource description information to the target business cluster; when it is determined according to the controller deployment information that the target slave resource controller is not deployed, sending the deployment instructions corresponding to the target slave resource controller to the target business cluster.
[0121] Optionally, the method further includes: determining a target service node in the target service cluster, and generating a deployment instruction corresponding to the target slave resource controller based on the target service node.
[0122] The present application provides a resource management method applied to a master resource controller, including receiving resource declaration information; determining a business cluster in a resource management system to which the control cluster belongs, and determining a target business cluster in the business cluster according to the resource declaration information; and sending a deployment instruction corresponding to a target slave resource controller to the target business cluster. The master resource controller on the control cluster is responsible for monitoring the resource declaration information of the business cluster, and the target slave resource controller is deployed on the target business cluster according to the resource declaration information, so that the target slave resource controller determines the scheduling of the target business cluster resources, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and distributing the pressure of resource scheduling to the control cluster.
[0123] This specification also provides an embodiment of a resource management method applied to a slave resource controller. Figure 7 FIG. 1 shows a flow chart of a resource management method applied to a slave resource controller provided by an embodiment of the present application. Figure 7 As shown, the resource management method includes:
[0124] Step 702: Determine a control cluster in the resource management system to which the target service cluster belongs.
[0125] Step 704: monitor the resource description information sent by the main resource controller in the control cluster to the target service cluster.
[0126] Step 706: Schedule the resources in the target service cluster based on the resource description information.
[0127] Optionally, the method further includes: in a case where the current processing event is associated with the resource declaration information in the resource description information, scheduling the target resource associated with the resource declaration information in the target service cluster based on the resource declaration information.
[0128] Optionally, the method further includes: monitoring resource status information of associated service applications in the target service cluster, and sending the resource status information to the master resource controller.
[0129] The present application provides a resource management method for an application and a slave resource controller, including determining a control cluster in a resource management system to which the target business cluster belongs; monitoring resource description information sent by a master resource controller in the control cluster to the target business cluster; and scheduling resources in the target business cluster based on the resource description information. By implementing resource scheduling within the business cluster through a slave resource controller, the scheduling pressure of the master resource controller is reduced, and each business cluster corresponds to its own slave resource controller, so that when a business cluster fails, it will not affect other business clusters.
[0130] Corresponding to the above method embodiment, the present application also provides a resource management device embodiment applied to a master resource controller, Figure 8 FIG. 1 shows a schematic diagram of a resource management device applied to a master resource controller according to an embodiment of the present application. Figure 8 As shown, the device comprises:
[0131] The receiving module 802 is configured to receive resource declaration information;
[0132] The determination module 804 is configured to determine a service cluster in the resource management system to which the control cluster belongs, and determine a target service cluster in the service cluster according to the resource declaration information;
[0133] The sending module 806 is configured to send a deployment instruction corresponding to the target slave resource controller to the target service cluster.
[0134] Optionally, the device further includes a judgment module configured to:
[0135] It is determined whether the target slave resource controller has been deployed in the target service cluster. If so, the resource declaration information is sent to the target slave resource controller. If not, a deployment instruction corresponding to the target slave resource controller is sent to the target service cluster.
[0136] Optionally, the device further comprises a releasing module configured to:
[0137] When the resource description information is not received within a preset detection time interval, a release instruction corresponding to the target slave resource controller is sent to the target service cluster, wherein the release instruction is used to release the target slave resource controller in the target service cluster.
[0138] Optionally, the device further includes a feedback module configured to:
[0139] When the controller state meets the preset configuration condition, controller deployment information of the target service cluster is obtained, and feedback information associated with the controller deployment information is sent to the target service cluster based on the controller deployment information.
[0140] Optionally, the feedback module is further configured to:
[0141] When it is determined according to the controller deployment information that the target slave resource controller is deployed, the resource description information is sent to the target business cluster; when it is determined according to the controller deployment information that the target slave resource controller is not deployed, the deployment instruction corresponding to the target slave resource controller is sent to the target business cluster.
[0142] Optionally, the device further includes a generating module configured to:
[0143] A target service node is determined in the target service cluster, and a deployment instruction corresponding to the target slave resource controller is generated based on the target service node.
[0144] The present application provides a resource management device applied to a master resource controller, in which the master resource controller on the control cluster is responsible for monitoring the resource declaration information of the business cluster, and deploys a target slave resource controller on the target business cluster according to the resource declaration information, so that the target slave resource controller determines the scheduling of the target business cluster resources, thereby effectively reducing the scheduling pressure of the resource controller on the control cluster and dispersing the pressure of resource scheduling to the control cluster.
[0145] The above is a schematic scheme of the resource management device of this embodiment. It should be noted that the technical scheme of the resource management device and the technical scheme of the resource management method described above are of the same concept, and the details not described in detail in the technical scheme of the resource management device can be found in the description of the technical scheme of the resource management method described above.
[0146] Corresponding to the above method embodiment, the present application also provides a resource management device embodiment applied to a slave resource controller, Fig. 9 FIG. 1 shows a schematic diagram of a resource management device applied to a slave resource controller provided by an embodiment of the present application. Fig. 9 As shown, the device comprises:
[0147] A determination module 902 is configured to determine a control cluster in a resource management system to which the target service cluster belongs;
[0148] A monitoring module 904 is configured to monitor resource description information sent by a main resource controller in the control cluster to the target service cluster;
[0149] The scheduling module 906 is configured to schedule the resources in the target service cluster based on the resource description information.
[0150] Optionally, the scheduling module 906 is further configured to:
[0151] In a case where the current processing event is associated with the resource declaration information in the resource description information, the target resource associated with the resource declaration information in the target service cluster is scheduled based on the resource declaration information.
[0152] Optionally, the device further includes a sending module configured to:
[0153] Monitor resource status information of associated service applications in the target service cluster, and send the resource status information to the main resource controller.
[0154] The present application provides a resource management method applied to a slave resource controller, which implements resource scheduling within the business cluster through the slave resource controller, thereby reducing the scheduling pressure of the master resource controller. In addition, each business cluster corresponds to its own slave resource controller, and when a business cluster fails, it will not affect other business clusters.
[0155] The above is a schematic scheme of the resource management device of this embodiment. It should be noted that the technical scheme of the resource management device and the technical scheme of the resource management method described above are of the same concept, and the details not described in detail in the technical scheme of the resource management device can be found in the description of the technical scheme of the resource management method described above.
[0156] Fig.10 The structure block diagram of a computing device 1000 provided according to an embodiment of the present application is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.
[0157] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0158] In one embodiment of the present application, the above components of the computing device 1000 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0159] The computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 1000 may also be a mobile or stationary server.
[0160] When the processor 1020 executes the computer instructions, the steps of the resource management method are implemented.
[0161] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the resource management method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the resource management method described above.
[0162] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which implement the steps of the resource management method as described above when executed by a processor.
[0163] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the resource management method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the resource management method described above.
[0164] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0165] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0166] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0168] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.
Claims
1. A resource management system, characterized in that: The resource management system includes a control cluster and a service cluster, and the control cluster is configured with a main resource controller; The master resource controller is used to receive resource declaration information, determine a target business cluster in the business cluster according to the resource declaration information, and send a deployment instruction corresponding to the target slave resource controller to the target business cluster, wherein the resource declaration information is information corresponding to an operation request for a business application; When the target slave resource controller is deployed in the target business cluster according to the deployment instruction, the target slave resource controller is used to listen to the resource description information of the target business cluster and schedule the resources in the target business cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
2. The resource management system according to claim 1, characterized in that: The master resource controller is also used to determine whether the target slave resource controller has been deployed in the target business cluster. If so, it sends the resource declaration information to the target slave resource controller. If not, it sends the deployment instructions corresponding to the target slave resource controller to the target business cluster.
3. The resource management system according to claim 1, characterized in that: The target slave resource controller is further configured to schedule the target resource associated with the resource declaration information in the target service cluster based on the resource declaration information when the current processing event is associated with the resource declaration information in the resource description information.
4. The resource management system according to claim 1, characterized in that: The master resource controller is further used to send a release instruction corresponding to the target slave resource controller to the target business cluster when the resource description information is not received within a preset detection time interval, wherein the release instruction is used to release the target slave resource controller in the target business cluster.
5. The resource management system according to claim 1, characterized in that: The target slave resource controller is further used to monitor resource status information of associated service applications in the target service cluster and send the resource status information to the master resource controller; The master resource controller is further configured to receive the resource status information and feed back the resource status information to the user terminal.
6. The resource management system according to claim 1, characterized in that: The master resource controller is further configured to obtain controller deployment information of the target service cluster when the controller state meets a preset configuration condition, and send feedback information associated with the controller deployment information to the target service cluster based on the controller deployment information.
7. The resource management system according to claim 6, characterized in that: The master resource controller is also used to send the resource description information to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is deployed; and send the deployment instructions corresponding to the target slave resource controller to the target business cluster when it is determined according to the controller deployment information that the target slave resource controller is not deployed.
8. The resource management system according to claim 1, characterized in that: The master resource controller is further configured to determine a target service node in the target service cluster, and generate a deployment instruction corresponding to the target slave resource controller based on the target service node.
9. A resource management method, characterized in that: Applicable to the main resource controller deployed in the control cluster, including: Receive resource declaration information, wherein the resource declaration information is information corresponding to an operation request for a business application; Determine a service cluster in a resource management system to which the control cluster belongs, and determine a target service cluster in the service cluster according to the resource declaration information; Sending a deployment instruction corresponding to a target slave resource controller to the target business cluster, wherein the target slave resource controller is used to monitor resource description information of the target business cluster, and schedule resources in the target business cluster based on the resource description information, wherein the resource description information includes the resource declaration information.
10. A resource management method, characterized in that: Applicable to the target slave resource controller deployed in the target business cluster, including: Determining a control cluster in a resource management system to which the target service cluster belongs; Listen to the resource description information sent by the master resource controller in the control cluster to the target business cluster, wherein the master resource controller is used to receive resource declaration information, determine the target business cluster in the business cluster according to the resource declaration information, and send the deployment instruction corresponding to the target slave resource controller to the target business cluster, wherein the resource declaration information is information corresponding to the operation request for the business application; The resources in the target service cluster are scheduled based on the resource description information.
11. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 9 or 10 are implemented.
12. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 9 or 10 are implemented.
Citation Information
Patent Citations
Resource deployment method and device, electronic equipment and storage medium
CN113448686A
service deployment method and device for multiple Kubernetes clusters
CN113849266A