A resource scheduling method, device, apparatus and storage medium

Abstract

The application discloses a resource scheduling method and device, equipment and a storage medium, wherein the method comprises: determining a changed scheduling strategy in the Kubernetes cluster; updating a scheduling strategy set in the cache of each node in the Kubernetes cluster based on the changed scheduling strategy; obtaining a deployment strategy of a newly created Pod in the Kubernetes cluster; determining a target scheduling strategy in the updated scheduling strategy set based on the deployment strategy; determining a target node in the node set of the Kubernetes cluster based on the target scheduling strategy; and deploying the newly created Pod to the target node.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of cloud computing, and relates to but not limited to a resource scheduling method, device, equipment and storage medium. BACKGROUND

[0002] In a Kubernetes cluster, a service or an application usually interacts and communicates with services in the same system or even external services. An application in an enterprise system often needs the cooperation of dozens or even hundreds of individual services, resulting in more complex application topology of microservice architecture, reduced interaction stability between services, and the failure of a service possibly affecting other services on the call chain, forming an avalanche effect. For some interdependent services, if these services are distributed in different nodes in the cluster, when a node in the cluster fails or the network communication between nodes fails, other services and applications that depend on certain services on the node will be affected to varying degrees, and even cannot work normally. At this time, the operation and maintenance personnel can only wait for Kubernetes to perform service migration or manually migrate the service to other available nodes, and the availability of the service will be affected.

[0003] Although the scheduling strategy of the original Kubernetes scheduler can be used to cope with the problem of uneven allocation of node resources, such as BalancedResourceAllocation. However, the original Kubernetes scheduler cannot perceive the actual resource usage, and is based on static configuration for allocation. Since the resource allocation is static, it cannot represent the real resource usage, and the utilization rates of the central processing unit and the memory of the node are often in an unbalanced state. SUMMARY

[0004] Therefore, the embodiment of the present application provides a resource scheduling method, device, equipment and storage medium.

[0005] The technical scheme of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, the embodiment of the present application provides a resource scheduling method applied to a Kubernetes cluster, and the method comprises the following steps:

[0007] determine a changed scheduling policy in the Kubernetes cluster; update a scheduling policy set in a cache of each node in the Kubernetes cluster based on the changed scheduling policy; obtain a deployment policy of a new Pod in the Kubernetes cluster; determine a target scheduling policy in the updated scheduling policy set based on the deployment policy; determine a target node in a node set of the Kubernetes cluster based on the target scheduling policy; and deploy the new Pod to the target node.

[0008] In a second aspect, an embodiment of the present application provides a resource scheduling apparatus, and the apparatus comprises:

[0009] a first determining module configured to determine a changed scheduling policy in the Kubernetes cluster; an updating module configured to update a scheduling policy set in a cache of each node in the Kubernetes cluster based on the changed scheduling policy; an obtaining module configured to obtain a deployment policy of a new Pod in the Kubernetes cluster; a second determining module configured to determine a target scheduling policy in the updated scheduling policy set based on the deployment policy; a third determining module configured to determine a target node in a node set of the Kubernetes cluster based on the target scheduling policy; and a deploying module configured to deploy the new Pod to the target node.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the above method when executing the program.

[0011] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for causing a processor to implement the above method when executing the instructions.

[0012] In the embodiment of the present application, first, a changed scheduling policy in the Kubernetes cluster is determined; a scheduling policy set in a cache of each node in the Kubernetes cluster is updated based on the changed scheduling policy; then, a deployment policy of a new Pod in the Kubernetes cluster is obtained; a target scheduling policy is determined in the updated scheduling policy set based on the deployment policy; finally, a target node in a node set of the Kubernetes cluster is determined based on the target scheduling policy; and the new Pod is deployed to the target node. In this way, better scheduling flexibility can be achieved, the scheduling policy is dynamically configured according to the calling and resource usage characteristics of different services (Pods), so that more suitable scheduling policies are adopted for different service types; and the scheduling policy is hot modified, so that the scheduling policy is adjusted according to actual needs in cluster operation. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 An implementation flowchart of a resource scheduling method provided for an embodiment of the present application is shown in the figure.

[0014] Figure 2A A structure diagram of a scheduling strategy provided for an embodiment of the present application is shown in the figure.

[0015] Figure 2B A flowchart of a strategy cache provided for an embodiment of the present application is shown in the figure.

[0016] Figure 2C A diagram of node annotation provided for an embodiment of the present application is shown in the figure.

[0017] Figure 3 A comparison diagram of node resources after deployment of a Pod provided for an embodiment of the present application is shown in the figure.

[0018] Figure 4A A Pod deployment diagram provided for an embodiment of the present application is shown in the figure.

[0019] Figure 4B A Pod deployment diagram provided for an embodiment of the present application is shown in the figure.

[0020] Figure 5 A Pod scheduling workflow diagram provided for an embodiment of the present application is shown in the figure.

[0021] Figure 6A A Pod deployment flowchart provided for an embodiment of the present application is shown in the figure.

[0022] Figure 6B A Pod deployment flowchart provided for an embodiment of the present application is shown in the figure.

[0023] Figure 7 A node optimization flowchart provided for an embodiment of the present application is shown in the figure.

[0024] Figure 8 A component structure diagram of a resource scheduling apparatus provided for an embodiment of the present application is shown in the figure.

[0025] Figure 9 A hardware entity diagram of an electronic device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the embodiments of the present application will be further described in detail below with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0027] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets as each other and can be combined with each other as long as there is no conflict.

[0028] In the following description, the terms "first", "second", "third" are only to distinguish similar objects, and do not represent a specific order of the objects. It is understood that "first", "second", "third" can be interchanged in a specific order or sequence as long as it is allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the application and are not intended to limit the application.

[0030] Before the embodiments of the application are further described in detail, the terms and phrases involved in the embodiments of the application are explained, and the terms and phrases involved in the embodiments of the application are applicable to the following explanations.

[0031] Kubernetes, abbreviated as K8s, is a shortened form of replacing the eight characters "ubernete" in the middle of the name with 8. It is an open source that is used to manage containerized applications on multiple hosts in the cloud platform, and provides a mechanism for application deployment, planning, updating and maintenance.

[0032] Pod is the basic building block in K8s, representing the deployment unit and atomic running unit of Kubernetes, that is, a single running instance of an application, which is usually composed of one or more application containers that share resources and have close relationships.

[0033] ETCD is the default storage system provided by Kubernetes, which saves all cluster data, and a backup plan needs to be provided for ETCD data when used.

[0034] Node is the smallest computing hardware unit in Kubernetes, which represents a single machine in the cluster.

[0035] The embodiments of the application provide a resource scheduling method, which is applied to a Kubernetes cluster, as shown in Figure 1 The method comprises the following steps:

[0036] Step S110, determining a changed scheduling strategy in the Kubernetes cluster;

[0037] Here, the scheduling policy, i.e., the resource allocation policy, can be a scheduling policy used by the Kubernetes resource scheduler to deploy the to-be-scheduled Pod to a suitable node when scheduling resources. In the implementation process, multiple preselected schemes can be set according to actual conditions, and in the case of deploying a Pod, a suitable preselected scheme is selected according to actual conditions. The structure of the scheduling policy can be defined as the policy structure shown in Figure 2A The changed scheduling policy, i.e., the scheduling policy added, modified or deleted by the user according to actual needs.

[0038] In the implementation process, a unified scheduling policy module, i.e., a ConfigMap, can be added in the Kubernetes cluster to uniformly record all Pod scheduling policies. The user can flexibly add or modify the corresponding scheduling policy configuration for different types of services; and the change of the ConfigMap can determine the changed scheduling policy in the Kubernetes cluster.

[0039] Here, the changed scheduling policy can be a newly added scheduling policy, a scheduling policy modified based on the original scheduling policy, or a reduced scheduling policy. In the implementation process, a monitoring program can be set to monitor the scheduling policy ConfigMap to determine the changed scheduling policy; and an automatic reporting program can be set in the scheduling policy ConfigMap to report the changed scheduling policy when the scheduling policy changes.

[0040] Figure 2B A flowchart of a policy cache provided by an embodiment of the present application is shown in Figure 2B The flowchart includes the following steps:

[0041] Step S21, monitoring whether the policy is updated;

[0042] Here, a unified policy ConfigMap is added in the Kubernetes cluster, and the user can flexibly add or modify the corresponding scheduling policy configuration for different types of services. In the implementation process, a daemon program can be added in each node in the cluster to monitor whether the monitoring policy in the policy ConfigMap changes.

[0043] Step S22, refreshing the memory cache;

[0044] When it is determined that the policy ConfigMap changes, the changes are updated to the scheduling policy cache in the memory of each node. In the implementation process, the scheduling policy cache in the memory of each node can be added to store the latest aggregated scheduling policy.

[0045] By performing the above steps S21 and S22, the hot modification of the scheduling strategy can be realized, that is, the scheduling strategy is adjusted according to the actual demand in the cluster running.

[0046] Step S120, updating the scheduling strategy set in the cache of each node in the Kubernetes cluster based on the changed scheduling strategy.

[0047] In the implementation process, the latest scheduling strategy set can be stored in the cache of each node. Here, the scheduling strategy set can include at least one scheduling strategy. As shown in the figure, by performing step S22, the scheduling strategy set in the cache of each node in the Kubernetes cluster can be updated. Figure 2B

[0048] In some embodiments, the scoring rules of the node can also be stored in the cache of each node, and the user can add, modify or delete the scoring rules according to the actual demand. Here, the score of each node can be determined based on real-time monitoring of hardware availability, node service performance and node scheduling performance, and can also be determined for each deployment service according to historical and real-time service call statistics.

[0049] Step S130, obtaining the deployment strategy of the new Pod in the Kubernetes cluster.

[0050] In the implementation process, the deployment strategy can be determined based on the application supported by the Pod, or can be determined based on the function implemented by the Pod. For example, the deployment strategy can be determined based on the call condition of the application, or can be determined based on the demand of the Pod for node resources.

[0051] Step S140, determining a target scheduling strategy in the updated scheduling strategy set based on the deployment strategy.

[0052] In the case of creating a new service or Pod, the application programming interface server (API Server) can select a target scheduling strategy in the scheduling strategy set in the node cache according to the deployment strategy of the Pod. For example, in the case of the deployment strategy being the deployment strategy based on the call condition of the application, the deployment strategy based on the call relationship can be selected; in the case of the deployment strategy being the deployment strategy based on the demand of the Pod for node resources, the deployment strategy based on the node performance can be selected.

[0053] Step S150, determining a target node in the node set of the Kubernetes cluster based on the target scheduling strategy.

[0054] ​In the implementation process, at least a scoring rule for the node is included in the target scheduling strategy, the nodes after scoring are sorted, and the target node can be determined in the node set of the Kubernetes cluster.

[0055] Step S160, deploying the new Pod to the target node.

[0056] In the embodiments of the present application, first, the changed scheduling strategy in the Kubernetes cluster is determined; based on the changed scheduling strategy, the scheduling strategy set in the cache of each node in the Kubernetes cluster is updated; then, the deployment strategy of the new Pod in the Kubernetes cluster is obtained; based on the deployment strategy, the target scheduling strategy is determined in the updated scheduling strategy set; finally, based on the target scheduling strategy, the target node is determined in the node set of the Kubernetes cluster; and the new Pod is deployed to the target node. In this way, better scheduling flexibility can be achieved, the scheduling strategy is dynamically configured according to the calling of different services and the resource use characteristics, so that a more suitable scheduling strategy is adopted for different service types; and the scheduling strategy is hot modified, so that the scheduling strategy is adjusted according to the actual demand in the cluster operation.

[0057] In some embodiments, the updated scheduling strategy includes at least one of the following: a first scheduling strategy based on node performance, a second scheduling strategy based on calling relationship, a third scheduling strategy based on the node performance and the calling relationship;

[0058] The above step S140 of determining the target scheduling strategy in the updated scheduling strategy set based on the deployment strategy can be implemented by the following steps:

[0059] Step 141, determining that the deployment strategy is a deployment strategy based on the node performance;

[0060] Here, the node performance can comprehensively consider the hardware availability (CPU, memory, disk, etc.) of the node, the node service performance (service throughput, service response time, etc.), and the node scheduling performance (time consumption from Pod creation to being bound to the host). In order to obtain the node performance index, a service calling and monitoring statistical component can be introduced in the Pod scheduling process and running period.

[0061] Step 142, determining the first scheduling strategy as the target scheduling strategy;

[0062] Here, the user can set the first scheduling strategy stored in the node cache as the scheduling strategy based on the node performance according to the actual scheduling requirement, determine the first scheduling strategy as the target scheduling strategy in a case where the deployment strategy of the to-be-deployed Pod is the deployment strategy based on the node performance, and satisfy the requirement of the deployment strategy of the Pod on the node performance.

[0063] Step 143, determine that the deployment strategy is the deployment strategy based on the invocation relationship.

[0064] Here, the invocation relationship includes invocation traffic and invocation frequency between nodes in a case where a service supported by the Pod runs. In the implementation process, the service history and real-time invocation relationship can be obtained by using a service invocation statistical component.

[0065] Step 144, determine the second scheduling strategy as the target scheduling strategy.

[0066] Here, the user can set the second scheduling strategy stored in the node cache as the scheduling strategy based on the invocation relationship according to the actual scheduling requirement, determine the second scheduling strategy as the target scheduling strategy in a case where the deployment strategy of the to-be-deployed Pod is the deployment strategy based on the invocation relationship, and satisfy the requirement of the deployment strategy of the Pod on the invocation relationship.

[0067] Step 145, determine that the deployment strategy is the deployment strategy based on the node performance and the invocation relationship.

[0068] Step 146, determine the third scheduling strategy as the target scheduling strategy.

[0069] Here, the user can set the third scheduling strategy stored in the node cache as the scheduling strategy based on the node performance and the invocation relationship according to the actual scheduling requirement, determine the third scheduling strategy as the target scheduling strategy in a case where the deployment strategy of the to-be-deployed Pod is the deployment strategy based on the node performance and the invocation relationship, and satisfy the requirement of the deployment strategy of the Pod on the node performance and the invocation relationship.

[0070] In the embodiments of the present application, the updated scheduling strategy includes at least one of the following: a first scheduling strategy based on node performance, a second scheduling strategy based on call relationship, a third scheduling strategy based on the node performance and the call relationship; the deployment strategy is determined to be a deployment strategy based on the node performance; the first scheduling strategy is determined to be the target scheduling strategy; the deployment strategy is determined to be a deployment strategy based on the call relationship; the second scheduling strategy is determined to be the target scheduling strategy; the deployment strategy is determined to be a deployment strategy based on the node performance and the call relationship; and the third scheduling strategy is determined to be the target scheduling strategy. In this way, a suitable scheduling strategy can be selected based on the deployment strategy of the Pod to meet the deployment requirements of the Pod.

[0071] In some embodiments, the step S150 "determining a target node in the node set of the Kubernetes cluster based on the target scheduling strategy" described above can be implemented by the following steps:

[0072] Step 151, determining a candidate node set based on the resource information of each node in the Kubernetes cluster;

[0073] In the implementation process, the nodes in the cluster that are not suitable for running can be filtered first for the pod to be deployed, that is, based on the resource information of each node in the Kubernetes cluster, the nodes that have running problems or have no available resources can be filtered out to determine the candidate node set.

[0074] Step 152, obtaining hardware available information, service performance information and scheduling performance information of each node in the candidate node set based on the first scheduling strategy;

[0075] Here, the first scheduling strategy is a scheduling strategy based on node performance. The hardware available information, service performance information and scheduling performance information of each node in the candidate node set need to be obtained to score each node. The hardware available information at least includes the utilization rates of CPU, memory and disk and the like; the service performance information at least includes service throughput and service response time and the like; and the scheduling performance information at least includes the number of Pods bound to each node and the time consumption from the creation of each Pod to its being bound to a node.

[0076] In the implementation process, in the Pod scheduling process and running period, the service calling and monitoring and statistics component can be added to realize the connection service flow collection data and resource monitoring component. The monitoring and statistics component is used to monitor the hardware available information, service performance information and scheduling performance information of each node in real time. The monitoring and statistics component can also synchronize the collected monitoring data in the system, including the hardware usage, the scheduling performance of the Pod on the node, to the node annotation regularly. In the scheduling, the actual running condition of the cluster can be better perceived, so that the effect of effective utilization of the cluster resources after the Pod scheduling is finally achieved.

[0077] Figure 2C A node annotation schematic diagram provided for the embodiment of the application is shown in FIG. 3, which includes: a node annotation 23 used to store available CPU information (avaiable_cpu), available memory information (avaiable_memory), service throughput (service.throught), service response latency (service.latency) and binding time (scheduler.bindtime). Figure 2C

[0078] Step 153, scoring each node in the candidate node set based on the hardware available information, service performance information and scheduling performance information of the node, to obtain a node scoring order;

[0079] Step 154, determining a target node in the candidate node set based on the node scoring order.

[0080] In the implementation process, each node in the candidate node set is scored and sorted, and the node with the highest score is selected as the target node.

[0081] Figure 3 A node resource comparison diagram after the Pod deployment provided for the embodiment of the application is shown in FIG. 4, where the left part is the node resource after the Pod deployment of the prior art, and the right part is the node resource after the Pod deployment based on the first scheduling strategy, which obtains the hardware available information, service performance information and scheduling performance information of each node in the candidate node set, scores each node based on the hardware available information, service performance information and scheduling performance information of each node in the candidate node set, and completes the node resource after the Pod deployment. Figure 3

[0082] Figure 3 The left part: the new Pod is scheduled to node2, but the actual resource utilization of the node is relatively low, and the idle resource is not effectively utilized.

[0083] Figure 3 ​​Right part: node1, under the same resource allocation, the allocatable resource is recalculated, so that the new service Pod can be scheduled to the node, improving the resource utilization of the node.

[0084] In the embodiments of the present application, first, a candidate node set is determined based on the resource information of each node in the Kubernetes cluster; then, in the case of determining that the scheduling strategy is the first scheduling strategy, hardware available information, service performance information and scheduling performance information of each node in the candidate node set are obtained; the score of each node in the candidate node set is calculated based on the hardware available information, service performance information and scheduling performance information of each node, and a node scoring order is obtained; finally, the target node is determined in the candidate node set based on the node scoring order. In this way, through real-time monitoring of system resources and services, dynamic cluster running state perception is realized, and the node with the highest efficiency can be effectively determined as the target node.

[0085] In some embodiments, the step S150 "determining a target node in the node set of the Kubernetes cluster based on the target scheduling strategy" further includes the following steps:

[0086] Step 155, based on the second scheduling strategy, obtaining the call score of each node in the candidate node set associated with the new Pod;

[0087] In the implementation process, in the case of determining that the target scheduling strategy is the second scheduling strategy, i.e., the scheduling strategy based on the call relationship is the target scheduling strategy.

[0088] Step 156, based on the call score of each node in the candidate node set associated with the new Pod, scoring each node to obtain a node scoring order.

[0089] In the implementation process, in the Pod scheduling flow and running cycle, the service call and monitoring statistical component can be added to realize the connection of service traffic collection data and resource monitoring components. The service call statistical component obtains the historical and real-time call relationship, statistics service call information, historical resource usage of each node and Pod scheduling details of each node, and scores all nodes.

[0090] The call score associated with the new Pod can be determined based on the predicted scheduling traffic consumed by running the Pod supporting application after deploying the new Pod on each node. Based on historical and real-time service call information statistics, the call relationship score corresponding to each node can be calculated for each deployed service; and the calculated score is periodically maintained in the ETCD table.

[0091] Figure 4AA Pod deployment schematic diagram provided for an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, services with frequent calling relationship are scheduled to different nodes, and when Node2 fails, the services provided by Pod1 and Pod2 will be affected. Figure 4A

[0092] Figure 4B A Pod deployment schematic diagram provided for an embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, when a new Pod is created, services with frequent calling relationship are scheduled to the same node based on the calling score of the node, and when Node2 fails, the three services on Node1 are not affected. Figure 4B

[0093] Figure 4A Figure 4B Figure 4B Based on the calling score of the Pod, the impact on the services in the case of node failure is less than that of the Pod deployed in Figure 4A

[0094] In the embodiment of the present application, when the target scheduling strategy is determined to be the second scheduling strategy, the calling score of each node in the candidate node set associated with the new Pod is obtained; and each node is scored based on the calling score of the new Pod associated with each node in the candidate node set, and the node scoring order is obtained. In this way, through real-time monitoring of service calling conditions, dynamic cluster running state perception is realized, and the node with the highest calling score associated with the new Pod can be effectively determined as the target node. The services with high correlation are closely distributed in the Kubernetes cluster, and when a node in the system fails / goes offline, the impact on the application services on other nodes is reduced; similarly, when the system is running normally, the network traffic / load between nodes is reduced.

[0095] In some embodiments, the step S150 "determining a target node in the node set of the Kubernetes cluster based on the target scheduling strategy" further includes the following steps:

[0096] Step 157, based on the third scheduling strategy, obtaining the hardware available information, service performance information, scheduling performance information of each node in the candidate node set, and the calling score associated with the new Pod;

[0097] In the implementation process, the hardware available information, service performance information, and scheduling performance information of each node in the candidate node set are obtained by reading the resource usage in the node annotation; and the calling score associated with the new Pod is obtained by reading the "service-node calling score" in the ETCD.

[0098] ​​​​​​Step 158, scoring each node in the candidate node set based on the hardware available information, service performance information, scheduling performance information, and the call score associated with the new Pod of the node, to obtain the node scoring order.

[0099] In the implementation process, based on the call score between the existing services on the node and the to-be-deployed service, when the node resource usage rate is lower than the threshold value, the service is preferentially scheduled to the node with a higher "service-node call relationship score". The "service-node call relationship score" is calculated by the service call and monitoring statistics component and is used for the scoring module to calculate the scheduling order score of each node.

[0100] For example, in the node scoring and ordering stage, the following formula (1) can be used to calculate the total score of each node:

[0101] Total score = a * Norm(resource score) + b * Norm(call score) (1);

[0102] Wherein, a and b configured in the scheduling strategy represent the user's inclination between resources and call relationship, and Norm() represents the normalization operation on the score, which can be normalized by using the following formula (2):

[0103]

[0104] Wherein, max is the maximum value of all node scores, min is the minimum value, and x is the resource score or call score of the current node.

[0105] In the embodiment of the application, when the target scheduling strategy is determined to be the third scheduling strategy, the hardware available information, service performance information, scheduling performance information, and call score associated with the new Pod of each node in the candidate node set are obtained; each node in the candidate node set is scored based on the hardware available information, service performance information, scheduling performance information, and call score associated with the new Pod of the node, to obtain the node scoring order. In this way, the node with the highest score can be calculated by comprehensively considering the node efficiency and the call relationship.

[0106] In some embodiments, the above step 152 of "obtaining the hardware available information, service performance information, and scheduling performance information of each node in the candidate node set" can be implemented by the following steps:

[0107] Step 1521, determining the hardware available information based on the memory usage rate, CPU usage rate, and disk usage rate of the node;

[0108] In the implementation process, the hardware available information score of the node can be obtained based on the following formula (3):

[0109]

[0110] wherein, S hardware represents the hardware available information score of the node, Usage cpu represents the CPU utilization of the node, Usage memory represents the memory utilization of the node.

[0111] Step 1522, determining the service performance information of the node based on the service throughput and the service response time of the node.

[0112] In implementation, the service performance information score of the node can be obtained based on the following formula (4):

[0113]

[0114] wherein, S service represents the service performance information score of the node, Elasped n represents the service response time of the nth Pod deployed on the node, Throughput n represents the service throughput of the nth Pod deployed on the node.

[0115] Step 1523, determining the scheduling performance information of the node based on the Pod deployment time and the number of bound Pods of the node.

[0116] In implementation, the scheduling performance information score of the node can be obtained based on the following formula (5):

[0117]

[0118] wherein, S schedule represents the scheduling performance information score of the node, BindTime n represents the binding time of the nth Pod deployed on the node, n pod represents the number of bound Pods of the node.

[0119] The resource score of the node can be obtained based on the following formula (6):

[0120]

[0121] wherein, Score represents the resource score of the node, a represents the coefficient of the hardware available information score, b represents the coefficient of the service performance information score, and c represents the coefficient of the scheduling performance information score.

[0122] In implementation, the higher the resource score, the higher the performance of the node, and the more suitable for deploying Pods.

[0123] In the embodiments of the present application, the hardware available information can be effectively determined based on the memory usage, CPU usage and disk usage of the node; the service performance information can be effectively determined based on the service throughput and service response time of the node; and the scheduling performance information can be effectively determined based on the Pod deployment duration and the number of bound Pods of the node.

[0124] In some embodiments, the step 155 of "obtaining the call score of each node in the candidate node set associated with the new Pod" can be implemented by the following steps:

[0125] Step 1551, determining at least one service associated with the new Pod and the call traffic corresponding to each service;

[0126] In the implementation process, for example, according to the service call history data, it can be determined that the to-be-deployed Pod has a call relationship with three services, service_0, service_1 and service_2 respectively; and the call traffic of the three services and the deployed Pod is obtained as traffic_0, traffic_1 and traffic_2 respectively.

[0127] Step 1552, determining the node deployed by each service;

[0128] It is determined that there are three nodes node_0, node_1 and node_2 in the current cluster. Among them, service_0 and service_2 are deployed on node_0, and service_1 is deployed on node_2.

[0129] Step 1553, based on the node deployed by each service and the call traffic corresponding to each service, determining the call score of each node in the candidate node set associated with the new Pod.

[0130] Based on the node service and the call traffic, it can be determined that:

[0131] The call score of node_0 is:

[0132] The call score of node_1 is 0, because there is no service associated with the current Pod deployed on the node;

[0133] The call score of node_2 is:

[0134] In the embodiments of the present application, first, at least one service associated with the new Pod and the calling traffic corresponding to each service are determined; then the nodes on which each service is deployed are determined; and finally, based on the nodes on which each service is deployed and the calling traffic corresponding to each service, the calling score of each node in the candidate node set associated with the new Pod is determined. In this way, the node score calculated can effectively calculate the calling condition of the node associated with the to-be-deployed Pod. In the scheduling process, the scheduling strategy can preferentially schedule the Pods of services supporting close calling relationship to the same node.

[0135] In some embodiments, the step S110 "determining the changed scheduling strategy in the Kubernetes cluster" can be implemented by the following process:

[0136] Monitoring the scheduling strategy of the Kubernetes cluster to determine the changed scheduling strategy in the Kubernetes cluster.

[0137] In the implementation process, a node-in-cluster scheduling strategy daemon can be added to each node in the cluster to monitor the scheduling strategy ConfigMap, and when the ConfigMap changes, the changed scheduling strategy is updated to the scheduling strategy cache in the memory of each node.

[0138] In the embodiments of the present application, the changed scheduling strategy in the Kubernetes cluster can be effectively determined by monitoring the scheduling strategy of the Kubernetes cluster.

[0139] Figure 5 A Pod scheduling workflow schematic diagram provided in the embodiments of the present application is shown in FIG. 1, which includes a scheduling process (Scheduling Cycle) 51 and a binding process (Binding Cycle) 52, wherein, Figure 5

[0140] The scheduling process 51 and the binding process 52 are combined together and referred to as a scheduling context.

[0141] The scheduling process 51 includes pre-filtering (Pre-filter), filtering (Filter), pre-scoring (PreScore), scoring (Score), normalizing score (Normalize score), reserving (Reserve) and permitting (Permit). The scheduling process 51 is the core process of scheduling, and performs the above steps to make a scheduling decision and select a unique node.

[0142] ​The binding process 52 includes WaitOnPermit, Pre-bind, Bind and Post-bind. By executing the above steps, the Pod can be bound to the selected unique node.

[0143] Figure 5 The Pod scheduling strategy shown is the Kubernetes own scheduling strategy, i.e., a completely static scheduling strategy, which cannot be customized according to the actual needs of the service; during service migration, the service will be temporarily unavailable; there is a risk that the service cannot be started during migration.

[0144] There are two Pod scheduling schemes in the prior art:

[0145] Scheme one, set the nodeName of the service Pod with high coupling degree, specify that these service Pods can only be scheduled to a certain specified node. For example, if the dependence degree of service Pod1 and Pod2 is high, the scheduling scheme is set as: nodeName: nodeX, i.e., Pod1 and Pod2 can only be scheduled to nodeX node.

[0146] Scheme two, set the affinity scheduling, specify a group of nodes or Pod screening conditions for the service, so that the associated services are scheduled to the same node.

[0147] The above two Pod scheduling schemes have the following problems:

[0148] Problem of scheme one: cannot effectively use the scheduling strategy of Kubernetes, each Pod can only be scheduled to a specified node. If the Pod is evicted due to insufficient resources, it cannot be scheduled to other nodes, resulting in service unavailability.

[0149] Problem of scheme two: the affinity configuration of the deployed service can only be manually updated by operation and maintenance or development personnel, and the configuration is static and cannot be automatically adjusted according to the actual running situation of the cluster.

[0150] Figure 6A A flowchart of deploying a Pod provided by an embodiment of the present application is shown in Figure 6A The flowchart includes the following steps:

[0151] Step S601, in the case of needing to create a Pod, an API service obtains a preset strategy;

[0152] Step S602, schedule the Pod based on the Pod annotation and the preset strategy to create the Pod.

[0153] Here, the Pod annotation at least includes the identity of the Pod and the deployment strategy, and the Pod can be scheduled based on the deployment strategy of the Pod and the preset strategy to achieve the creation of the Pod.

[0154] Figure 6B A flowchart for deploying a Pod provided by an embodiment of the present application is shown in FIG. 6, which includes the following steps: Figure 6B

[0155] In step S610, the analysis controller reads the service call data and the monitoring data.

[0156] In step S611, the analysis controller analyzes the monitoring data to obtain a resource score, writes the monitoring data and the resource score into the node annotation, and analyzes the service call data to obtain a call score, and writes the call score into the ETCD.

[0157] In the implementation process, the analysis controller analyzes the monitoring data, and writes the monitoring data and the resource score into the annotation of the node through the node annotator component; the analysis controller simultaneously analyzes the service call data, calculates the "service-node call score" through the service call statistical component, and writes the "service-node call score" into the ETCD.

[0158] In step S612, the Pod annotation is obtained.

[0159] In step S613, in the case of needing to create a Pod, the API service obtains the latest strategy from the strategy cache of the node.

[0160] Here, the scheduling strategy daemon in Kubernetes updates the changes in the scheduling strategy ConfigMap to the scheduling strategy cache of the node in real time.

[0161] In the implementation process, when there is a request to create a Pod, the API Server reads the latest scheduling strategy from the scheduling strategy cache, and selects the appropriate scheduling strategy in combination with the Pod annotation.

[0162] In step S614, the Pod is scheduled based on the node annotation, the call score in the ETCD, the Pod annotation, and the latest strategy to achieve the creation of the Pod.

[0163] Figure 6B Compared with the Pod deployment process shown in FIG. 5, the Pod deployment process shown in FIG. 6 adds the node annotation and the cache of the latest strategy. Figure 6A Figure 6B The Pod deployment process shown in FIG. 6 has the following beneficial technical effects:

[0164] ​​1. Better scheduling flexibility, more suitable scheduling strategies can be used for different service types; scheduling strategies can be hot modified, and scheduling strategies can be adjusted according to actual needs in cluster operation;

[0165] 2. Real-time monitoring of system resources and services to realize dynamic cluster running state perception.

[0166] Figure 7 A node optimization flowchart provided by the embodiment of the application is shown in Figure 7 The flowchart includes the following steps:

[0167] Step S701, read node annotations and call scores;

[0168] The node annotations at least include: node actual available resource details, node service performance statistics and node scheduling performance statistics. The call score is the call score of each node obtained by the analysis controller analyzing service call data.

[0169] Step S702, calculate node scores and sort according to the current strategy setting;

[0170] The strategy at least includes the inclination weight of each node index. If the selected scheduling strategy is based on the call relationship of the service, the node scoring (optimization) stage will combine the resource utilization of the node with the score calculated by the call relationship in ETCD to obtain the final score.

[0171] Step S703, complete Pod deployment based on the final optimization sorting of the node.

[0172] In the implementation process, the Pod will be deployed on the node with the highest score.

[0173] By executing the above steps, the sorting of the node can be completed, the target node can be selected, and the Pod deployment can be completed.

[0174] Based on the foregoing embodiment, the embodiment of the application provides a resource scheduling device, which includes various modules, each module includes various sub-modules, and each sub-module includes various units, which can be realized by a processor in an electronic device. Of course, it can also be realized by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0175] Figure 8 The composition structure diagram of the resource scheduling device provided by the embodiment of the application is shown in Figure 8 The device 800 includes:

[0176] The first determining module 810 is configured to determine a changed scheduling policy in the Kubernetes cluster.

[0177] The updating module 820 is configured to update a scheduling policy set in each node cache in the Kubernetes cluster based on the changed scheduling policy.

[0178] The obtaining module 830 is configured to obtain a deployment policy of a new Pod in the Kubernetes cluster.

[0179] The second determining module 840 is configured to determine a target scheduling policy in the updated scheduling policy set based on the deployment policy.

[0180] The third determining module 850 is configured to determine a target node in a node set of the Kubernetes cluster based on the target scheduling policy.

[0181] The deployment module 860 is configured to deploy the new Pod to the target node.

[0182] In some embodiments, the updated scheduling policy comprises at least one of a first scheduling policy based on node performance, a second scheduling policy based on a calling relationship, a third scheduling policy based on the node performance and the calling relationship; and the second determining module 840 comprises a first determining sub-module, a second determining sub-module, a third determining sub-module, a fourth determining sub-module, a fifth determining sub-module and a sixth determining sub-module, wherein the first determining sub-module is configured to determine that the deployment policy is a deployment policy based on the node performance; the second determining sub-module is configured to determine the first scheduling policy as the target scheduling policy; the third determining sub-module is configured to determine that the deployment policy is a deployment policy based on the calling relationship; the fourth determining sub-module is configured to determine the second scheduling policy as the target scheduling policy; the fifth determining sub-module is configured to determine that the deployment policy is a deployment policy based on the node performance and the calling relationship; and the sixth determining sub-module is configured to determine the third scheduling policy as the target scheduling policy.

[0183] In some embodiments, the third determining module 830 comprises a seventh determining sub-module, a first obtaining sub-module, a first scoring sub-module, and an eighth determining sub-module. The seventh determining sub-module is configured to determine a candidate node set based on resource information of each node in the Kubernetes cluster. The first obtaining sub-module is configured to obtain hardware available information, service performance information, and scheduling performance information of each node in the candidate node set based on the first scheduling strategy. The first scoring sub-module is configured to score each node in the candidate node set based on the hardware available information, the service performance information, and the scheduling performance information of the node to obtain a node scoring order. The eighth determining sub-module is configured to determine a target node in the candidate node set based on the node scoring order.

[0184] In some embodiments, the third determining module 830 further comprises a second obtaining sub-module and a second scoring sub-module. The second obtaining sub-module is configured to obtain a call score of each node in the candidate node set associated with the new Pod based on the second scheduling strategy. The second scoring sub-module is configured to score each node in the candidate node set based on the call score of the node associated with the new Pod to obtain the node scoring order.

[0185] In some embodiments, the third determining module 830 further comprises a third obtaining sub-module and a third scoring sub-module. The third obtaining sub-module is configured to obtain hardware available information, service performance information, scheduling performance information, and a call score of each node in the candidate node set associated with the new Pod based on the third scheduling strategy. The third scoring sub-module is configured to score each node in the candidate node set based on the hardware available information, the service performance information, the scheduling performance information, and the call score of the node associated with the new Pod to obtain the node scoring order.

[0186] In some embodiments, the first obtaining sub-module comprises a first determining unit, a second determining unit, and a third determining unit. The first determining unit is configured to determine the hardware available information based on a memory usage rate, a CPU usage rate, and a disk usage rate of the node. The second determining unit is configured to determine the service performance information based on a service throughput and a service response time of the node. The third determining unit is configured to determine the scheduling performance information based on a Pod deployment duration and a number of bound Pods of the node.

[0187] In some embodiments, the second obtaining sub-module includes a fourth determining unit, a fifth determining unit, and a sixth determining unit. The fourth determining unit is configured to determine at least one service associated with the new Pod and calling traffic corresponding to each service. The fifth determining unit is configured to determine a node on which each service is deployed. The sixth determining unit is configured to determine, based on the node on which each service is deployed and the calling traffic corresponding to each service, a calling score of each node in the candidate node set associated with the new Pod.

[0188] In some embodiments, the first determining module 810 is further configured to monitor a scheduling policy of the Kubernetes cluster to determine a changed scheduling policy in the Kubernetes cluster.

[0189] The above device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0190] It should be noted that, in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0191] Correspondingly, the embodiments of the present application provide a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the resource scheduling method provided in the above embodiments.

[0192] Correspondingly, the embodiments of the present application provide an electronic device, Figure 9 A hardware entity schematic diagram of the electronic device provided in the embodiments of the present application is shown in FIG. 9, which includes a memory 901 and a processor 902. The memory 901 stores a computer program executable on the processor 902, and the processor 902 implements the steps of the resource scheduling method provided in the above embodiments when executing the program. Figure 9 The hardware entity of the device 900 includes the memory 901 and the processor 902. The memory 901 stores a computer program executable on the processor 902, and the processor 902 implements the steps of the resource scheduling method provided in the above embodiments when executing the program.

[0193] The memory 901 is configured to store instructions and applications executable by the processor 902, and can also cache data (for example, image data, audio data, voice communication data, and video communication data) to be processed by the processor 902 and modules in the electronic device 900, and can be implemented by FLASH or Random Access Memory (RAM).

[0194] It should be noted that the above description of the storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application.

[0195] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the execution order, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above sequence number of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0196] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or apparatus including the element.

[0197] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The above described device embodiments are merely exemplary. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices, or units, and can be electrical, mechanical, or in other forms.

[0198] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on multiple network units; and some or all of the units can be selected as needed to achieve the purposes of the embodiments.

[0199] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; and the integrated unit can be implemented in the form of hardware or hardware plus software functional units.

[0200] Those of ordinary skill in the art can understand that all or part of the steps of the above-described method embodiments can be completed by a program instructing related hardware, and the aforementioned program can be stored in a computer readable storage medium, and when the program is executed, the steps of the method embodiments are executed.

[0201] Alternatively, when the integrated units of the present application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for making an electronic device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: mobile storage devices, ROM, magnetic disks, or optical disks, and various other media that can store program codes.

[0202] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0203] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0204] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0205] The above merely provides a method for implementing the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A resource scheduling method applied to a Kubernetes cluster, the method comprising: Determine the changing scheduling policies in the Kubernetes cluster; Based on the changed scheduling policy, update the set of scheduling policies cached in each node of the Kubernetes cluster; The deployment strategy of the newly created Pod in the Kubernetes cluster is obtained based on the newly created Pod annotation, wherein the Pod annotation includes at least the Pod identifier and the deployment strategy; Based on the deployment strategy, the target scheduling strategy is determined from the updated set of scheduling strategies; Based on the target scheduling strategy and the annotation of each node in the node set of the Kubernetes cluster, the target node is determined in the node set of the Kubernetes cluster. The node annotation includes at least: node hardware availability information, service performance information and scheduling performance information. Deploy the newly created Pod to the target node.

2. The method as described in claim 1, wherein the updated scheduling strategy includes at least one of the following: a first scheduling strategy based on node performance, a second scheduling strategy based on call relationship, and a third scheduling strategy based on the node performance and the call relationship; The step of determining the target scheduling strategy from the updated scheduling strategy set based on the deployment strategy includes: The deployment strategy is determined to be a deployment strategy based on the node performance; The first scheduling strategy is determined as the target scheduling strategy; The deployment strategy is determined to be a deployment strategy based on the call relationship; The second scheduling strategy is determined as the target scheduling strategy; The deployment strategy is determined to be a deployment strategy based on the node performance and the call relationship; The third scheduling strategy is determined as the target scheduling strategy.

3. The method as described in claim 2, wherein determining the target node in the node set of the Kubernetes cluster based on the target scheduling strategy includes: The candidate node set is determined based on the resource information of each node in the Kubernetes cluster. Based on the first scheduling strategy, obtain the hardware availability information, service performance information and scheduling performance information of each node in the candidate node set; Based on the hardware availability information, service performance information, and scheduling performance information of each node in the candidate node set, each node is scored to obtain a node score ranking. Based on the node scoring and ranking, the target node is determined from the set of candidate nodes.

4. The method of claim 3, further comprising: Based on the second scheduling strategy, obtain the call score of the newly created Pod associated with each node in the candidate node set; Based on the call score associated with each node in the candidate node set for the newly created Pod, each node is scored to obtain the node score ranking.

5. The method of claim 3, further comprising: Based on the third scheduling strategy, obtain the hardware availability information, service performance information, scheduling performance information, and call score associated with the newly created Pod for each node in the candidate node set; Based on the hardware availability information, service performance information, scheduling performance information, and call score associated with the newly created Pod, each node in the candidate node set is scored to obtain the node score ranking.

6. The method as described in claim 3, wherein obtaining the hardware availability information, service performance information, and scheduling performance information of each node in the candidate node set includes: The hardware availability information is determined based on the node's memory usage, CPU usage, and disk usage. The service performance information is determined based on the service throughput and service response latency of the node. The scheduling performance information is determined based on the Pod deployment time and the number of Pods bound to the node.

7. The method as described in claim 4, wherein obtaining the call score associated with the newly created Pod for each node in the candidate node set includes: Identify at least one service that is associated with the newly created Pod and the call traffic corresponding to each service; Determine the node where each service is deployed; Based on the nodes deployed for each service and the call traffic corresponding to each service, the call score associated with the newly created Pod for each node in the candidate node set is determined.

8. The method of claim 1, wherein determining the changing scheduling policy in the Kubernetes cluster includes: Monitor the scheduling policy of the Kubernetes cluster to identify any changes in the scheduling policy within the Kubernetes cluster.

9. A resource scheduling device, the device comprising: The first determining module is used to determine the changing scheduling policies in the Kubernetes cluster; An update module is used to update the set of scheduling policies cached in each node of the Kubernetes cluster based on the changed scheduling policies. The acquisition module is used to obtain the deployment strategy of the newly created Pod in the Kubernetes cluster based on the newly created Pod annotation. The Pod annotation includes at least the Pod identifier and the deployment strategy. The second determining module is used to determine the target scheduling strategy from the updated set of scheduling strategies based on the deployment strategy. The third determining module is used to determine the target node in the node set of the Kubernetes cluster based on the target scheduling strategy and the annotation of each node in the node set of the Kubernetes cluster. The node annotation includes at least: node hardware availability information, service performance information and scheduling performance information. The deployment module is used to deploy the newly created Pod to the target node.

10. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1 to 8.

11. A storage medium storing executable instructions for causing a processor to execute, thereby implementing the steps of the method according to any one of claims 1 to 8.

Images