Resource scheduling method and device, electronic equipment and storage medium

Abstract

The invention provides a resource scheduling method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the division processing of a to-be-selected resource node in a resource node cluster according to a partition label corresponding to a partition configuration operation in response to the partition configuration operation, and obtaining a divided to-be-selected resource partition; in response to the restriction information configuration operation, establishing a corresponding to-be-selected resource pool for each to-be-selected resource partition according to the configured restriction information; when the first working load is received, distributing a corresponding first resource pool for the first working load according to a resource pool identifier contained in the first working load; wherein the first workload comprises a first task or a plurality of first container groups; and scheduling the first resource node for the first workload according to the real-time load information of each to-be-selected resource node in the to-be-selected resource partition corresponding to the first resource pool. The resource allocation strategy can be automatically adjusted according to the real-time load, and it is ensured that the system can still operate efficiently when the load changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource scheduling, in particular to a resource scheduling method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the demand for computing resources for AI training and inference tasks is increasing rapidly, especially relying on heterogeneous acceleration resources such as GPU and TPU. These resources are characterized by scarcity, high cost and difficulty in fragmentation, unlike CPU resources that are easy to schedule, which puts higher requirements on the scheduling system.

[0003] Currently, Kubernetes is the mainstream platform for container orchestration, and its default scheduler kube-scheduler performs well in general scenarios, but has obvious limitations in AI training and inference scenarios: on the one hand, it takes a single Pod as the scheduling unit, which is difficult to adapt to the collaborative scheduling needs of a group of associated Pods in AI tasks, and only refers to resource application conditions (request / limit), which can easily lead to excessive resource reservation and waste; on the other hand, some solutions that support real-time load scheduling are also limited to CPU and memory, ignoring GPU load information, which is the core of AI tasks, and cannot meet the heterogeneous resource scheduling needs. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a resource scheduling method, device, electronic equipment and storage medium to overcome the problems in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a resource scheduling method, which comprises: In response to a partition configuration operation, dividing the candidate resource nodes in the resource node cluster according to the partition label corresponding to the partition configuration operation to obtain the divided candidate resource partitions; In response to a limit information configuration operation, establishing a corresponding candidate resource pool for each of the candidate resource partitions according to the configured limit information; wherein the candidate resource pool represents the resource scheduling range and resource usage quota of the candidate resource; When a first workload is received, assigning a corresponding first resource pool to the first workload according to the resource pool identifier contained in the first workload; wherein the first workload includes a first task or a plurality of first container groups; According to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool, scheduling a first resource node for the first workload.

[0006] In some technical solutions of the present application, the first workload is obtained by the above method in the following way: in response to the first task creation operation, constructing the first workload; or in response to a container group creation operation, generating a corresponding target container group; obtaining the first workload by the target container group and an associated container group from the first container group; wherein the associated container group is a container group having an association relationship with the target container group.

[0007] In some technical solutions of the present application, the above method further comprises: in response to a priority level setting operation, setting a corresponding priority level for the first workload.

[0008] In some technical solutions of the present application, the above method further comprises: when the first resource pool can meet the scheduling requirements of the first workload, selecting a candidate resource node with the lowest resource utilization from the candidate resource nodes in the first resource pool corresponding to the candidate resource partition as the first resource node.

[0009] In some technical solutions of the present application, the above method further comprises: when the first resource pool cannot meet the scheduling requirements of the first workload, releasing resources in the first resource pool according to the first priority level corresponding to the first workload; after the resource release, selecting a candidate resource node with the lowest resource utilization from the candidate resource nodes in the first resource pool corresponding to the candidate resource partition as the first resource node.

[0010] In some technical solutions of the present application, the above method further comprises: determining a to-be-released container group in the first resource pool that is lower than the first priority level; releasing resources in order of priority of the to-be-released container group until the first resource pool can meet the scheduling requirements of the first workload.

[0011] In some technical solutions of the present application, the above method further comprises: when a second workload is received, allocating a corresponding second resource pool for the second workload according to a resource pool identifier contained in the second workload; wherein the second workload includes a second task or a plurality of second container groups; According to real-time load information of each candidate resource node in the candidate resource partition corresponding to the second resource pool, a second resource node is scheduled for the second work load.

[0012] In a second aspect, an embodiment of the present application provides a resource scheduling apparatus, which comprises: A first response module is configured to, in response to a partition configuration operation, divide candidate resource nodes in a resource node cluster according to a partition label corresponding to the partition configuration operation, to obtain divided candidate resource partitions. A second response module is configured to, in response to a limit information configuration operation, establish a corresponding candidate resource pool for each candidate resource partition according to configured limit information, wherein the candidate resource pool represents a resource scheduling range and a resource usage quota of the candidate resource. An allocation module is configured to, when a first work load is received, allocate a corresponding first resource pool for the first work load according to a resource pool identifier contained in the first work load, wherein the first work load comprises a first task or a plurality of first container groups. A scheduling module is configured to, according to real-time load information of each candidate resource node in the candidate resource partition corresponding to the first resource pool, schedule a first resource node for the first work load.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements steps of the above-mentioned resource scheduling method when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement steps of the above-mentioned resource scheduling method.

[0015] The technical scheme provided by the embodiments of the present application can have the following beneficial effects: The method comprises: in response to a partition configuration operation, performing partition processing on candidate resource nodes in a resource node cluster according to a partition label corresponding to the partition configuration operation, to obtain a partitioned candidate resource partition; in response to a restriction information configuration operation, establishing a corresponding candidate resource pool for each of the candidate resource partitions according to configured restriction information; wherein the candidate resource pool represents a resource scheduling range and a resource usage quota of the candidate resource; upon receiving a first workload, allocating a corresponding first resource pool for the first workload according to a resource pool identifier contained in the first workload; wherein the first workload comprises a first task or a plurality of first container groups; and scheduling a first resource node for the first workload according to real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool. The resource allocation strategy can be automatically adjusted according to real-time load, to ensure that the system can still operate efficiently when the load changes.

[0016] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be referred to, as follows. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0018] Figure 1 A flowchart of a resource scheduling method provided by an embodiment of the present application is shown; Figure 2 A schematic diagram of a first embodiment provided by an embodiment of the present application is shown; Figure 3 A schematic diagram of a second embodiment provided by an embodiment of the present application is shown; Figure 4 A schematic diagram of extracting real-time load information provided by an embodiment of the present application is shown; Figure 5 A schematic diagram of a third embodiment provided by an embodiment of the present application is shown; Figure 6 A schematic diagram of a fourth embodiment provided by an embodiment of the present application is shown; Figure 7 A schematic diagram of a resource scheduling device provided by an embodiment of the present application is shown; Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and demonstration, and should not be used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual scale. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0020] In addition, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] With the rapid development of artificial intelligence technology, the demand for computing resources, data management, and deployment flexibility is also growing. In this context, Kubernetes, as the leader in the field of container orchestration, is gradually becoming the core infrastructure for hosting AI applications with its powerful resource management, automated deployment, and elastic scaling capabilities. AI applications (such as model training) often require a large amount of computing resources, especially relying on heterogeneous acceleration resources such as GPUs and TPUs, which are scarce, expensive, and cannot be easily fragmented, unlike CPUs that are easy to schedule. In order to meet the complex scheduling needs of AI applications, the scheduling system of Kubernetes is evolving from a single scheduler mode to a multi-scheduler, multi-scenario coexistence architecture. The default scheduler kube-scheduler is mature in function, suitable for most general scenarios, and supports extending scheduling policies through a plugin mechanism, which is the core scheduling component in current mainstream production environments. In essence, kube-scheduler is designed for general-purpose applications, and for task scheduling in AI training environments, it needs to rely more on GPU computing resources for scheduling, and kube-scheduler is a bit "handicapped".

[0023] The general-purpose scheduler usually only supports one aspect in the AI training environment, for example, the default scheduler only refers to the resource request / limit conditions in the scheduling indicators, which often leads to excessive reservation of resources when applying, resulting in resource waste. Some schedulers can also schedule according to real-time load, but they mostly only consider CPU and memory load conditions, ignoring GPU load, and have failed to meet the scheduling needs of AI environments. At the same time, in the AI training environment, the urgency of training and inference tasks varies, and in the case of limited GPU resources, how to make tasks run smoothly, i.e., high-priority tasks are given priority, and low-priority tasks can be preempted as needed. Although some schedulers have realized priority scheduling, the implementation method is often to arbitrarily evict low-priority tasks, although it meets the priority of high-priority tasks, it will cause low-priority tasks to be unable to run stably.

[0024] Based on this, the embodiments of the present application provide a resource scheduling method, device, electronic equipment and storage medium, which can automatically adjust the resource allocation strategy according to the real-time load to ensure that the system can still run efficiently when the load changes. At the same time, according to the task priority and the real-time state of the resource, the computing resource is dynamically divided to realize the rapid response of high-priority tasks, while ensuring the stable operation of low-priority tasks. The following will be described through embodiments.

[0025] Figure 1A flowchart of a method for resource scheduling is shown, wherein the method comprises steps S101-S104; specifically: S101, in response to the partition configuration operation, performing partitioning processing on the candidate resource nodes in the resource node cluster according to the partition label corresponding to the partition configuration operation, to obtain the partitioned candidate resource partitions; S102, in response to the restriction information configuration operation, establishing a corresponding candidate resource pool for each of the candidate resource partitions according to the configured restriction information; wherein the candidate resource pool represents the resource scheduling range and resource usage quota of the candidate resource; S103, upon receiving a first workload, allocating a corresponding first resource pool for the first workload according to the resource pool identifier contained in the first workload; wherein the first workload includes a first task or a plurality of first container groups; S104, according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool, scheduling a first resource node for the first workload.

[0026] Some embodiments of the present application will be described in detail below. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] The embodiments of the present application provide a method for resource scheduling, which is based on Kubernetes. Kubernetes (K8s for short) is an open-source container orchestration platform for automating container deployment, expansion and management. It aims to provide a cross-host cluster container scheduling, orchestration, high availability and automated platform, so that users can more easily manage the life cycle of applications.

[0028] The existing Kubernetes contains multiple core components, mainly including: kube-apiserver: as the front-end interface of the entire system, it is responsible for handling requests from users and other components, providing RESTful API services, and is the hub of communication between components.

[0029] etcd: a distributed key-value storage system, used to save all configuration information of the Kubernetes cluster, and is the database of the cluster.

[0030] kube-scheduler: responsible for scheduling cluster resources, scheduling pods to appropriate nodes for running according to predetermined scheduling strategies.

[0031] kube-controller-manager: runs a series of controller processes such as node controller, replica controller, endpoint controller, etc., responsible for maintaining the state of the cluster, ensuring that the actual state is consistent with the desired state.

[0032] kubelet: a proxy program running on each node, responsible for maintaining the life cycle of containers, ensuring that containers run according to the Pod specification.

[0033] kube-proxy: running on each node, responsible for maintaining network rules on the node, implementing communication between Pods and between Pods and external networks.

[0034] In addition, Kubernetes also defines a series of abstract concepts, such as Pod (the smallest deployment unit, containing one or more containers), Service (providing stable network access entry for Pods), Deployment (declarative definition of Pod and ReplicaSet, used to manage stateless applications), StatefulSet (used to manage stateful applications, providing stable network identity and storage for Pods), ConfigMap and Secret (used for configuration management, storing application required configuration information and sensitive data), etc., which together constitute the core function system of Kubernetes.

[0035] Kubernetes as a resource node cluster, including a plurality of candidate resource nodes, Kubernetes manages and controls the nodes through its core components (such as kubelet, kube-proxy, etc.). Specifically, the Kubernetes cluster is composed of multiple nodes, which serve as actual computing resource carriers, running containerized applications (such as Pods).

[0036] In the prior art, when Kubernetes performs scheduling, it will have an impact on all resources of Kubernetes. In order to avoid the impact on all resources of Kubernetes during resource scheduling, before scheduling the resources, the embodiments of the present application first perform partitioning processing on each candidate resource node in Kubernetes, as shown in Figure 2

[0037] ​In this embodiment, the partitioning of Kubernetes candidate resource nodes is based on user-defined criteria. Specifically, users can set corresponding partition labels for each candidate resource node according to the specific needs of their business system, and then divide the candidate resource nodes into multiple candidate resource partitions based on these labels. These partition labels can be set according to the attributes of the candidate resource nodes or set by the user. For example, CPU and GPU labels can be set to divide the system into CPU partitions, or CPU1 and CPU2 can be set to divide the system into first and second CPU partitions. Labels can also be set according to the purpose of the nodes, dividing them into Ascend partitions, compute partitions, elastic partitions, etc. An Ascend partition generally refers to a specific area allocated for related computing tasks of the Ascend AI processor in a system using an Ascend processor. A compute partition typically refers to an area specifically used for handling regular computing tasks. An elastic partition is a partitioning method that dynamically adjusts resource allocation based on system load and demand. By operating on partitions, it is possible to avoid the impact of resource scheduling in the current partition on other partitions. For example, resource scheduling in the first partition will not affect the resources in the second partition.

[0038] After partitioning, to avoid resource waste in each partition, this embodiment of the application also sets up a candidate resource pool for each candidate resource partition. Here, the candidate resource pool represents the resource scheduling range and resource usage limit of the candidate resources. That is, when a user uses resources, they can only use them within the restrictions of the corresponding resource pool, avoiding the situation where they occupy all or most of the resources, leaving no resources available for other users.

[0039] When setting up a candidate resource pool for a candidate resource partition, the setting basis in this embodiment is also user-defined. Users set corresponding restriction information for different candidate resource partitions based on the needs of their specific business systems, and construct the candidate resource pool corresponding to that partition based on the restriction information. In specific implementation, the restriction information here can be a specific threshold. For example, the restriction information could be 5c, 16G, 2*8G GPU (5c means the resource pool can use the computing resources of 5 CPU cores in the corresponding partition; 16G means the resource pool can use 16GB of memory resources in the corresponding partition; 8G means the video memory capacity of a single GPU is 8GB; "2*" means the resource pool can use 2 such GPUs in the corresponding partition, i.e., a total of 16GB of GPU resources).

[0040] Create corresponding resource pools on top of the partitions. The resource pools are configured with the quota information of the partitions. For example, a shared resource pool contains quotas of 5c, 16G, and 2*8G GPUs for elastic partitions.

[0041] When partitioning and resource pool establishing, the following steps can be taken: First, define two third-party resources Partition and ResourcePool. Partition represents a partition, which is used to organize multiple nodes into a partition. ResourcePool is used to divide the resources of the partition. As shown in Figure 3 , a task (workload) specifies a resource pool, which references the relevant partition, and the partition references the relevant nodes according to nodeLabel. Thus, the task can only be deployed within the relevant partition range.

[0042] Custom resources are shown as follows: apiVersion: gientech.com / v1beta1 kind: Partition metadata: name: "gpu-partition" spec: nodeLabels: node-type: node-910b nodeLabels specifies the labels of the nodes.

[0043] apiVersion: gientech.com / v1beta1 kind: ResourcePool metadata: name: "share-pool" spec: preemption: policy: LowerPriority / Never resources: - name: "gpu-partition" resources: - name: "cpu" quota: 6 - name: "memory" quota: 12Gi - name: "gpu-vendor.example / example-gpu" quota: 4 The resource pool includes a resources array, each element of which refers to the usage quota information (limit information) of the partition. The above example shows that the gpu-partition partition uses 6 CPUs, 12 Gi of memory, and 4 GPUs. The quota information of the resource pool ensures the reasonable allocation of the cluster and prevents malicious plunder of cluster resources. A preemption attribute is also included, which specifies whether the current resource pool enables preemption, i.e., when resources are insufficient, low-priority tasks are preempted by high-priority tasks.

[0044] After the above configuration, Kubernete can start resource scheduling. The start of resource scheduling is to receive any workload (Workload). For ease of description, the first workload and the second workload are distinguished in the embodiment of the present application. It should be noted that the workload in the embodiment of the present application includes a task (job) or a plurality of pod groups (pods), rather than the prior art which processes a single pod as an object. The pod (container group) is the smallest unit of scheduling, which is composed of one or more closely related containers. These containers share network, storage and other resources, and cooperatively complete specific functions. In the AI scenario, a pod can correspond to a specific execution instance in a training or inference task, such as a service node for an inference task or a computing node for a training task. Multiple related pods are organized into the same Workload through labels, annotations and other methods to jointly complete complex AI tasks (such as training or inference). The job (task) is an abstract concept for managing "one-time tasks", mainly for short-term running and terminating tasks after completion. In the AI scenario, a job usually corresponds to a complete training task (such as a model training process), and its goal is to execute computation through a group of pods, and the pods are terminated after the task is completed. The Workload can be understood as a task (job) or a group of related pods, i.e., when the Workload corresponds to a single-execution training task, it can be regarded as a job. The job and the pod are both components of the Workload, and the unified scheduling, resource allocation and priority management are implemented through the Workload to adapt to the scene requirements of AI training and inference integration.

[0045] The specified resource pool and priority level are included in the work load. When the resource pool is specified, the resource pool identifier can be used, that is, the resource pool identifier of the corresponding resource pool is set in the work load. When the first work load is received, the corresponding first resource pool is allocated to the first work load according to the resource pool identifier contained in the first work load. The priority level can be set by the user or determined according to the priority level identifier contained in the work load. For example, the priority level includes a first priority level, a second priority level, and a third priority level (the first priority level is higher than the second priority level, and the second priority level is higher than the third priority level), or includes an A level, a B level, and a C level (the A level is higher than the B level, and the B level is higher than the C level). The priority level is the basis for subsequent resource preemption and the basis for the scheduling order.

[0046] In an optional embodiment, the process of generating the first work load can be to construct the first work load in response to a first task creation operation, or to generate a corresponding target container group in response to a container group creation operation; the target container group and an associated container group are used to obtain the first work load as the first container group; wherein the associated container group is a container group having an association relationship with the target container group. In response to a priority level setting operation, the corresponding priority level is set for the first work load.

[0047] After determining the first resource pool corresponding to the first work load, the corresponding first resource node needs to be scheduled for the first work load to use the first resource node to process the first work load. When determining the first resource node, the determination range of the embodiment of the present application is each candidate resource node in the candidate resource partition corresponding to the first resource pool. By obtaining the real-time load information of each candidate resource node in the candidate resource partition corresponding to the first resource pool, then comparing all the real-time load information, the first resource node processing the first work load is determined.

[0048] The real-time load information of the candidate resource node is realized by a custom CRD nodemetric. The embodiment of the present application will start a daemonset, that is, an exporter application is deployed on each candidate resource node to collect the real-time load information of the candidate resource node, and then a custom nodemetric controller is used to obtain the real-time load information of the candidate resource node and store it into the kube-apiserver. For example, Figure 3 The green line part in the figure, or Figure 4 As shown in the steps: Step 1: The node load controller obtains the real-time load information of each candidate resource node in the candidate resource partition corresponding to the first resource pool from the kube-apiserver; Step 2: Edit the node, get the real-time load information of each candidate resource node in the candidate resource partition corresponding to the first resource pool from the metric-exporter of the node. The metric-exporter of the node will store some index information within a certain time range, and the node load controller can directly read this information without needing to calculate it by itself.

[0049] Step 3: The node load controller stores the real-time load information of the node into the kube-apiserver.

[0050] The third-party resource introduced is as follows: apiVersion: gientech.com / v1alpha1 kind: NodeMetric metadata: name: node1 spec: nodeMetric: - duration: 5m usage: p50: resources: cpu: 2671m memory: 10194404Ki gpu: 20194204Ki p90: resources: cpu: 3514m memory: 10214100Ki gpu: 30224130Ki p95: resources: cpu: 3617m memory: 10219260Ki gpu: 10114100Ki p99: resources: cpu: 4194m memory: 10232516Ki gpu: 11214120Ki In an optional embodiment, when scheduling the first resource node for the first workload, two situations can occur: the first resource pool can meet the scheduling requirement of the first workload and the first resource pool cannot meet the scheduling requirement of the first workload.

[0051] For the situation that the first resource pool can meet the scheduling requirement of the first workload, the first resource node can be directly selected from the candidate resource nodes in the candidate resource partition corresponding to the first resource pool. The candidate resource node with the lowest resource usage is selected as the first resource node. Then, the first resource node is used to process the first workload.

[0052] For the situation that the first resource pool cannot meet the scheduling requirement of the first workload, resource preemption is needed, that is, some resources in the first resource pool are evicted to make the first resource pool meet the scheduling requirement of the first workload after the resources are released. The basis of the resource preemption is the first priority level corresponding to the first workload. The resource evictions of the first resource pool are the resources with a priority level lower than the first priority level. After the resources are released, the first resource node is selected from the candidate resource nodes in the candidate resource partition corresponding to the first resource pool. The specific evictions are as follows: determine the to-be-released container groups (pods) in the first resource pool with a priority level lower than the first priority level; release the resources in the to-be-released container groups in order of the priority level, until the first resource pool meets the scheduling requirement of the first workload.

[0053] For example, the resources with a priority level lower than the first priority level include pods with a second priority level and pods with a third priority level (the second priority level is higher than the third priority level). When the resources are released, the pods with the third priority level are released first. If the first resource pool still cannot meet the scheduling requirement of the first workload after the pods with the third priority level are released, the pods with the second priority level are released, until the first resource pool meets the scheduling requirement of the first workload.

[0054] In an optional embodiment, when implemented, the following method can be used: Figure 5 ​First, the embodiment of the present application designs a Workload CRD resource, and the scheduler takes workload as the basic scheduling unit. It can be understood that workload is a task (job) or for AI application, it is a group of related pods. The workload has a corresponding node load controller workload-controller. The priority of workload determines whether the load will be preempted for resources. When the resources are insufficient, the tasks in the same resource pool can preempt each other's resources, and the high-priority task can preempt the resources of the low-priority task. This design meets the needs of the AI training and pushing scene. When a low-priority training task occupies the resources of the resource pool, at this time a high-priority inference task is submitted to the resource pool, the inference task will have the right to use the resources preferentially. When the resource pool is sufficient, the inference task is directly allocated resources. When the resource pool is insufficient, the inference task will sort all tasks with lower priority than it, and expel them from low to high priority until the high-priority inference task can be deployed.

[0055] For example, step 1: the user creates a pod; Step 2: workload-controller monitors the unscheduled pods in the cluster; Step 3: Check if the unscheduled pod has a related affiliation, such as deployment, statefulset, etc. Step 4: Create related workload resources according to the affiliation of the pod. Different pods can be organized into the same workload through labels, annotations, etc. And save it to kube-apiserver for the scheduler to use.

[0056] The workload definition is roughly as follows: apiVersion: kueue.x-k8s.io / v1beta1 kind: Workload metadata: name: wl-1 namespace: default spec: priority: 0 resourcePool: share-pool pods: - count: 1 name: test-1 template: metadata: {} spec: ... - count: 2 name: test-2 template: metadata: {} spec: ... The specific scheduling process can also be carried out in the following manner: Figure 6 Step 1: The user submits a pod. Step 2: The workload-controller creates a workload according to the above description. Step 3: The kube-apiserver saves the workload.

[0057] Step 4: The scheduler listens to the newly created workload and adds it to the queue. The entire scheduler queue has only one queue, and the queue contains newly added workloads to be scheduled and pods to be scheduled after being evicted. The queue is sorted according to priority to ensure that high-priority workloads can be scheduled first.

[0058] Step 5: The scheduler gets the highest-priority workload from the queue for scheduling.

[0059] Step 6: According to the resource pool referenced in the workload.

[0060] Step 7: Traverse the partition referenced by the resource pool to get a partition information. If there is no partition to try, stop this scheduling.

[0061] Step 8: Get the partition information from the kube-apiserver. This step can check the query times through the cache listening method.

[0062] Step 9: Determine whether the current partition quota can deploy the pod in the workload. If yes, go to step 13, otherwise continue to the next step of preemption. Step 10: Check whether the current resource pool supports preemption. If not, jump to step 7 to traverse the next partition. Otherwise, continue to the next step.

[0063] Step 11: Check the low-priority tasks in the resource pool. Sort the low-priority tasks and evict the low-priority tasks.

[0064] ​Step 12: Determine whether there are enough available resources to schedule the current pod. When the remaining resources of the resource pool are sufficient, proceed to the next step. Otherwise, jump to step 7 to iterate the next partition.

[0065] Step 13: Set the nodeselector of the pod to the label of the partition to ensure that the pod remains in this partition when performing other scheduling in the future.

[0066] Step 14: Obtain real-time load information of nodes in the partition, and obtain real-time information of nodes in the partition from the kube-apiserver. Step 15: According to the real-time load information and other information such as topology, affinity, and scheduling rules, select a node with the highest score as the scheduling node. Set the nodeName of the pod to this node, and continue to iterate the other pods in the workload.

[0067] In an optional embodiment, the embodiments of the present application can also simultaneously or sequentially receive a second workload. The process for processing the second workload is the same as the process for processing the first workload, which will not be described here. According to specific actual business requirements, the second resource pool here can be the same as the first resource pool, or can be different from the first resource pool. When the second resource pool is different from the first resource pool, the two resource pools do not affect each other when performing resource scheduling.

[0068] Figure 7 A structure diagram of a resource scheduling device provided by an embodiment of the present application is shown, and the device comprises: A first response module is configured to divide and process the candidate resource nodes in the resource node cluster according to the partition label corresponding to the partition configuration operation in response to the partition configuration operation, to obtain the divided candidate resource partitions. A second response module is configured to establish a corresponding candidate resource pool for each of the candidate resource partitions according to the configured limit information in response to the limit information configuration operation. The candidate resource pool represents the resource scheduling range and resource usage quota of the candidate resources. An allocation module is configured to allocate a corresponding first resource pool for the first workload according to the resource pool identifier included in the first workload when the first workload is received. The first workload includes a first task or a plurality of first container groups. A scheduling module is configured to schedule a first resource node for the first workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool.

[0069] The first workload is obtained by the following method: in response to a first task creation operation, constructing the first workload; or in response to a container group creation operation, generating a corresponding target container group; obtaining the first workload by the target container group and an associated container group from the first container group; wherein the associated container group is a container group having an association relationship with the target container group.

[0070] a third response module configured to set a corresponding priority level for the first workload in response to a priority level setting operation.

[0071] The first resource node is scheduled for the first workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool, including: When the first resource pool can meet the scheduling requirements of the first workload, the candidate resource node with the lowest resource usage rate among the candidate resource nodes in the candidate resource partition corresponding to the first resource pool is selected as the first resource node.

[0072] When the first resource pool cannot meet the scheduling requirements of the first workload, the first resource pool is released according to the first priority level corresponding to the first workload; After resource release, the candidate resource node with the lowest resource usage rate among the candidate resource nodes in the candidate resource partition corresponding to the first resource pool is selected as the first resource node.

[0073] The first resource pool is released according to the first priority level corresponding to the first workload, including: determining the to-be-released container group in the first resource pool that is lower than the first priority level; According to the priority level of the to-be-released container group, resource release is performed in turn until the first resource pool can meet the scheduling requirements of the first workload.

[0074] The allocation module is configured to, when receiving a second workload, allocate a corresponding second resource pool for the second workload according to the resource pool identifier contained in the second workload; wherein the second workload includes a second task or a plurality of second container groups; The scheduling module is configured to schedule a second resource node for the second workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the second resource pool.

[0075] As Figure 8As shown, the embodiment of the present application provides an electronic device for executing the method of resource scheduling in the present application, the device comprises a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to realize the steps of the method of resource scheduling.

[0076] Specifically, the memory and the processor can be general memory and processor, which are not specifically limited herein, and when the processor executes the computer program stored in the memory, the method of resource scheduling can be executed.

[0077] Corresponding to the method of resource scheduling in the present application, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the method of resource scheduling.

[0078] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc., and the computer program on the storage medium can execute the method of resource scheduling when executed.

[0079] In the embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division, and another division manner can be used in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces. The coupling or communication connection can be electrical, mechanical or in other forms.

[0080] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, which can be located in one place, or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0081] In addition, each functional unit in the embodiments provided in the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0082] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0083] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0084] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. These modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of resource scheduling, the method comprising: The method comprises: in response to a partition configuration operation, performing division processing on the candidate resource nodes in the resource node cluster according to the partition label corresponding to the partition configuration operation, to obtain the divided candidate resource partitions; in response to a limit information configuration operation, establishing a corresponding candidate resource pool for each of the candidate resource partitions according to the configured limit information; wherein the candidate resource pool represents the resource scheduling range and resource usage quota of the candidate resource; when a first workload is received, allocating a corresponding first resource pool for the first workload according to the resource pool identifier contained in the first workload; wherein the first workload includes a first task or a plurality of first container groups; scheduling a first resource node for the first workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool.

2. The method of claim 1, wherein, The method obtains the first workload in the following way: in response to a first task creation operation, constructing the first workload; or in response to a container group creation operation, generating a corresponding target container group; obtaining the first workload by taking the target container group and an associated container group as the first container group; wherein the associated container group is a container group having an association relationship with the target container group.

3. The method of claim 2, wherein, The method further comprises: in response to a priority level setting operation, setting a corresponding priority level for the first workload.

4. The method of claim 1, wherein, The scheduling of a first resource node for the first workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool comprises: when the first resource pool can meet the scheduling requirements of the first workload, selecting a candidate resource node with the lowest resource usage rate among the candidate resource nodes in the candidate resource partition corresponding to the first resource pool as the first resource node.

5. The method of claim 4, wherein, The method further comprises: when the first resource pool cannot meet the scheduling requirements of the first workload, releasing resources from the first resource pool according to the first priority level corresponding to the first workload; after the release of resources, selecting a candidate resource node with the lowest resource usage rate among the candidate resource nodes in the candidate resource partition corresponding to the first resource pool as the first resource node.

6. The method of claim 5, wherein, The release of resources from the first resource pool according to the first priority level corresponding to the first workload comprises: determining the to-be-released container groups in the first resource pool that are lower than the first priority level; releasing resources in turn according to the priority levels of the to-be-released container groups until the first resource pool can meet the scheduling requirements of the first workload.

7. The method of claim 1, wherein, The method further comprises: when a second workload is received, allocating a corresponding second resource pool for the second workload according to the resource pool identifier contained in the second workload; wherein the second workload includes a second task or a plurality of second container groups; scheduling a second resource node for the second workload according to the real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the second resource pool.

8. An apparatus for resource scheduling, the apparatus comprising: The device comprises: A first response module, configured to, in response to a partition configuration operation, perform partition processing on candidate resource nodes in a resource node cluster according to a partition label corresponding to the partition configuration operation, to obtain partitioned candidate resource partitions; A second response module, configured to, in response to a limit information configuration operation, establish a corresponding candidate resource pool for each of the candidate resource partitions according to the configured limit information; wherein the candidate resource pool represents a resource scheduling range and a resource usage quota of the candidate resource; An allocation module, configured to, when a first workload is received, allocate a corresponding first resource pool for the first workload according to a resource pool identifier contained in the first workload; wherein the first workload comprises a first task or a plurality of first container groups; A scheduling module, configured to, according to real-time load information of each of the candidate resource nodes in the candidate resource partition corresponding to the first resource pool, schedule a first resource node for the first workload.

9. An electronic device, comprising: Comprise: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the resource scheduling method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program is executed by the processor to execute the steps of the resource scheduling method in any one of claims 1 to 7.

Images