A batch task scheduling method based on Kubernetes

Abstract

The present invention discloses a batch task scheduling method based on Kubernetes, and belongs to the field of cloud computing technology. The method of the present invention first allocates new tasks to ordinary task queues or batch task queues according to task labels; then, based on the optimization strategy, batch tasks are preferentially selected as tasks to be executed; if the tasks to be executed are batch tasks, they are scheduled to be executed on a single node or multiple adjacent nodes that meet their resource requirements according to the resources required by the batch tasks; if the batch tasks are MPI tasks, a service service is created for the pod of the MPI task to obtain the IP of the MPI task pod; then an SSH key pair is generated, and the public key is transmitted to each pod to realize SSH password-free login. The method of the present invention solves the problem that the dynamic scheduling of containers in Kubernetes causes the MPI application to be unable to establish static SSH communication.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing, and more specifically, relates to a batch task scheduling method based on Kubernetes. Background Art

[0002] Kubernetes is a container orchestration platform widely used in cloud-native systems. It can automatically deploy, scale, and manage containerized applications in units of Pods, decoupling applications from the platform. Kubernetes starts containers and starts running services in sequence based on the containers' requests for resources such as CPU and memory. However, this scheduling method does not support batch task scheduling.

[0003] Batch tasks refer to a group of tasks that are automatically processed and executed by the system by submitting multiple jobs of the same type to the job queue at one time. High-performance computing tasks use MPI (Message Passing Interface) as a message passing mechanism to implement parallel computing in distributed systems, which is a typical batch task. At present, Kubernetes' scheduling support for MPI parallel tasks is still imperfect, and it cannot perceive the overall scheduling of the job, and lacks a scheduling strategy based on the job as a whole. In addition, current cloud computing systems often adopt heterogeneous computing architectures, which are equipped with heterogeneous computing resources such as GPUs and FPGAs in addition to CPU resources. For heterogeneous parallel applications, in addition to implementing the scheduling of the entire job, specific computing tasks must be scheduled to run on nodes with certain types of heterogeneous resources. At present, Kubernetes has not optimized the scheduling of heterogeneous resources. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a batch task scheduling method based on Kubernetes, which aims to solve the technical problem that Kubernetes has insufficient support for scheduling parallel computing batch tasks and heterogeneous resources in existing heterogeneous cloud native systems.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a batch task scheduling method based on Kubernetes, the method comprising:

[0006] Assign new tasks to the normal task queue or batch task queue based on the task label;

[0007] Based on the optimization strategy, batch tasks are preferentially selected as tasks to be executed;

[0008] If the task to be executed is a batch task, it is scheduled to be executed on a single node or multiple adjacent nodes that meet its resource requirements according to the resources required by the batch task;

[0009] If the batch task is an MPI task, create a service for the pod of the MPI task to obtain the IP address of the MPI task pod; then generate an SSH key pair and pass the public key to each pod to implement SSH password-free login.

[0010] Preferably, if the task to be executed is a batch task and requires heterogeneous resources, all nodes in the cluster are queued, with nodes containing the required heterogeneous resources placed in the front row, nodes containing only ordinary resources placed in the middle row, and nodes containing non-required heterogeneous resources placed in the back row. The nodes in the front row, middle row, and back row are arranged from high to low according to load; all nodes in the queue are traversed from front to back, and if a node that meets the resource requirements of the task to be executed is found, the task to be executed is scheduled on the node.

[0011] Preferably, if the task to be executed is a batch task and does not require heterogeneous resources, all nodes in the cluster are queued, with nodes containing only common resources in the front row and nodes containing heterogeneous resources in the back row. The nodes in the front and back rows are arranged from high to low according to load; all nodes in the queue are traversed from front to back, and if a node that meets the resource requirements of the task to be executed is found, the task to be executed is scheduled on the node.

[0012] Preferably, if there is no single node in the queue that can meet the resource requirements of the task to be executed, multiple adjacent nodes that can meet the resource requirements of the task to be executed are searched from the queue to jointly schedule the task to be executed.

[0013] Preferably, if the idle resources in the cluster cannot meet the resource requirements of the tasks to be executed, the tasks to be executed are returned to the task queue based on the task tags.

[0014] Preferably, the preferred strategy is: determine whether the batch task queue is empty, if so, select ordinary tasks from the ordinary task queue as tasks to be executed; otherwise, select batch tasks from the batch task queue as tasks to be executed.

[0015] Preferably, the preferred strategy is: within a unit of time, N batch tasks are selected as tasks to be executed, and M ordinary tasks are selected as tasks to be executed; wherein N / M>1.

[0016] Preferably, the tasks in the task queue comply with the first-in-first-out rule.

[0017] In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

[0018] (1) MPI is a message passing mechanism used to implement parallel computing in a distributed system. It is a typical batch task. Kubernetes' scheduling support for MPI parallel tasks is still imperfect. Due to the dynamic scheduling characteristics of containers, MPI applications cannot establish static SSH communication. The present invention obtains the IP address of the pod of the MPI task by creating a service in Kubernetes, and then transmits the SSH key to each pod to complete SSH password-free login.

[0019] (2) The default scheduling strategy of Kubernetes does not specifically optimize the scheduling scheme for heterogeneous resource devices and batch tasks. The present invention proposes a matching scheduling mechanism for heterogeneous resources, thereby quickly solving the matching scheduling problem between batch tasks and heterogeneous resources.

[0020] (3) The present invention introduces two task queues to isolate batch tasks from ordinary tasks. Users can set the priorities of the two queues to prevent node resources from being preempted by ordinary tasks and affecting the scheduling of batch tasks.

[0021] (4) Batch tasks usually involve a group of pods working together. The scheduling strategy adopted by the present invention is to schedule this group of pods to the same node or adjacent nodes to reduce communication delays and save communication overhead.

[0022] (5) Batch computing tasks often require a group of containers to work together. In this case, there may be competition for queue resources, resulting in slow task execution or even deadlock. In the scheduling method of the present invention, the idle resources of the cluster are checked before task scheduling to determine whether they meet the requirements of the task. Scheduling is performed only when the task requirements are met, thus avoiding deadlock problems and improving the utilization rate of cluster resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the composition of the scheduling system provided in an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of the composition of a scheduler provided in an embodiment of the present invention;

[0025] Figure 3 is a task scheduling flow chart provided in an embodiment of the present invention;

[0026] Figure 4 is a batch task scheduling flow chart provided in an embodiment of the present invention;

[0027] Figure 5 It is an MPI task execution flow chart provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present invention, unless otherwise specified, “multiple” means two or more than two. For example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0031] Next, the technical solution provided in the embodiment of the present invention is introduced:

[0032] Kubernetes allows users to define their own API objects, which are custom resource definitions (CRDs). Task queues are defined through CRDs. The specific task queue definitions are as follows:

[0033] The task queue is used to hold a group of job tasks. The present invention designs two task queues, a batch task queue and a common task queue. Different types of tasks are arranged in corresponding task queues to achieve isolation, and batch tasks can be quickly scheduled without being interrupted by common tasks. Since the present invention is aimed at batch task scheduling, the scheduling priority of the batch task queue is higher than that of the common task queue. The scheduler schedules in the order of scheduling two tasks in the batch task queue and scheduling one task in the common queue.

[0034] The specific scheduling system composition diagram is as follows: Figure 1 As shown in the figure: The scheduling system consists of two queues, batch tasks and common tasks, and a scheduler. On the left are tasks submitted by users, which are queued in the corresponding task queues according to the task type. Each task in the common task queue is composed of a pod, and each task in the batch task queue is completed by multiple pods working together. For example, the first task requires 4 pods to complete, and the resources required by each pod are shown in the figure. The scheduler will prioritize the batch tasks in the batch task queue.

[0035] The scheduler consists of three parts: the scheduling algorithm, the resource table of all nodes, and the resource table of task consumption. Figure 2As shown in the figure. The all-node resource table maintains the information of free resources and total resources of all nodes. The task consumption resource table records the resources consumed by each task and the node information of its scheduling, so that it is convenient to update the all-node resource table after the task is completed. The scheduling algorithm performs scheduling through the all-node resource table.

[0036] The following is an example of the implementation process of the method of the present invention. Figure 3 As shown, the following steps are included:

[0037] Step S11: The user creates a new job task and caches the jobs in the order in which they are submitted;

[0038] Step S12: Determine whether the job task type is a batch task or a common task by using the label of each job task, and assign the job task to the corresponding common task queue or batch task queue. The job task just assigned to the task queue is in a queue state, waiting to be scheduled.

[0039] Step S13: Select one of the two task queues in a customized manner, select the earliest queued job task, and prepare for scheduling.

[0040] The present invention is a scheduling method for batch tasks, so the priority of the batch task queue is higher. The selection strategy of the present invention is that the scheduler selects the queue in the order of one common task and two batch tasks;

[0041] Step S14: By calculating the idle resources of the cluster, determine whether the cluster meets the resource requirements of the job task. If yes, execute step S15; otherwise, return to step S12 and wait for the next scheduling;

[0042] Step S15: determine whether the job task is a batch task through the job task tag. If it is a batch task, execute step S16; if it is a common task, execute step S17;

[0043] Step S16: Schedule the pods of the batch task job to the same node or adjacent nodes to reduce communication delay. For specific batch task scheduling methods, see Figure 4 ;

[0044] Step S17: Ordinary jobs are scheduled to nodes with lower loads. The method for selecting nodes with lower loads is as follows: first, select nodes that can meet job requirements according to the resource table of all nodes, then calculate the loads of these nodes, and select nodes with lower loads for scheduling;

[0045] Step S18: Wait for the job task to be completed, and feedback information to the scheduler, which then updates the node resource table.

[0046] The following is a detailed description of the batch task scheduling method. Figure 4 As shown, the following steps are included:

[0047] Step S21, the batch task job is ready to be scheduled;

[0048] Step S22, determine whether the batch task job requires heterogeneous resources, if it requires heterogeneous resources, execute step S23, if not, execute step S24;

[0049] Step S23, sort all nodes in queues according to the heterogeneous resources required by the batch task job, put nodes including required heterogeneous resources in the front row, nodes including only common resources in the middle row, and nodes including non-required heterogeneous resources in the back row. For example, if a batch task job requires NPU heterogeneous resources, the nodes with NPU resources are in the front row, the middle row is the nodes including only common resources, and the back row is the nodes including other heterogeneous resources, such as GPU nodes. The nodes in the front row, middle row, and back row are then arranged in descending order according to the load;

[0050] Step S24 is similar to step S23, but ordinary nodes are placed in the front of the queue and nodes of heterogeneous resources are placed in the back of the queue. After the order is arranged, step S25 is executed;

[0051] Step S25, obtaining the idle resource status of each node from all node resource tables, and traversing the nodes in sequence according to the order of the nodes in the queue;

[0052] Step S26, determine whether the idle resources in the traversed nodes meet the requirements of the batch task job, if not, return to step S25 and continue to inspect the next node, if yes, execute step S27;

[0053] Step S27: Schedule the batch task job on the traversed nodes, and record the nodes and resource amounts allocated to each pod of the batch task job in the task consumption resource table. In this way, the batch task scheduling is completed, and the process returns to step S21 to continue scheduling the next task.

[0054] Executing MPI tasks in containers requires solving communication problems between containers, such as Figure 5 The execution process of the MPI task is shown in Figure 1, which includes the following steps:

[0055] Step S31: The user submits an MPI task. Since the MPI task is completed once, the job type is selected instead of deployment.

[0056] Step S32: The scheduler allocates a pod to the MPI task according to the submitted job information;

[0057] Step S33: Create a service for the pod of the MPI task so that the pod can obtain the IP addresses of other pods through the service. On the master pod, obtain the IP address of each worker pod through the service and save it in / etc / mpiworker.host;

[0058] Step S34: Generate an SSH key pair on the master's pod, and then pass the public key to other worker's pods, thus achieving SSH password-free login;

[0059] Step S35: Through the host file created in step S33, the master's pod can specify which workers to execute the MPI task together, and finally execute the mpirun or mpiexec command to complete the MPI task.

[0060] It should be understood that the various numerical numbers involved in the embodiments of the present invention are only used for the convenience of description and are not used to limit the scope of the embodiments of the present invention.

[0061] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A batch task scheduling method based on Kubernetes, characterized in that: The method comprises: Assign new tasks to the normal task queue or batch task queue based on the task label; Based on the optimization strategy, batch tasks are preferentially selected as tasks to be executed; If the task to be executed is a batch task, it is scheduled to be executed on a single node or multiple adjacent nodes that meet its resource requirements according to the resources required by the batch task; If the batch task is an MPI task, create a service for the pod of the MPI task to obtain the IP address of the MPI task pod; then generate an SSH key pair and pass the public key to each pod to implement SSH password-free login; If the task to be executed is a batch task and requires heterogeneous resources, all nodes in the cluster are queued and sorted, with nodes containing the required heterogeneous resources in the front row, nodes containing only common resources in the middle row, and nodes containing non-required heterogeneous resources in the back row. The nodes in the front row, middle row, and back row are arranged from high to low according to the load; all nodes in the queue are traversed from front to back, and if a node that meets the resource requirements of the task to be executed is found, the task to be executed is scheduled on the node; If the task to be executed is a batch task and does not require heterogeneous resources, all nodes in the cluster are queued and sorted, with nodes containing only common resources in the front row and nodes containing heterogeneous resources in the back row. The nodes in the front and back rows are arranged from high to low according to load; all nodes in the queue are traversed from front to back, and if a node that meets the resource requirements of the task to be executed is found, the task to be executed is scheduled on the node.

2. The method according to claim 1, characterized in that If there is no single node in the queue that can meet the resource requirements of the task to be executed, multiple adjacent nodes that can meet the resource requirements of the task to be executed are searched from the queue to jointly schedule the task to be executed.

3. The method according to claim 1, characterized in that If the idle resources in the cluster cannot meet the resource requirements of the tasks to be executed, the tasks to be executed are returned to the task queue based on the task tags.

4. The method according to claim 1, characterized in that: The preferred strategy is: determine whether the batch task queue is empty, if so, select ordinary tasks from the ordinary task queue as tasks to be executed; otherwise, select batch tasks from the batch task queue as tasks to be executed.

5. The method according to claim 1, characterized in that The preferred strategy is: within a unit time, N Batch tasks are selected as tasks to be executed. M Ordinary tasks are used as tasks to be executed; among them, .

6. The method according to claim 4 or 5, characterized in that: The tasks in the task queue follow the first-in-first-out rule.

Images