Container-based batch processing job scheduling method and device and computer equipment

Abstract

The invention relates to a container-based batch processing job scheduling method and device, computer equipment and a storage medium. The method comprises the steps that a workload packaging module reads a workload description configured by a user to obtain a corresponding operation unit container mirror image, and packages the workload description and the operation unit container mirror image toobtain a workload; a workload queue management module adjusts the sequence of the workload queues according to a scheduling priority strategy through a multi-queue queuing system comprising a preemption queue, a resource sensitive queue and a common queue; and a workload scheduling module takes out load scheduling execution and the workload operation monitoring module carries out operation monitoring. According to the invention, the efficiency of containerized batch processing job scheduling can be improved, and the balance between the resource utilization rate of the elastic cloud operationenvironment and the workload execution efficiency is realized.

Description

technical field [0001] The present application relates to the technical field of big data processing, in particular to a containerized batch job scheduling method, device, computer equipment and storage medium. Background technique [0002] In the field of deep learning and big data processing technologies, batch processing jobs are the most typical type of workload, usually occupying most of the working hours of deep learning and big data processing platforms. At present, the most widely used batch job scheduling systems include Slurm for high-performance computing, YARN and Mesos for big data analysis and processing applications. Among them, Slurm lacks a fault-tolerant mechanism, which leads to a lack of flexibility in job scheduling and a long period of job restart; YARN and Mesos are more tolerant to errors than Slurm, but they are also facing problems such as weak job encapsulation and job dependency conflicts. In addition, YARN and Mesos There is no effective support...

AI Technical Summary

Problems solved by technology

Among them, Slurm lacks a fault-tolerant mechanism, which leads to a lack of flexibility in job scheduling and a long period of job restart; YARN and Mesos are more tolerant to errors than Slurm, but they are also facing problems such as weak job encapsulation and job dependency conflicts. In addition, YARN and Mesos There is no effective support for the important acceleration devices in the deep learning field such as GPU; in order to solve the problems in the above job scheduling system, the new generation of job management system represented by Kubernetes and container technology has attracted people's attention, but Kubernetes and container The technology is more for scheduling enterprise-level service-type jobs, which is characterized by long-running (Long-Running), that is, jobs do not stop after startup, and provide external services in a request-response manner

This has led to the design principle of Kubernetes to ensure that as many jobs as possible get resources, and this principle is exactly in contradiction with the goal of batch processing to allow a single job to obtain as many resources as possible, using Kubernetes and container technology for batch jobs There are still great difficulties in scheduling

[0003] Therefore, the existing containerized batch job technology has the problems of low resource utilization and low workload execution efficiency

Images