Resource management and job scheduling method and system combining pull mode and push mode

Abstract

The invention discloses a resource management and job scheduling method, system and medium combining a Pull mode and a Push mode. The invention includes analyzing or identifying a type of job and scheduling by type. Distribute in pull mode: allocate computing nodes for the high-performance computing job, push the high-performance computing job to the assigned computing node for execution; for big data processing jobs, use Pull mode for distribution: wait for the job request from the computing node, and When the computing node actively sends a job request, the big data processing job is sent to the computing node that actively sends the job request for execution. The present invention can disperse the workload of the master node in the traditional system, reduce the bottleneck effect caused by it, and improve the resource utilization rate of the system. The present invention has good versatility, high resource utilization rate, high system throughput rate and good scalability The advantages.

Description

technical field [0001] The invention relates to the field of resource management and job scheduling of a large-scale computer system, in particular to a resource management and job scheduling method, system and medium combining a Pull mode and a Push mode. Background technique [0002] Resource management and job scheduling have always been a challenging problem in large-scale computing systems. At present, the scale of computing systems is getting larger and larger. For example, the Sunway TaihuLight supercomputer is composed of 40,000 computing nodes, and the number of processor cores reaches tens of millions. Shows heterogeneity, for example, the fastest Summit supercomputer currently contains 6 GPUs per node; since large-scale computing has become the basic means to promote technological progress in various industries, a large number of jobs with different characteristics are submitted to the computing system, lead to disorderly competition for resources. In the above ...

AI Technical Summary

Problems solved by technology

The so-called centralized means that the maintenance of job load information, the management of job queues, the maintenance of system resource information, and the job scheduling and allocation to idle resources are all undertaken by a specific central server or daemon process. There is a serious hidden danger in the architecture: the central scheduler can easily become the performance bottleneck and single point of failure of the entire system

Especially when the scale of the system expands, the type of workload becomes more and more complex, and the heterogeneity of system resources is further enhanced. If the centralized architecture is continued to be used and only the master node is used to complete all scheduling tasks, it will seriously affect the performance of the entire system. Operational Efficiency and Scalability

Images