Pull mode and Push mode combined resource management and job scheduling method and system and medium

Abstract

The invention discloses a Pull mode and Push mode combined resource management and job scheduling method and system and a medium. The method comprises the following steps: analyzing or identifying jobs to determine types and scheduling the jobs according to the types, and for high-performance computing jobs, distributing the jobs by adopting a Push mode: distributing computing nodes for the high-performance computing jobs, and pushing the high-performance computing jobs to the distributed computing nodes for execution; for the big data processing operation, adopting a Pull mode to perform distribution: waiting for an operation request of the computing node, and sending the big data processing operation to the computing node which actively sends the operation request to execute when the computing node actively sends the operation request. According to the method, the workload of the main node in the traditional system can be dispersed, the bottleneck effect caused by the workload is reduced, the resource utilization rate of the system is improved, and the method has the advantages of good universality, high resource utilization rate, high system throughput rate and good expandability.

Description

technical field [0001] The invention relates to the field of resource management and job scheduling of a large-scale computer system, in particular to a resource management and job scheduling method, system and medium combining a Pull mode and a Push mode. Background technique [0002] Resource management and job scheduling have always been a challenging problem in large-scale computing systems. At present, the scale of computing systems is getting larger and larger. For example, the Sunway TaihuLight supercomputer is composed of 40,000 computing nodes, and the number of processor cores reaches tens of millions. Shows heterogeneity, for example, the fastest Summit supercomputer currently contains 6 GPUs per node; since large-scale computing has become the basic means to promote technological progress in various industries, a large number of jobs with different characteristics are submitted to the computing system, lead to disorderly competition for resources. In the above ...

AI Technical Summary

Problems solved by technology

The so-called centralized means that the maintenance of job load information, the management of job queues, the maintenance of system resource information, and the job scheduling and allocation to idle resources are all undertaken by a specific central server or daemon process. There is a serious hidden danger in the architecture: the central scheduler can easily become the performance bottleneck and single point of failure of the entire system

Especially when the scale of the system expands, the type of workload becomes more and more complex, and the heterogeneity of system resources is further enhanced. If the centralized architecture is continued to be used and only the master node is used to complete all scheduling tasks, it will seriously affect the performance of the entire system. Operational Efficiency and Scalability

Images