Distributed job scheduling method suitable for heterogeneous computational capability cluster

Abstract

The invention discloses a distributed job scheduling method suitable for a heterogeneous computational capability cluster. The method comprises the following five steps: (1) automatic discovery of node resources, wherein each node in the cluster transmits resource information to a network in a form of a multicast message; and a scheduler receives the multicast message and automatically discovers cluster nodes; (2) a job scheduling mechanism based on first input first output (FIFO) and resource utilization maximization; (3) a data set-based dynamic rescheduling mechanism, wherein a job manager dynamically adjusts the data sets of various tasks; (4) a delay scheduling mechanism, wherein partial tasks of the job are deployed on all nodes; and the remaining tasks are submitted to a delay queue to be scheduled; and (5) use of a redundant mutual preparation mechanism, wherein the scheduler deploys a backup task for each task to ensure that the backup task still can provide a computation result after the task is in a fault or off-line. According to the distributed job scheduling method, the distributed processing capacity of the cluster is improved; the resource utilization rate of the system is improved; and the job processing reliability is ensured.

Description

technical field [0001] The invention relates to a distributed job scheduling method suitable for heterogeneous clusters of computing capabilities, which belongs to the field of automation technology. Background technique [0002] When the current power grid dispatching system deploys jobs, it mainly distributes the tasks of the job to statically designated nodes to run. This method is no longer applicable in clusters with heterogeneous computing capabilities. On the one hand, each node in a heterogeneous cluster has different processing capabilities, and the statically designated job deployment nodes cannot fully utilize the performance advantages of nodes with high processing capabilities; On the other hand, the tasks of the job cannot be fully deployed to nodes with high processing capacity, and the processing efficiency of the job cannot be improved. Therefore, it is necessary to introduce a new scheduling algorithm to make full use of cluster resources, improve job proc...

AI Technical Summary

Problems solved by technology

This method is no longer applicable in clusters with heterogeneous computing capabilities. On the one hand, each node in a heterogeneous cluster has different processing capabilities, and the statically designated job deployment nodes cannot fully utilize the performance advantages of nodes with high processing capabilities; On the other hand, the tasks of the job cannot be fully deployed to the nodes with high processing capacity, and the processing efficiency of the job cannot be improved.

Images