Cluster resource scheduling method based on user job type and node performance bias

Abstract

The invention relates to the technical field of cluster scheduling, in particular to a cluster resource scheduling method based on the user job type and node performance bias. The method comprises the following steps: 1) a Master node collects static index values of nodes in a cluster and calculates static performance of the nodes; 2) the Master node calculates the type of a user job to be operated currently in the job queue; 3) the Master nodeacquires dynamic state data of each node in the cluster , and calculates the dynamic performance of each node; 4) according to the dynamic and static performance of each node, the Master node calculates the performance bias of each node; 5) the Master node allocates appropriate node resources to the user job according to the type of the user job and the performance bias of the node, and after the job is completed, the Master node returns the execution result of the job; 6) if all the user jobs are executed, the system operation is ended, and otherwise, the step 2) is returned to. The method has the beneficial effects that the performance of the cluster is effectively improved, and the execution time of user operation is shortened.

Description

Technical field[0001]The present invention relates to the field of cluster scheduling techniques, and more particularly to a cluster resource scheduling method based on user job type and node performance biased.Background technique[0002]With the arrival of the big data era, the data generated by the user has grown in an exponential level. Single nodes and traditional computing models have no ability to meet the performance and efficiency requirements of big data processing. Apache Spark has become the most popular big data processing platform due to its outstanding performance and extensive application support. In the SPARK STANDALONE mode, both spreadout and non-Spreadout scheduling algorithms are used by default for multi-job resource scheduling. These two resource scheduling algorithms use very simple logic, which is scheduling by comparing the resource requirements of the job and the available resources of the node. Such a resource scheduling algorithm is not considered when a c...

AI Technical Summary

Problems solved by technology

When updating the node label, only its CPU and IO usage are considered, and factors such as the number of node CPU cores, memory size, disk capacity, and real-time read and write speed of the disk are not considered.

Images