Minimum-cost maximum-flow based large-scale resource scheduling system and minimum-cost maximum-flow based large-scale resource scheduling method

Abstract

The invention relates to a minimum-cost maximum-flow based large-scale resource scheduling system and a minimum-cost maximum-flow based large-scale resource scheduling method. The system comprises a task state table, a cluster state table, a scheduling target table, a minimum-cost maximum-flow constructor, a minimum-cost maximum-flow solver and a task executor. The task state table is used for receiving and storing task states submitted by users, and the task states include task CPU (central processing unit) utilization rate, memory utilization rate, network I / O, magnetic disk I / O and priority. The cluster state table is used for storing cluster state information including cluster CPU utilization rate, memory utilization rate and network and magnetic disk I / O and updating cluster states when the cluster state change. The scheduling target table is used for storing scheduling targets configured by the users, and the scheduling targets include priority, placement constraint and fairness currently. The minimum-cost maximum-flow constructor is used for selecting the scheduling targets from the scheduling target table according to information of the task state table and the cluster state table to construct a minimum-cost maximum-flow graph. The minimum-cost maximum-flow solver is used for solving the minimum-cost maximum-flow graph constructed by the minimum-cost maximum-flow constructor according to an incremental algorithm. The task executor is responsible for specific execution of tasks. The minimum-cost maximum-flow based large-scale resource scheduling system and the minimum-cost maximum-flow based large-scale resource scheduling method meet the requirement on flexibility of practical business scenarios.

Description

technical field [0001] The invention relates to a large-scale resource scheduling system and method based on minimum cost and maximum flow, which belongs to the field of big data resource management, especially resource scheduling problems in large-scale environments. Background technique [0002] With the rapid development of technologies such as the Internet (Internet) and the Internet of Things (IoT), data (Data) has begun to transform from simple processing objects to basic services, and multiple jobs (Job) are submitted at the same time and decomposed into parallel execution tasks (Task). , running processing on physical servers with a scale of at least 10,000 has become a mainstream application mode, which is called the "concurrent job" problem. For example, Github needs to process more than 20 million jobs every year, and Facebook responds to nearly 10,000 job requests every day. [0003] Large-scale resource scheduling refers to the decision-making process of optima...

AI Technical Summary

Problems solved by technology

It can be seen that the existing work usually only applies to fixed targets, but related research shows that the same job set is scheduled with different targets, which may lead to extremely low resource utilization, such as the average resource utilization of Twitter is less than 20%. ; or cause a 5-fold drop in job performance

Images