Resource scheduling method, scheduler and system based on OpenFlow

Abstract

The invention discloses a resource scheduling method based on OpenFlow, a scheduler and a system thereof. The scheduler comprises the following steps: when a Reduce task to be scheduled arrives, current topology information of the cluster is obtained Determining an execution node for executing a Reduce task to be scheduled and a first transmission path for transmitting an intermediate result corresponding to the Reduce task to be scheduled to the execution node from an idle Reduce node according to the current topology information of the cluster; Scheduling a Reduce task to be scheduled to anexecution node, reserving bandwidth for the Reduce task to be scheduled according to the first transmission path and updating cluster topology information; The intermediate result corresponding to theReduce task to be scheduled is transmitted to the execution node according to the first transmission path, so that the Reduce task to be scheduled is executed by the execution node. The invention canminimize the sum of the use costs of the computing resources and the network resources in the Reduce stage, thereby improving the execution efficiency of the task.

Description

technical field [0001] The invention belongs to the field of cloud computing resource scheduling, and more particularly relates to an OpenFlow-based joint scheduling method of cloud computing resources. Background technique [0002] Hadoop is currently one of the most popular open source cloud computing platforms. Its MapReduce programming model uses the parallel operation of clusters for high-speed computing and storage to analyze and process massive data. In the resource scheduling problem of MapReduce, the Map tasks in the Map stage are of the same size and large in number, which can achieve better load balance; while in the Reduce stage, due to the large difference in the granularity of the Reduce tasks and the heterogeneous capabilities of the Reduce nodes, when the amount of intermediate data is large, When it is large, it is easy to have unbalanced reduce node load, transmission path congestion, and long task queuing time. [0003] Since the execution of tasks in Map...

AI Technical Summary

Problems solved by technology

In the resource scheduling problem of MapReduce, the Map tasks in the Map stage are of the same size and large in number, which can achieve better load balance; while in the Reduce stage, due to the large difference in the granularity of the Reduce tasks and the heterogeneous capabilities of the Reduce nodes, when the amount of intermediate data is large, When it is large, it is easy to have unbalanced reduce node load, transmission path congestion, and long task queuing time, etc.

[0003] Since the execution of tasks in MapReduce requires both computing resources (CPU, memory, etc.) and network resources (bandwidth, etc.), the lack of any aspect will affect the execution efficiency of tasks, and the existing MapReduce design for Hadoop and its core Various resource scheduling methods proposed for the resource scheduling problem, such as scheduling algorithms focusing on data locality, scheduling algorithms focusing on load balancing, adaptive perception scheduling algorithms, fair scheduling algorithms, etc., do not consider network resource allocation and computing resources at the same time allocation, so the problem of joint optimal allocation of computing resources and network resources is worthy of further study

[0004] At present, some studies have proposed OpenFlow-based scheduling methods to optimize data transmission in Hadoop / MapReduce. These methods have used the bandwidth control capabilities of OpenFlow to optimize the scheduling of network bandwidth resources, but have not further optimized the scheduling objectives. Comprehensive measurement of resources and network bandwidth resources

Images