Method for scheduling cloud computing tasks based on network bandwidth estimation

Abstract

The invention discloses a method for scheduling cloud computing tasks based on network bandwidth estimation. The platform of the method is divided into three stages which are a data center, shelves and compute nodes, and is uniformly managed by a management node, and the compute nodes are used for executing the tasks and acquiring available bandwidth information; when executing tasks, the compute nodes read input data from a local disk or by a network, and when the tasks are finished, the compute nodes request for new tasks from the management node; the management node is used for managing the available bandwidth information and scheduling the tasks, when managing the available bandwidth information, the management node collects bandwidth data of the compute nodes, specifies two compute nodes in different shelves to execute available bandwidth estimation programs, and counts the available bandwidth information; and when the compute nodes request for the tasks, the management node determines a task scheduling strategy by combining network bandwidths in the shelves, the available bandwidth between the shelves and load of the data center.

Description

technical field [0001] The invention relates to the field of computer networks, in particular to task scheduling technology, in particular to a cloud computing task scheduling method based on network bandwidth perception. Background technique [0002] With the widespread application of massive data processing (such as Web log analysis, OLAP, OLTP), cloud computing platforms (such as MapReduce, Hadoop, Dryad, etc.) for massive data processing are developing rapidly. [0003] These cloud computing platforms are deployed in data centers with a Shared Nothing architecture. In terms of data storage, the data file is divided into several file blocks of equal size, stored in the local hard disk of the computing node, and managed by the distributed file system. To ensure high reliability, multiple copies of each file block are stored. In terms of computing, the data processing job (Job) is divided into several independent subtasks (Task), and each task reads input file blocks from...

AI Technical Summary

Problems solved by technology

[0004] However, network bandwidth is a scarce resource in data centers and seriously affects the performance of cloud computing platforms

First of all, in order to maintain high cost performance, computing nodes in most data centers are not interconnected with high-speed network equipment (such as 10 Gigabit Ethernet); secondly, some data centers adopt a hierarchical network topology (which can be divided into data centers, racks, computing There is a bandwidth bottleneck when computing nodes in different racks communicate; again, some computing tasks need to read data through the network, and network transmission overhead affects task completion time

Finally, operations such as reading and writing large-scale data, creating and migrating file block copies require a large amount of network bandwidth

Images