A hadoop scheduling method and system based on bandwidth awareness

Abstract

The invention discloses a Hadoop scheduling method based on bandwidth perception, which includes: establishing a job time completion model for Hadoop task scheduling, building a mathematical model for the Hadoop scheduling system, and converting the Hadoop task scheduling problem into a job to be scheduled The problem of finding a task scheduling method that makes the job completion time the shortest; using the real-time network management and flow control functions provided by SDN, a network bandwidth allocation mechanism based on time slots is proposed, and the remaining bandwidth of each link The occupation period is divided into equal time slots, based on the job completion time model and the network slot bandwidth allocation mechanism; before assigning computing nodes to a task, the locality of the task and the real-time network bandwidth are considered comprehensively, for Each task is assigned a computing node that can provide the earliest completion time. The invention solves the problem that in the existing method, task scheduling cannot be performed from two aspects of the global perspective and the available bandwidth of the actual network at the same time.

Description

technical field [0001] The invention belongs to the field of information processing and data calculation, and more specifically relates to a Hadoop scheduling method and system based on bandwidth perception. Background technique [0002] With the advancement of science and technology, Internet technology has developed rapidly, which not only promotes social development, but also greatly enriches people's network life. With the arrival of WEB2.0, the Internet has undergone earth-shaking changes. A prominent feature of WEB2.0 is user generated content (User generated content). A large number of user generated content has resulted in explosive growth of data. Faced with the challenges of large-scale data processing technology, cloud computing has been proposed as a new model for computing and processing large-scale data. Thanks to the joint evolution of multiple technologies such as distributed and virtualization, cloud computing has emerged as a new type of big data processi...

AI Technical Summary

Problems solved by technology

[0006] For the above defects or improvement needs of the prior art, the present invention provides a Hadoop scheduling method and system based on bandwidth perception. low technical issues

Images