Hadoop task scheduling method and device

Abstract

The present invention provides a Hadoop task scheduling method and device, wherein the method includes the following steps: acquiring information of multiple tasks that have been run in each node job, and determining each The job type of the node; according to the job type of each node, the load of each node is predicted; according to the job type of each node and the load of each node, the job type of each node and the adaptability of each node are determined; according to the job type of each node Compatibility with each node, assign tasks that are not running in the job to each node. This solution is based on Hadoop task scheduling based on load prediction, which can improve the resource utilization of Hadoop clusters.

Description

technical field [0001] The invention relates to the technical field of Hadoop task scheduling, in particular to a Hadoop task scheduling method and device. Background technique [0002] Hadoop is an open source distributed storage and processing system used for big data batch jobs, such as big data analysis, web page indexing, etc. Hadoop's default job scheduling algorithm is implemented based on FIFO. Currently, Hadoop is configured with a variety of job schedulers, including FIFO (default scheduler), Capacity Scheduler (computing power scheduler), and Fair Scheduler (fair scheduler). [0003] Clusters often run different types of jobs at the same time, and these different types of workloads have different resource requirements. For example, I / O-intensive workloads will use more I / O resources, while CPU-intensive workloads will use more I / O resources. will use more computing resources. FIFO prioritizes the tasks of the same job at the top of the queue, which can reduce th...

AI Technical Summary

Problems solved by technology

FIFO preferentially runs the tasks of the same job at the top of the queue, which can reduce the throughput of the entire system, but severely limits the processing power of the cluster, because although the tasks of the same job often have the same characteristics, the I / O and CPU resources are not Not fully used, and a task performing I / O is blocked because the scheduler prevents it from using the CPU until the I / O operation is complete

With the increase in the number of users and user programs in the Hadoop cluster, the FIFO scheduler cannot make good use of cluster resources, nor can it meet the service quality requirements of different applications, and in severe cases, it will also affect the normal operation of jobs.

[0004] Capacity Scheduler divides resources to each queue in proportion, and sets strict constraints to prevent resource monopoly, which solves the problem of multi-user scheduling, but the scheduling strategy lacks support for load balancing, and the data locality is not ideal

[0005] Fair Scheduler tries to allocate resources equally to all jobs. If a user submits a new job, some resources will be released to the new job. This method ensures that all jobs get the same amount of resources, but the locality of data is not ideal. , resulting in the need to obtain data from other nodes, increasing I / O resource usage and network bandwidth consumption

[0006] For the above problems, no effective solutions have been proposed so far

Images