Hadoop task scheduling method and device

Abstract

The invention provides a Hadoop task scheduling method and device, and the method comprises the following steps: obtaining the information of a plurality of tasks which have been operated in each nodeoperation, and determining the operation type of each node according to the information of the plurality of tasks which have been operated in each node operation; predicting the load of each node according to the operation type of each node; determining the adaptability of the operation type of each node and each node according to the operation type of each node and the load of each node; and according to the operation type of each node and the adaptability of each node, allocating the tasks which are not operated in the operation to each node. According to the scheme, based on Hadoop task scheduling of load prediction, the resource utilization rate of the Hadoop cluster can be increased.

Description

technical field [0001] The invention relates to the technical field of Hadoop task scheduling, in particular to a Hadoop task scheduling method and device. Background technique [0002] Hadoop is an open source distributed storage and processing system for big data batch processing jobs, such as big data analysis, web page indexing and other jobs. Hadoop's default job scheduling algorithm is based on FIFO. Currently, Hadoop is configured with a variety of job schedulers, mainly including FIFO (default scheduler), Capacity Scheduler (computing capacity scheduler) and Fair Scheduler (fair scheduler). [0003] Clusters often run different types of jobs at the same time, and these different types of workloads have different resource requirements. For example, I / O-intensive workloads use more I / O resources, while CPU-intensive workloads use more I / O resources. More computing resources will be used. FIFO preferentially runs the tasks of the same job at the top of the queue, whic...

AI Technical Summary

Problems solved by technology

FIFO preferentially runs the tasks of the same job at the top of the queue, which can reduce the throughput of the entire system, but severely limits the processing power of the cluster, because although the tasks of the same job often have the same characteristics, the I / O and CPU resources are not Not fully used, and a task performing I / O is blocked because the scheduler prevents it from using the CPU until the I / O operation is complete

With the increase in the number of users and user programs in the Hadoop cluster, the FIFO scheduler cannot make good use of cluster resources, nor can it meet the service quality requirements of different applications, and in severe cases, it will also affect the normal operation of jobs.

[0004] Capacity Scheduler divides resources to each queue in proportion, and sets strict constraints to prevent resource monopoly, which solves the problem of multi-user scheduling, but the scheduling strategy lacks support for load balancing, and the data locality is not ideal

[0005] Fair Scheduler tries to allocate resources equally to all jobs. If a user submits a new job, some resources will be released to the new job. This method ensures that all jobs get the same amount of resources, but the locality of data is not ideal. , resulting in the need to obtain data from other nodes, increasing I / O resource usage and network bandwidth consumption

[0006] For the above problems, no effective solutions have been proposed so far

Images