Task resource scheduling processing method and system

Abstract

The invention discloses a task resource scheduling processing method and a task resource scheduling processing system. The task resource scheduling processing method comprises the following steps that: at least two virtual machines are installed on a task executing hardware node, and a virtual TT (Task Tracker) is arranged on each virtual machine; hardware resources on the task executing hardware node are divided into a resource pool and a resource queue, wherein the resource queue comprises at least two resource blocks, each resource block is correspondingly allocated to one virtual TT arranged on the hardware node to be used, and the resource pool is correspondingly allocated to at least one virtual TT arranged on the hardware node to be used; after a JT (Job Tracker) receives a task request, the resource quantity MTN required by a task is determined; whether the MTN is greater than the resource quantity MRN of the largest resource block in the resource queue or not is judged; if so, one virtual TT corresponding to the resource pool is called for processing the task; and if not, one virtual TT corresponding to the resource block in the resource queue is called for processing the task. The task resource scheduling processing method and the task resource scheduling processing system have the advantages that the resource utilization rate of the system can be improved, and the task processing efficiency is improved.

Description

technical field [0001] The present application relates to the technical field of computer data processing, and in particular to a task resource scheduling processing method and system. Background technique [0002] Hadoop is a computing system that can perform distributed processing on large amounts of data. There are more and more massive data services, and the use of Hadoop is becoming more and more extensive. With the increasing scale of a single cluster (the first-generation Hadoop cluster can support about 4,000 machines), how to improve the utilization of cluster resources has gradually become a topic of concern. The key to improving cluster resource utilization is cluster scheduling. [0003] Hdoop MapReduce task scheduling system, referred to as Hadoop scheduler, is one of the core components of Hadoop, which is used to allocate idle resources in the system to tasks according to certain policies. In Hadoop, the scheduler is a pluggable module, and users can design ...

AI Technical Summary

Problems solved by technology

[0011] Each processing node in the scheduling system in the prior art must be physical machine hardware, and both the node where the task scheduler and the task executor are located must be on the physical hardware node, and each hardware node can only be assigned one When a task scheduler or task executor schedules a task, even if the task occupies a small amount of resources, it still needs to occupy a hardware node to process the task alone, and other tasks need to queue up in the task queue, which leads to A large number of resources are idle, the resource utilization efficiency of the entire system is very low, and the task processing efficiency is low

Images