A job scheduling method for map/reduce massive data processing platform

Abstract

A job scheduling method for a Map / Reduce massive data processing platform belongs to the field of massive data processing, and particularly relates to job scheduling and resource management in the Map / Reduce massive data processing platform. The present invention proposes a Map / Reduce job scheduling method based on Reduce task resource preemption, which realizes that when the Reduce task is idle and waiting, the computing resources occupied by it are allocated to the Map task for use. The invention mainly includes: preempting resources, a job scheduling module, an application management module, a task execution module, a node management module, a task suspension and recovery method, and a job scheduling method oriented to preempting resources. The present invention can be applied to the job scheduling and resource management of the data center. Through the preemptive scheduling of the Reduce task in the Map / Reduce job, the reasonable allocation of computing resources in the data center can be realized, the utilization rate of computing resources can be improved, and the running time of jobs can be shortened.

Description

Technical field [0001] The invention belongs to the field of massive data processing, and particularly relates to job scheduling and resource management in a Map / Reduce type massive data processing platform. Background technique [0002] The Map / Reduce massive data processing platform is the latest technological advancement in the field of massive data processing. It mainly serves big data applications with a one-time write, multiple read data access mode and an Embarrassing Parallel computing mode. The Map / Reduce platform provides the Map / Reduce parallel computing model and its corresponding runtime environment. The Map / Reduce parallel computing model abstracts the application data processing flow into the Map phase and the Reduce phase. The Map phase and the Reduce phase can be respectively mapped to multiple Map tasks and Reduce tasks for parallel execution. Among them, the Map phase mainly performs data transposition and deformation, and the Reduce phase mainly performs dat...

AI Technical Summary

Problems solved by technology

In the existing scheduling method, when the Reduce task is idle and waiting, the computing resources allocated to it are not released, which greatly reduces the resource utilization of the Map / Reduce platform

Images