Map/Reduce type mass data processing platform-orientated job scheduling method

Abstract

The invention discloses a Map / Reduce type mass data processing platform-orientated job scheduling method belonging to the field of mass data processing, and particularly relates to job scheduling and resource management in a Map / Reduce type mass data processing platform. The invention provides a Map / reduce job scheduling method based on Reduce task resource preemption, which realizes that when a Reduce task is in an idle waiting state, computing resources occupied by the Reduce task are distributed to a Map task for being used. The Map / Reduce type mass data processing platform-orientated job scheduling method mainly comprises: preemptible resources, a job scheduling module, an application management module, a task executing module, a node management module, a task suspending and recovering method, and a preemptible-resource-orientated job scheduling method. The Map / Reduce type mass data processing platform-orientated job scheduling method can be applied to job scheduling and resource management of a data center, and realizes reasonable distribution of computing resources of the data center by preemption scheduling of a Reduce task in a Map / Reduce job, thereby improving the computing resource utilization rate and shortening the job running time.

Description

technical field [0001] The invention belongs to the field of massive data processing, and in particular relates to job scheduling and resource management in a Map / Reduce type massive data processing platform. Background technique [0002] The Map / Reduce massive data processing platform is the latest technological development in the field of massive data processing, mainly serving big data applications with a write-once, multiple-read data access mode and an Embarrassing Parallel computing mode. The Map / Reduce platform provides the Map / Reduce parallel computing model and its corresponding runtime environment. The Map / Reduce parallel computing model abstracts the data processing flow of the application into the Map stage and the Reduce stage. The Map stage and the Reduce stage can be respectively mapped to multiple Map tasks and Reduce tasks to be executed in parallel. Among them, the Map stage mainly performs data transposition and deformation, and the Reduce stage mainly p...

AI Technical Summary

Problems solved by technology

In the existing scheduling method, when the Reduce task is idle and waiting, the computing resources allocated to it are not released, which greatly reduces the resource utilization of the Map / Reduce platform

Images