High-performance ordering method for MapReduce calculation frame

Abstract

The invention relates to a high-performance ordering method for a MapReduce calculation frame. The method comprises the steps that buffer chains are respectively constructed in the Map stage according to partitions, the requirement that the partitions are ordered is removed, data of each partition are organized according to blocks, and the cost of data in copy and file IO aspects in internal storage is lowered; the ordering operation is not executed in the Map stage, a large buffer pool is adopted as the basic unit of ordering of one time in the Reduce stage, and the total merging approach quantity in the ordering merging state is made to be a value capable of being adjusted and optimized by a user. Through a mixed internal storage ordering algorithm, the two ordering stages in the MapReduce calculation frame are optimized, the influence of ordering on the performance of the calculation frame is basically eliminated, and then the resource validity of the calculation frame is improved, and the overall resource consumption of a cluster is reduced.

Description

technical field [0001] The invention belongs to the field of information technology, and relates to an optimization method for a distributed computing framework, in particular to a method for improving sorting performance in a MapReduce computing framework. Background technique [0002] MapReduce is a standard framework in distributed computing, but in terms of resource consumption, the existing MapReduce framework is not efficient, resulting in the waste of a lot of cluster resources. [0003] The key / value pairs of intermediate data need to be sorted in the existing MapReduce framework, and sorting constitutes the main resource consumption in the existing computing framework. We take Hadoop, an open source implementation of MapReduce, as an example to illustrate the above problems. [0004] Such as figure 1 Shown is a schematic diagram of the traditional MapReduce data flow. Among them, the Hadoop file system (referred to as HDFS, Hadoop File System) is responsible for ...

AI Technical Summary

Problems solved by technology

For jobs with a large number of maps, the cost of this part will be higher than the cost of the first part and consume most of the resources

[0008] Therefore, the two stages of sorting in the existing MapReduce computing framework are inefficient due to the selection of algorithms or improper processing procedures, and the sorting operation is essential for every job, resulting in a large waste of resources

Images