GPU (graphics processing unit) sorting-based MapReduce optimizing method

Abstract

The invention discloses a GPU sorting-based MapReduce optimizing method. According to the GPU sorting-based MapReduce optimizing method, MapReduce is composed of a Map stage, a Shuffle stage and a Reduce stage, wherein the Map stage comprises a Spill process and a Merge process; the Reduce stage comprises a Merge process; during the Spill process of the Map stage, a GPU-based rapid sorting process is performed; during the Merge process of the Map stage and the Merge process of the Reduce stage, a GPU-based merging sorting process is performed. By substituting traditional CPU-based (central processing unit-based) rapid sorting, merging sorting and heap sorting algorithms through the GPU-based rapid sorting and merging sorting algorithms, the GPU sorting-based MapReduce optimizing method can improve the intermediate data processing speed and further improve the performance of the MapReduce.

Description

technical field [0001] The invention relates to MapReduce technology, in particular to a MapReduce optimization processing method. Background technique [0002] MapReduce is a distributed programming framework that is widely used in cloud computing and big data processing. The MapReduce process is as follows figure 1 As shown, it is divided into three stages: Map, Shuffle, and Reduce. The Map and Reduce stages execute the Map() and Reduce() programs written by the user respectively. The Shuffle stage is between the Map stage and the Reduce stage, and is used to generate data for the Map stage. The intermediate results are processed to prepare data for the Reduce phase. Specifically, the Map phase includes the following steps: [0003] Data read (Read): read data from the distributed file system; [0004] Map execution (Map): Execute the Map() function written by the user; [0005] Data collection (Collect): store the results generated by Map into the buffer; [0006] C...

AI Technical Summary

Problems solved by technology

The optimization method related to the user program requires the user to participate in the task allocation, and the user needs to be very familiar with the processing flow of the program and the programming specification of the GPU, and the optimization method between different user programs cannot be reused

However, the existing MapReduce optimization method based on GPU sorting is only for the optimization of the first sorting process, that is, to replace the CPU-based quick sorting algorithm with a GPU-based quick sorting algorithm or a GPU-based dual-tone sorting algorithm, while for the other three sorting The operation is not concerned, and the improvement of MapReduce performance is limited

Images