Mapreduce optimization method based on gpu sorting

Abstract

The present invention proposes a MapReduce optimization method based on GPU sorting, wherein MapReduce includes a Map stage, a Shuffle stage, and a Reduce stage, the Map stage includes a Spill process and a Merge process, and the Reduce stage includes a Merge process, wherein the Spill process in the Map stage The GPU-based quick sorting process is adopted in the process, and the GPU-based merge sorting process is adopted in the Merge process of the Map stage and the Merge process of the Reduce stage. By replacing the traditional CPU-based quick sort, merge sort, and heap sort algorithms with GPU-based quick sort and merge sort algorithms, the intermediate data processing speed is improved, thereby improving the performance of MapReduce.

Description

technical field [0001] The invention relates to MapReduce technology, in particular to a MapReduce optimization processing method. Background technique [0002] MapReduce is a distributed programming framework that is widely used in cloud computing and big data processing. The MapReduce process is as follows figure 1 As shown, it is divided into three stages: Map, Shuffle, and Reduce. The Map and Reduce stages execute the Map() and Reduce() programs written by the user respectively. The Shuffle stage is between the Map stage and the Reduce stage, and is used to generate data for the Map stage. The intermediate results are processed to prepare data for the Reduce phase. Specifically, the Map phase includes the following steps: [0003] Data read (Read): read data from the distributed file system; [0004] Map execution (Map): Execute the Map() function written by the user; [0005] Data collection (Collect): store the results generated by Map into the buffer; [0006] C...

AI Technical Summary

Problems solved by technology

The optimization method related to the user program requires the user to participate in the task allocation, and the user needs to be very familiar with the processing flow of the program and the programming specification of the GPU, and the optimization method between different user programs cannot be reused

However, the existing MapReduce optimization method based on GPU sorting is only for the optimization of the first sorting process, that is, to replace the CPU-based quick sorting algorithm with a GPU-based quick sorting algorithm or a GPU-based dual-tone sorting algorithm, while for the other three sorting The operation is not concerned, and the improvement of MapReduce performance is limited

Images