GPU (graphics processing unit) sorting-based MapReduce optimizing method

An optimization method and quick sorting technology, applied in the field of MapReduce, can solve problems such as unreusable optimization methods and limited performance improvement of MapReduce, and achieve the effect of low difficulty

Inactive Publication Date: 2017-06-06
TIANZE INFORMATION IND
View PDF2 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The optimization method related to the user program requires the user to participate in the task allocation, and the user needs to be very familiar with the processing flow of the program and the programming specification of the GPU, and the optimization method between different user programs cannot be reused
However, the existing MapReduce optimizatio

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • GPU (graphics processing unit) sorting-based MapReduce optimizing method
  • GPU (graphics processing unit) sorting-based MapReduce optimizing method
  • GPU (graphics processing unit) sorting-based MapReduce optimizing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] As described in the prior art, MapReduce includes: a Map phase, a Shuffle phase, and a Reduce phase, the Map phase includes a Spill process and a Merge process, and the Reduce phase includes a Merge process.

[0050] In the MapReduce optimization method based on GPU sorting of the present invention, adopt GPU quick sorting to replace CPU-based quick sorting in the Spill process of Map stage; Adopt GPU-based merge sorting to replace CPU-based merge sorting in the Merge process of Map stage; Adopt based on The GPU merge sort replaces the CPU-based merge sort in the Merge process of the Reduce phase.

[0051] The GPU-based quicksort process includes:

[0052] (1.1) Store the data into the global storage space of the GPU and divide it into m non-overlapping data blocks, each of which is processed by a thread block; figure 2 In a sequence division and b thread traversal are shown;

[0053] (1.2) m thread blocks traverse the corresponding data block in parallel, and n thre...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a GPU sorting-based MapReduce optimizing method. According to the GPU sorting-based MapReduce optimizing method, MapReduce is composed of a Map stage, a Shuffle stage and a Reduce stage, wherein the Map stage comprises a Spill process and a Merge process; the Reduce stage comprises a Merge process; during the Spill process of the Map stage, a GPU-based rapid sorting process is performed; during the Merge process of the Map stage and the Merge process of the Reduce stage, a GPU-based merging sorting process is performed. By substituting traditional CPU-based (central processing unit-based) rapid sorting, merging sorting and heap sorting algorithms through the GPU-based rapid sorting and merging sorting algorithms, the GPU sorting-based MapReduce optimizing method can improve the intermediate data processing speed and further improve the performance of the MapReduce.

Description

technical field [0001] The invention relates to MapReduce technology, in particular to a MapReduce optimization processing method. Background technique [0002] MapReduce is a distributed programming framework that is widely used in cloud computing and big data processing. The MapReduce process is as follows figure 1 As shown, it is divided into three stages: Map, Shuffle, and Reduce. The Map and Reduce stages execute the Map() and Reduce() programs written by the user respectively. The Shuffle stage is between the Map stage and the Reduce stage, and is used to generate data for the Map stage. The intermediate results are processed to prepare data for the Reduce phase. Specifically, the Map phase includes the following steps: [0003] Data read (Read): read data from the distributed file system; [0004] Map execution (Map): Execute the Map() function written by the user; [0005] Data collection (Collect): store the results generated by Map into the buffer; [0006] C...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/38G06F9/50
CPCG06F9/3877G06F9/5044
Inventor 李鹏飞丁有伟孙杰
Owner TIANZE INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap