Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Mapreduce optimization method based on gpu sorting

An optimization method and quick sorting technology, applied in the field of MapReduce, can solve the problems of non-reusability of optimization methods and limited performance improvement of MapReduce, and achieve the effect of less difficulty in implementation

Inactive Publication Date: 2019-08-02
TIANZE INFORMATION IND
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The optimization method related to the user program requires the user to participate in the task allocation, and the user needs to be very familiar with the processing flow of the program and the programming specification of the GPU, and the optimization method between different user programs cannot be reused
However, the existing MapReduce optimization method based on GPU sorting is only for the optimization of the first sorting process, that is, to replace the CPU-based quick sorting algorithm with a GPU-based quick sorting algorithm or a GPU-based dual-tone sorting algorithm, while for the other three sorting The operation is not concerned, and the improvement of MapReduce performance is limited

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mapreduce optimization method based on gpu sorting
  • Mapreduce optimization method based on gpu sorting
  • Mapreduce optimization method based on gpu sorting

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] As described in the prior art, MapReduce includes: a Map phase, a Shuffle phase, and a Reduce phase, the Map phase includes a Spill process and a Merge process, and the Reduce phase includes a Merge process.

[0052] In the MapReduce optimization method based on GPU sorting of the present invention, adopt GPU quick sorting to replace CPU-based quick sorting in the Spill process of Map stage; Adopt GPU-based merge sorting to replace CPU-based merge sorting in the Merge process of Map stage; Adopt based on The GPU merge sort replaces the CPU-based merge sort in the Merge process of the Reduce phase.

[0053] The GPU-based quicksort process includes:

[0054] (1.1) Store the data into the global storage space of the GPU and divide it into m non-overlapping data blocks, each of which is processed by a thread block; figure 2 In a sequence division and b thread traversal are shown;

[0055] (1.2) m thread blocks traverse the corresponding data block in parallel, and n thre...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention proposes a MapReduce optimization method based on GPU sorting, wherein MapReduce includes a Map stage, a Shuffle stage, and a Reduce stage, the Map stage includes a Spill process and a Merge process, and the Reduce stage includes a Merge process, wherein the Spill process in the Map stage The GPU-based quick sorting process is adopted in the process, and the GPU-based merge sorting process is adopted in the Merge process of the Map stage and the Merge process of the Reduce stage. By replacing the traditional CPU-based quick sort, merge sort, and heap sort algorithms with GPU-based quick sort and merge sort algorithms, the intermediate data processing speed is improved, thereby improving the performance of MapReduce.

Description

technical field [0001] The invention relates to MapReduce technology, in particular to a MapReduce optimization processing method. Background technique [0002] MapReduce is a distributed programming framework that is widely used in cloud computing and big data processing. The MapReduce process is as follows figure 1 As shown, it is divided into three stages: Map, Shuffle, and Reduce. The Map and Reduce stages execute the Map() and Reduce() programs written by the user respectively. The Shuffle stage is between the Map stage and the Reduce stage, and is used to generate data for the Map stage. The intermediate results are processed to prepare data for the Reduce phase. Specifically, the Map phase includes the following steps: [0003] Data read (Read): read data from the distributed file system; [0004] Map execution (Map): Execute the Map() function written by the user; [0005] Data collection (Collect): store the results generated by Map into the buffer; [0006] C...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/38G06F9/50
CPCG06F9/3877G06F9/5044
Inventor 李鹏飞丁有伟孙杰
Owner TIANZE INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com