Unlock instant, AI-driven research and patent intelligence for your innovation.

Data sorting method and system based on mapreduce model

A data sorting and model technology, applied in the field of data processing, can solve problems such as long waiting time and time-consuming

Inactive Publication Date: 2018-03-09
HUAWEI TECH CO LTD +1
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, with the development of computer Internet technology, the online text, image, audio and video data has increased by orders of magnitude, and in the processing of large-scale data sets, the sorting of large-scale data is a very time-consuming operation, so , when adopting the method of sorting based on the MapReduce model in the prior art, the user needs to wait for a long time to obtain the sorting result

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data sorting method and system based on mapreduce model
  • Data sorting method and system based on mapreduce model
  • Data sorting method and system based on mapreduce model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0098] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention

[0099] In order to achieve the purpose of quickly outputting the sorting results required by the user, the present invention provides a data sorting method based on the MapReduce model, such as figure 1 shown, including steps:

[0100] In the MAP phase:

[0101] S11. In the process of sorting the elements in the data blocks, each MAP module adds the element to the first-in-first-out data structure corresponding to the MAP module after each determination of the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A data sorting method and system based on a MapReduce model. The sorting method comprising the following steps: when each MAP module sorts elements in a data block in a MAP phase, each time a sorting result of one element is determined, adding the element to a first-in-first-out data structure; in a Reduce phase, a Reduce module acquiring all current elements in the first-in-first-out data structure; acquiring a current global extremum element; outputting the current global extremum element as a sorting result; removing the output current global extremum element from a Reduce-phase element set, and updating the Reduce-phase element set. With the method and the system, without the need of waiting until a sorting process in a MAP phase ends, an extremum element can be selected from current elements in a first-in-first-out cache queue in a Reduce phase, and a sorting result is output in a streamed manner. In this way, the sorting result can be output in advance, so as to reduce the waiting time of the user.

Description

technical field [0001] The invention belongs to the field of data processing, in particular to a data sorting method and system based on a MapReduce model. Background technique [0002] MapReduce is a programming model that can be used for parallel operations on large-scale data sets. The MapReduce model distributes large-scale operations on data sets to each node on the network; each node executes corresponding operations in parallel and returns work results. The MapReduce model can greatly facilitate programmers to run their programs on distributed systems without distributed parallel programming. [0003] In program applications, sorting operations are frequently used. Generally, firstly, in the MAP phase, local sorting is performed on each node in parallel to generate local sorting results, and then in the Reduce phase, full sorting is performed according to the local sorting results of all nodes. Generates sorted results when complete. [0004] However, with the deve...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/9024
Inventor 朱妤晴陈兴振王磊詹剑锋
Owner HUAWEI TECH CO LTD