Data equalization processing method, device and system based on mapping and specification

Abstract

The invention discloses a data equalization processing method, device and system based on mapping and agreement. Including: obtaining the data submitted by the client, performing initial partitioning on the obtained data according to the preset number of mappers; performing mapping processing on the data in the initial partitioning respectively to obtain the intermediate result data; calling the partitioner function to perform the intermediate result data according to Carry out fine-grained partitioning according to the preset number of fine-grained partitions, the preset number of fine-grained partitions is greater than the number of specified partitions; output the data amount information of the intermediate results in each fine-grained partition to the working server, and receive the fine-grained data returned by the working server. The corresponding relationship between the partition and the protocol partition, merge the intermediate result data belonging to the same protocol partition in the fine-grained partition and output it to the corresponding protocol partition; perform protocol processing on the intermediate result data in the protocol partition, and obtain the corresponding data processing results. By applying the invention, the data load can be balanced and the efficiency of data processing can be improved.

Description

technical field [0001] The invention relates to distributed data computing technology, in particular to a data balance processing method, device and system based on MapReduce (MapReduce). Background technique [0002] MapReduce is an existing system architecture applied to large-scale data processing. As a programming system architecture, it is widely used in parallel computing of large-scale data sets (such as data sets larger than 1TB), such as large-scale distributed filtering, Large-scale distribution sorting, web connection graph inversion, web access log analysis, inverted index construction, document clustering, machine learning, and statistical-based machine translation, etc. In the MapReduce system architecture, the data processing process is divided into two stages: the first stage is the Mapping (Map) stage, where the data to be processed is initially partitioned, and each element in the initial partition is calculated and output to the protocol partition. Among ...

AI Technical Summary

Problems solved by technology

In practical applications, the time required for data processing (the time at which the task ends) is determined by the Reducer1 with a long processing time. Therefore, due to the unbalanced load of the Reducer, the time for the system to process data will be extended. For example, the Reducer1 in the system is still Processing data, but at the same time, Reducer2 is in an idle state, so that TaskTracker resources cannot be effectively used, reducing the efficiency of data processing

Images