Data balance method based on genetic algorithm in MapReduce calculation module

Abstract

Provided is a data balance method based on genetic algorithm in a MapReduce calculation module. The data balance method based on the genetic algorithm in the MapReduce calculation module includes: obtaining global Map output information, utilizing the genetic algorithm to conduct combination optimization, collecting and coding metadata, conducting multiple random partition on population, forming a genome through each partition, calculating fitness function values of all subsets in each gene, applying a selection operator to a genome on the basis of evaluating fitness of each gene, utilizing a roulette algorithm to choose a plurality of high quality genes in the genome at random, conducting cross operation on the chosen genes, conducting mutation operation, choosing retained genes according to an elitism strategy after multiple evolutions, decoding the genes to obtain a optical combination of the metadata and guaranteeing that each data quantity which is processed by the reducer is approximate equal. The data balance method based on genetic algorithm in the MapReduce calculation module solves the problem of unbalance input data in the reduce phrase, saves calculation resource and reduces calculation cost.

Description

technical field [0001] The invention belongs to the technical field of computer MapReduce calculation models, in particular to a genetic algorithm-based data balance method in the MapReduce calculation model. Background technique [0002] Hadoop is a highly reliable and highly scalable storage and distributed parallel computing platform developed by the Apache open source organization. It was first developed as the basic platform of the open source search engine project Nutch, and then became independent from the Nutch project. Become one of the typical open source cloud computing platforms. The Hadoop core implements a block-based distributed file system (Hadoop Distributed File System, HDFS) and a MapReduce computing model for distributed computing. [0003] The MapReduce computing model is divided into two major task processing stages, Map and Reduce. During the MapReduce process, the Map stage converts the input data into the data form of <Key,Value> key-value pa...

AI Technical Summary

Problems solved by technology

This method is subject to factors such as hash calculation conflicts and the limited number of reduce. It is likely that a large number of keys will converge on the same partition, resulting in an unbalanced amount of data on each reduce task.

[0009] 2) Reduce calculation skew caused by the characteristics of the input data itself

In general, the input data skew in the Reduce phase will increase the execution time of some reduce tasks relative to other reduce tasks, prolong the running time of the entire Reduce phase, and ultimately affect the completion time of the entire MapReduce job

Images