Iterative data equilibrium optimization method for Spark parallel computing framework

Abstract

The present invention provides an iterative data equilibrium partitioning method for a Spark parallel computing framework. The method comprises: firstly, dividing a coarse-grained Block of big data into a fine-grained FG-Block, and creating a micro-partition and a micro-partition index according to the FG-Block; secondly, creating an equal quantity of Buckets according to a quantity of Reducers; thirdly, determining a timing and a quantity of an iterative data partition and a rule of iterative partitioning; fourthly, recording local and global data allocation conditions of each Bucket; fifthly, allocating a selected micro-partition to each Bucket according to a data equilibrium partitioning algorithm and an allocation condition; and finally, transmitting allocated data in the Bucket to a Reducer side. The present invention provides a new data equilibrium partitioning method for a Spark framework, which reduces data skew during big data processing and improves overall performance of big data of the Spark parallel computing framework.

Description

technical field [0001] The invention relates to the fields of big data processing and high-performance computing, and in particular proposes an iterative data balance optimization method oriented to the Spark parallel computing framework. Background technique [0002] MapReduce is a parallel computing model for big data processing proposed by Google in 2004. It improves the performance of data processing by simultaneously running multiple tasks on a large number of cheap cluster nodes to process massive data in parallel. It has been developed rapidly and widely used in recent years. Spark is a parallel computing framework based on MapReduce. It was developed by the AMPLab of the University of California, Berkeley in 2009. It has the advantages of MapReduce and saves the intermediate results of task calculations in memory, reducing disk read and write costs and improving performance. The performance of big data processing has become the mainstream framework for building big ...

AI Technical Summary

Problems solved by technology

However, it is difficult to grasp the timing of partitioning in this method, which is not universal

(3) Sampling data partitioning method: This method was proposed by Ramakrishnan et al. (Proceedings of the 3rd ACM Symposium on Cloud Computing, 2012), which combines sampling and data splitting, and adds an additional process in the execution of data processing to be responsible for Analyze the data distribution. When the data is processed to a certain proportion, the data is split and merged according to the analysis results of the sampling process, that is, the partition with a large amount of data is split and merged with the partition with a small amount of data; however, this method requires Additional overhead to collect data distribution will increase data access and data transmission overhead. Moreover, there is some uncertainty in data sampling. If sampling is too small, the accuracy will be insufficient, and sampling too much will add more additional overhead. (4) Delayed data partition method: This method is proposed by Kwon et al. (Proceedings of the 1st ACM Symposium on CloudComputing, 2010), which evaluates the size of the data partition by defining a cost model, and then evaluates the size of the generated data according to the cost model, and Start the data partition when the task runs to a certain point

However, this method requires additional data transmission costs, which delays the completion of the job to a certain extent;

Images