Spark partition load balancing method

Abstract

The invention discloses a load balancing method. The load balancing method comprises the following steps of after a Map task is started, obtaining and counting operation information through a partition monitor to obtain operation counting information; after the operation statistical information is obtained, calculating an intermediate data volume generated by each partition after 100% of mapping task load is completed through the partition size predictor; according to the intermediate data volume of the partitions, determining whether inclined partitions exist in all the partitions or not through a data inclination detection model; and if so, sorting the data in the inclined partitions in a descending order through a resource scheduler, and dynamically adjusting the original partition fileto balance the load of the Spark partitions. According to the method, the partition load of the Spark can be more balanced, a problem of data inclination is relieved, and the completion time of the application program can be shortened.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a Spark partition load balancing method. Background technique [0002] With the advent of the era of big data, the rise of various network technologies, the rapid expansion of information data, traditional processing and storage systems have been difficult to cope with massive data, and for the current popular big data analysis platforms such as Hadoop and Spark, the data skew to its Performance has been severely affected. At present, most of the solutions to the problem of data skew are based on the research of the Hadoop platform, and there are relatively few studies on the problem of data skew on the Spark platform. In Spark, the stage before Shuffle is called the Map stage, and the stage after that is called the Reduce stage. However, the default Spark partitioning algorithm will cause data skew after performing the Shuffle operation when the data distribution is uneven. ...

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present invention provides a load balancing method and device based on linear regression partition prediction to solve the technical problem that the existing solution to data skew in Spark causes the application program to run for too long, so as to achieve both Make Spark's partition load more balanced, alleviate the problem of data skew, and shorten the application completion time

Images