A distributed-aware binary equivalent connection tilt optimization method and system

Abstract

The invention discloses a binary equivalent connection tilt optimization method based on distributed sensing. The method comprises the steps of carrying out sampling on two data sets R and S preparedto be subjected to binary connection operation to obtain two sample sets, calculating weights and sizes of tilt keys in the sample sets respectively, comparing the weights and the sizes of the tilt keys in the sample sets, and selecting the data set corresponding to relatively large one of comparison results as a main table and the data set corresponding to the other as an auxiliary table; and obtaining a cost estimation value of each cluster in the selected main table and auxiliary table, determining a big cluster set and a small cluster set according to the cost estimation values, generatingRDD partition schemes for the big cluster set and the small cluster set respectively, partitioning the main table and the auxiliary table by using the RDD partition schemes of the big cluster set andthe small cluster set, and finishing connection operation in partitions at a Reduce end according to a result of partitioning the main table and the auxiliary table. According to the method, the technical problem that load tilt easily occurs at the Reduce end in an existing binary connection method can be solved.

Description

technical field [0001] The invention belongs to the technical field of parallel computing, and more specifically relates to a method and system for optimizing a binary equivalence connection tilt based on distribution perception. Background technique [0002] At present, the connection operation for the Spark platform in the distributed system is to first partition the collection of tables, and then implement the connection operation in the partition. The connection operation is based on the basic data structure of the platform - Resilient Distributed Datasets (RDD for short) based operations, each data table can be converted into a key-value pair RDD set of <key, value>. The existing partition methods of the Spark platform include the hash partition algorithm and the simple value domain partition algorithm. The binary join methods corresponding to these two partition algorithms are the hash join algorithm and the range join algorithm. [0003] However, the binary co...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the above deficiencies or improvement needs of the prior art, the present invention provides a method and system for tilt optimization of binary equivalent connection based on distribution perception, the purpose of which is to solve the problems existing in the binary connection method of the existing distributed system The reduce side is prone to load inclination, and the technical problem of high overhead when obtaining the total amount of the original data set

Images