Method and device for connecting tables

Abstract

The embodiment of the invention discloses a table connection method and device. The specific implementation mode of the method comprises the steps that a first table and a second table to be connected are obtained, and the data size of the first table is smaller than that of the second table; if the first table cannot be broadcasted, calculating the ratio of the line number of the first table to the line number of the second table; if the ratio is smaller than or equal to a preset threshold value, a real-time bloom filter is constructed according to the association keys of the first table, and the real-time bloom filter serves as a filtering operator to be inserted into the filtering condition of the second table; filtering data which do not meet the filtering condition from the second table to obtain a filtering table; and respectively shuffling and sequencing the first table and the filtering table, and then connecting the first table and the filtering table. According to the embodiment, the timeliness of the task of the connection table is improved, the computing resources are saved, and meanwhile, the cluster load is reduced.

Description

technical field [0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and device for joining tables. Background technique [0002] As a general-purpose computing engine for large-scale data processing, Spark has the characteristics of high throughput, low latency, general and easy expansion, and high fault tolerance. It has formed an ecosystem with rapid development and wide application. Its main modules include: SparkCore, Spark SQL , Spark Streaming, MLlib, and GraphX, among which Spark SQL is the most widely used in big data and is continuously applied to large-scale data processing. [0003] In production scenarios, many Spark SQL business scenarios require a small (relatively small) table to join a large table. The small table here means that BroadcastHashJoin (broadcast hash join) will not be triggered, and SortMergeJoin (sort merge join) will be used by default. The implementation logic of the sort...

AI Technical Summary

Problems solved by technology

For the scenario mentioned above, the join condition is based on the small table on the left, that is, when the two tables are joined, only the data on the large table side that satisfies the condition equal to the data on the small table side will be retained, and the data that does not meet the conditions on the large table side The data of the connection condition will be considered as invalid data. Since the invalid data participates in shuffling, sorting, and merge calculation, on the one hand, it will affect the timeliness of tasks. On the other hand, invalid shuffling data will increase the load of cluster disk IO and network IO. waste a lot of computing resources

Images