Multi-table correlation method and system, main service node and computing node

Abstract

An embodiment of the invention discloses a multi-table correlation method and system, a main service node and at least one computing node and relates to the information technology field. The correlation between a plurality of tables which are large in data quantity can be implemented and meanwhile the multi-table correlation efficiency is improved. The multi-table correlation method comprises obtaining the data quantity of a first table item and a correlation data column in the first data item through the main service node; dividing data in the correlation data column into a plurality of data groups through the main service mode if the data quantity is larger than a presser memory load threshold value; saving data in every data group and data of other data columns of the first table item through the main service node to enable the computing node to perform the table correlation operation of the first table item and a second table item according to the data in the corresponding data group and the corresponding data of the other data columns, wherein the data of the other data columns of the first table item are corresponding to the data in the corresponding data group.

Description

technical field [0001] The invention relates to the field of information technology, in particular to a multi-table association method, a main service node, a computing node and a system. Background technique [0002] At present, with the continuous development of the Internet and information technology, the application of multi-table association technology is becoming more and more extensive. The multi-table association technology specifically uses the same data column in two or more tables as an associated column to establish an association between multiple tables. In traditional databases, multi-table association operations are suitable for association operations between multiple tables with a small amount of data. When performing association operations between multiple tables with a large amount of data, it takes a long time and is less efficient. Low. [0003] Aiming at the problem that traditional databases cannot perform association operations between multiple table...

AI Technical Summary

Problems solved by technology

However, when performing multi-table association operations on the reduce side, a large amount of data transmission is required in the function shuffle stage, resulting in low efficiency of multi-table association operations; while performing multi-table association operations on the map side, due to the In this method, the data volume of at least one table needs to be less than or equal to the memory capacity of the machine, resulting in the multi-table association operation performed on the map side, which is not suitable for the association between multiple tables with large data volumes.

Images