Data distribution method and management node

Abstract

The invention discloses a data distribution method and a management node. The data distribution method comprises the steps of obtaining the amounts and serial numbers of sub-tables and nodes respectively included by a table A and a table B which need to be in cross connection according to a received cross connection request; sequentially arranging the serial numbers of the sub-tables included by the table A or the table B in the direction of the horizontal axis or longitudinal axis and respectively using the serial numbers as length sides or width sides of large rectangles to form a whole large rectangle; using a construction method to divide the large rectangle into two sub-rectangles, namely the first sub-rectangle and the second sub-rectangle; respectively and evenly dividing the first sub-rectangle and the second sub-rectangle to form a plurality of small rectangles, wherein each small rectangle corresponds to a serial number of one computational node and corresponds to the serial number of each sub-table; distributing the sub-tables corresponding to the small rectangles onto the computational nodes corresponding to the small rectangles. By adopting the data distribution method, the volume of transmitted data can be decreased, and the network transmission burden can be relieved.

Description

technical field [0001] The invention relates to the technical field of distributed data management, in particular to a data distribution method and a management node. Background technique [0002] Usually, using an ordinary computer to process hundreds of gigabytes (gigabits) of data, such as log files, often takes several hours or longer, and processing tens of terabytes (terabytes) of data may take hundreds of hours. For this reason, people use the method of data distribution to solve the above problems. The data distribution refers to that the collection of all data is called a table, and the collection of a part of all data is called a sub-table. The table includes several sub-tables, and the several sub-tables are distributed to several computing nodes, for example, 100 computing nodes, and each computing node is only responsible for the calculation of a part of all data. In this way, these computing nodes can perform calculations at the same time, which can greatly re...

AI Technical Summary

Problems solved by technology

When it is necessary to perform cross join calculation between two tables, that is, ({5}, {4}) cross join ({3}, {2}, {6}), the following problem occurs: Completing the cross join calculation Calculations between {5} and {6}, {4} and {3}, and {4} and {2} need to be performed, but these two subtables are on different computing nodes, so the calculation cannot be performed

However, the existing data distribution method will cause a large amount of data to be transmitted to the computing nodes, and the burden of network transmission will be heavy, especially when the number of computing nodes for data distribution is large and the number of sub-tables of the two tables is close , the problem of excessive data volume is more obvious

There is currently no technology that can reduce the amount of data transferred

Images