A data distribution method and management node

Abstract

The present invention discloses a data distribution method and a management node, wherein the method includes: according to the received cross-connection request, obtaining the number of sub-tables included in table A and table B that need to be cross-connected, the number of sub-tables The serial number, the number of nodes and the serial number of the node; the serial numbers of the sub-tables included in table A or table B are set in the direction of abscissa or ordinate in turn, and are respectively used as the length or width of the large rectangle to generate a large rectangle; according to The method of construction divides the large rectangle into two sub-rectangles, which are respectively the first sub-rectangle and the second sub-rectangle; the first sub-rectangle and the second sub-rectangle are evenly divided to form several small rectangles, each The small rectangle corresponds to the serial number of a computing node, and corresponds to the serial number of the subtable; the subtables corresponding to each small rectangle are distributed to the computing nodes corresponding to each small rectangle. The invention can reduce the amount of transmitted data and lighten the burden of network transmission.

Description

technical field [0001] The invention relates to the technical field of distributed data management, in particular to a data distribution method and a management node. Background technique [0002] Usually, using an ordinary computer to process hundreds of gigabytes (gigabits) of data, such as log files, often takes several hours or longer, and processing tens of terabytes (terabytes) of data may take hundreds of hours. For this reason, people use the method of data distribution to solve the above problems. The data distribution refers to that the collection of all data is called a table, and the collection of a part of all data is called a sub-table. The table includes several sub-tables, and the several sub-tables are distributed to several computing nodes, for example, 100 computing nodes, and each computing node is only responsible for the calculation of a part of all data. In this way, these computing nodes can perform calculations at the same time, which can greatly re...

AI Technical Summary

Problems solved by technology

When it is necessary to perform cross join calculation between two tables, that is, ({5}, {4}) cross join ({3}, {2}, {6}), the following problem occurs: Completing the cross join calculation Calculations between {5} and {6}, {4} and {3}, and {4} and {2} need to be performed, but these two subtables are on different computing nodes, so the calculation cannot be performed

However, the existing data distribution method will cause a large amount of data to be transmitted to the computing nodes, and the burden of network transmission will be heavy, especially when the number of computing nodes for data distribution is large and the number of sub-tables of the two tables is close , the problem of excessive data volume is more obvious

There is currently no technology that can reduce the amount of data transferred

Images