Full data export method, data export task distribution device and data export node device

Abstract

The present invention provides a method for exporting full data, a data export task allocation device, and a data export node device. The method includes: the data export task allocation device analyzes and manages the service unit of the data unit for each data unit in the data table to be exported , wherein, the data table to be exported includes at least one data unit; the data export task allocation device assigns each data unit export task in the data table to be exported to the service unit that manages the data unit. The host, so that the data export node device deployed on the host performs data export after weighted and average division of the data units it manages. Through the invention, it is possible to reduce network IO on the basis of improving data export efficiency.

Description

technical field [0001] The present invention relates to the communication field, and in particular to a method for exporting full data, a device for allocating data export tasks, and a device for data export nodes. Background technique [0002] Hbase is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. For the data in the query table, it provides two methods: Get and Scan. The Get method is used to obtain the only record according to the specified Rowkey. , and in the Scan method, by limiting the StartRowkey and EndRowkey, all records whose Rowkey is between StartRowkey and EndRowkey can be obtained at one time. The design characteristics of HBase determine that the efficiency of data retrieval based on Rowkey is very high, but if the retrieval condition is an ordinary column, a full table scan is required, that is, a Scan query object that does not specify StartRowkey and EndRowkey is constructed, and a request is initiated to e...

AI Technical Summary

Problems solved by technology

[0004] However, the MapReduce batch task method generally allocates tasks according to the CPU idleness of each node in HBase. Since the task to be exported may not exist in the host where the node is located, it is necessary to retrieve data from other hosts, resulting in There are a lot of data copies between the hosts in the cluster, which will generate high network IO. In extreme cases, it will affect the communication between the service processes in the cluster, but the CPU of the host is very idle, and various resources of the cluster have to be exhausted. to balanced utilization

In addition, due to the MapReduce batch task method, the data to be exported is generally distributed to each node according to the intensity of "one Region, one task". However, due to the large difference in the amount of data between each Region, there may be 80% of the tasks run only 20% of the time, and 20% of the tasks run 80% of the time, resulting in low data export efficiency

Images