Data allocation strategy in hadoop heterogeneous cluster

Abstract

The invention relates to a data allocation strategy in a hadoop heterogeneous cluster. The data allocation strategy is characterized by comprising the following steps of: step S01, testing and storing execution time for each node to treat data with different scales, and transforming the execution time into a static performance reference index; step S02, monitoring and storing storage load of each node and network transmission speed between the nodes, and transforming the storage load and the network transmission speed into dynamic performance reference indexes; and step S03, calculating the quantity of data blocks to be allocated to each node by utilizing a calculating module according to preset weight of each performance factor, and performing data block-node mapping and carrying out allocation transmission by using a data allocation server. Through flexible configuration of each performance factor of the static and dynamic performance reference indexes, the data allocation strategy disclosed by the invention enhances the adaptability, ensures the effectiveness, effectively increases the data locality, reduces the operation response time and network transmission, improves the load stability of the system, and optimizes the cluster resources.

Description

technical field [0001] The invention relates to a data distribution strategy in the field of high-performance clusters, in particular to a data distribution strategy in a Hadoop heterogeneous cluster based on comprehensive consideration of multi-performance factors such as node computing capability, network transmission capability and node load capability. Background technique [0002] Hadoop is a software framework capable of distributed processing of large amounts of data. Its high reliability, high scalability, high efficiency and high fault tolerance make it widely concerned in the field of business and research. Hadoop includes two relatively independent subsystems: the distributed parallel computing system MapReduce consists of JobTrackers and TaskTrackers; the distributed storage system HDFS stores files on all storage nodes in the Hadoop cluster. When executing MapReduce tasks, it is necessary to obtain the corresponding data blocks on HDFS for processing. In order t...

AI Technical Summary

Problems solved by technology

However, these strategies do not fully consider the differences in the inherent capabilities of each node in the heterogeneous cluster, such as thread switching capabilities, node storage capabilities, etc., resulting in insufficient consideration of data distribution issues and cannot ensure network stability.

Other methods also consider network transmission, but the network distance is estimated by the sum of the distance from each node in the topology structure to the nearest common ancestor. In the different network bandwidths of practical applications, this estimation has great limitations. sex, insufficient reference value

Images