Copy access method and device for Hadoop distributed file system and Hadoop distributed file system

Abstract

The invention discloses a copy access method and device for a Hadoop distributed file system and the Hadoop distributed file system so as to balance system load and accordingly improve reading / writing performance. The method comprises the steps that node load information of data nodes in the Hadoop distributed file system is acquired; copies with same block data are stored in a data node where a client is or a data node with the minimum load when stored according to the node load information; the copies are read from the data node with the minimum load and / or from the data node at the minimum distance from the client when read. According to the method, node load information of each data node is fully considered, so that the system load can be balanced, and the read-write operation performance is improved.

Description

technical field [0001] The invention relates to the field of distributed architecture, in particular to a copy access method and device of a Hadoop distributed file system and the Hadoop distributed file system. Background technique [0002] The distributed file system (Hadoop Distributed File System, HDFS) is a master-slave structure, and an HDFS cluster is composed of a name node and at least one data node. The name node in the HDFS cluster is a master server that manages the file namespace and regulates client access to files. It mainly operates file or directory operations in the file namespace, such as opening, closing, renaming, etc. At the same time, the name node also uses It is used to determine the mapping between blocks and data nodes. The data nodes in the HDFS cluster store specific data files, are responsible for read and write requests from file system clients, and also execute block creation, deletion, and block copy instructions from the name node. [0003...

AI Technical Summary

Problems solved by technology

[0005] However, the HDFS copy placement and reading strategies provided by the above-mentioned prior art only consider the network overhead of write / read operations, but do not consider the load balancing of other loads injected into the system such as CPU and disk, which may lead to serious uneven load on the system , thus affecting the performance of write / read operations

Images