Data processing method, device and system based on distributed file system

Abstract

The invention provides a distributed file system-based data processing method, a distributed file system based data processing device and a distributed file system based data processing system, wherein the method comprises the following steps: in the process that a data node simultaneously carries out communication interaction with a host master node and a backup master node in a double-master-node operating mode, monitoring the operating states of the host master node and the backup master node, and if determining that the host master node has a fault and the backup master node runs normally, sending a single master node operating mode switching instruction to the backup master node, so that the data node continues to carry out communication interaction with the backup master node in a single master node operating mode. Through the distributed file system-based data processing method, device and system provided by the invention, an effect that a data node carries out communication interaction with host / backup master nodes in two operating modes is achieved, the problem that the system interrupt time is overlong when the host master node has a fault is solved, and the effectiveness and stability of the system are greatly improved.

Description

technical field [0001] The present invention relates to the technical field of communication, in particular to a data processing method, device and system based on a distributed file system. Background technique [0002] Distributed file system (Hadoop Distributed File System, HDFS) is a master-slave structure system, including a master node and several data nodes, wherein, the data node mainly executes instructions from the master node, including block creation, deletion, and replication, and will The file blocks are stored in the local file system, the metadata of the file blocks are saved, and all existing file block information is sent to the master control node periodically. [0003] Since there is only one master control node in HDFS, once the master control node fails, the entire HDFS system will be paralyzed, causing a single-point bottleneck and affecting the availability of the entire system. Therefore, in the prior art, by configuring a standby master control nod...

AI Technical Summary

Problems solved by technology

[0004] However, due to the large amount of files on the active master control node, the time to import data to the standby master control node is too long, resulting in the interruption of the interaction between the active master control node and the data nodes for a long time, so that the entire HDFS system cannot Meet application needs

Images