Data processing method, device and system based on distributed file system

A distributed file and data processing technology, applied in the field of communication, can solve problems such as long time to import data, HDFS system cannot meet application requirements, etc., and achieve the effect of improving effectiveness and stability

Active Publication Date: 2016-02-10
CHINA UNITED NETWORK COMM GRP CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, due to the large amount of files on the active master control node, the time to import data to the standby master control node is too long, resulting in the interruption of the interaction between the active master control node and the data nodes for a long time, so that the entire HDFS system cannot Meet application needs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system based on distributed file system
  • Data processing method, device and system based on distributed file system
  • Data processing method, device and system based on distributed file system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] figure 1 It is a flowchart of an embodiment of the data processing method based on the distributed file system of the present invention, such as figure 1 As shown, the method specifically includes:

[0032] Step 100, during the process of the data node using the dual master control node working mode to simultaneously communicate and interact with the active master control node and the backup master control node, monitor the working status of the master master control node and the backup master control node, Wherein, the communication interaction includes: performing signaling interaction with the active master control node, and simultaneously performing data interaction with the active master control node and the backup master control node;

[0033] HDFS includes a master master node, a backup master node, and several data nodes. The initial files in the master master node and the backup master node are the same. The node and the standby master control node communicat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed file system-based data processing method, a distributed file system based data processing device and a distributed file system based data processing system, wherein the method comprises the following steps: in the process that a data node simultaneously carries out communication interaction with a host master node and a backup master node in a double-master-node operating mode, monitoring the operating states of the host master node and the backup master node, and if determining that the host master node has a fault and the backup master node runs normally, sending a single master node operating mode switching instruction to the backup master node, so that the data node continues to carry out communication interaction with the backup master node in a single master node operating mode. Through the distributed file system-based data processing method, device and system provided by the invention, an effect that a data node carries out communication interaction with host / backup master nodes in two operating modes is achieved, the problem that the system interrupt time is overlong when the host master node has a fault is solved, and the effectiveness and stability of the system are greatly improved.

Description

technical field [0001] The present invention relates to the technical field of communication, in particular to a data processing method, device and system based on a distributed file system. Background technique [0002] Distributed file system (Hadoop Distributed File System, HDFS) is a master-slave structure system, including a master node and several data nodes, wherein, the data node mainly executes instructions from the master node, including block creation, deletion, and replication, and will The file blocks are stored in the local file system, the metadata of the file blocks are saved, and all existing file block information is sent to the master control node periodically. [0003] Since there is only one master control node in HDFS, once the master control node fails, the entire HDFS system will be paralyzed, causing a single-point bottleneck and affecting the availability of the entire system. Therefore, in the prior art, by configuring a standby master control nod...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L1/22H04L29/08
Inventor 贾兴华张云勇陈清金
Owner CHINA UNITED NETWORK COMM GRP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap