Hadoop cluster file backup system and method

A hadoop cluster and file backup technology, applied in transmission systems, electrical components, digital data protection, etc., can solve problems such as inability to meet data backup requirements, lack of a backup system, and many things

Pending Publication Date: 2020-07-07
INFORMATION2 SOFTWARE SHANGHAI
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the existing data backup technology still generally relies on snapshots for backup. There are many things to configure and complicated to use. There is no simple and perfect backup system, which cannot meet the data backup needs of the big data era.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop cluster file backup system and method
  • Hadoop cluster file backup system and method
  • Hadoop cluster file backup system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0045] image 3 It is a schematic flow diagram of the main flow (TaskMain) of Hadoop file backup in the embodiment of the present invention, Figure 4 It is a detailed process for generating a data status list in the main process (TaskMain) in the embodiment of the present invention. Such as image 3 and Figure 4 , the Hadoop cluster file backup process of the present invention is as follows:

[0046] First, the main process (TaskMain) generates a Hadoop temporary file list: TaskMain traverses the directories that need to be backed up on the Hadoop file system, and obtains relevant file information (loop Hadoop files); TaskMain sends the obtained file information to the standby master node; The end master node queries the locally stored file status database (stores the data storage status of each backup data node) to obtain the information of the backup file; the backup master node compares the information of the two files to generate the final file FILE_ACTION, that is, t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Hadoop cluster file backup system and a Hadoop cluster file backup method. The system includes: a hadoop cluster which traverses a directory needing to be backed up on the cluster Hadoop file system, acquiring related file information and sending the related file information to a standby end main node, storing the file list information into a Hadoop temporary file list temporarily, processing the temporary file list information one by one, establishing connection according to the distributed target data node, and sending the data of each file in the temporary file list to the distributed standby end data node; a standby end main node which queries a local file state database when receiving the file information sent by the cluster, obtains the information of a standby end file, compares the file information sent by the cluster with the information of the standby end file, obtains file list information needing to be backed up at this time, and sends the file list information to the cluster; and a plurality of standby end data nodes which are used for receiving the file data sent by the cluster and carrying out state synchronization with the standby end mainnode.

Description

technical field [0001] The invention relates to the field of computer data backup and disaster recovery, in particular to a Hadoop cluster file backup system and method. Background technique [0002] With the popularization of computers and the advancement of information technology, especially the rapid development of computer networks, information has increasingly become an important basis for the survival and development of countries and enterprises, and has become the focus of attention of individuals, enterprises and society. Today's information centers are becoming more and more complex. Not only is the size of the system doubling every year, but the complexity of the system and the risks it faces are also increasing. However, as an important means of information protection, the importance of data backup is often overlooked by people. In fact, as long as data transmission, data storage, and data exchange occur, data failure may occur. At this time, if appropriate data ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/182G06F16/178G06F11/14G06F21/62H04L29/08
CPCG06F16/182G06F16/178G06F11/1464G06F21/6218H04L67/10
Inventor 温立涛杨彬陈勇铨周华
Owner INFORMATION2 SOFTWARE SHANGHAI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap