Method, device and system for file synchronization

Abstract

A file synchronization method, device, and system. The method comprises the following steps: a copy management device sending a last directory operation number of a previously synchronized file to a name node of a first HDFS cluster; receiving information about a file to be synchronized sent by the name node of the first HDFS cluster, the information about the file to be synchronized being information about a file corresponding to a directory operation number that is subsequent to the last directory operation number and determined by the name node of the first HDFS cluster; determining a target synchronization task according to the information about the file to be synchronized; a target copy execution device synchronizing, according to information about at least one file in the target synchronization task, the at least one file from a source data node to a destination data node. The synchronization method does not need to scan the entire file directory, thus improving the efficiency of file synchronization.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a file synchronization method, device and system. Background technique [0002] Hadoop Distributed File System (Hadoop Distributed File System, HDFS) is a highly fault-tolerant system that can provide high-throughput data access and is suitable for deploying big data services. [0003] HDFS can include multiple HDFS clusters. The layout of HDFS clusters adopts a master-slave structure. An HDFS cluster includes a master node (NameNode, NN) and several data nodes (DataNode, DN), and the master node is responsible for managing the HDFS cluster. Data, for example: information that manages files stored in the cluster. Data nodes are responsible for storing files. [0004] In the current big data environment, data synchronization, especially cross-region and cross-cluster data synchronization, is becoming more and more important and has broad application requirements. [000...

AI Technical Summary

Problems solved by technology

[0009] In order to solve the problem of low efficiency of HDFS cluster file synchronization in the prior art, the embodiment of the present invention provides a method for file synchronization, which can directly determine the directory operation number following the directory end operation number according to the directory end operation number of the previous synchronization The information of the corresponding file, and then determine the information of the file to be synchronized, and synchronize the file to be synchronized without scanning the entire file directory, thereby improving the efficiency of file synchronization

Images