Distributed file synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a distributed file synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a file synchronization request issued by a distributed file system, and verifying the file synchronization request; after the verification is passed, creating a synchronization task, and storing file data corresponding to the synchronization task in the distributed file system to a data storage center; when it is detected that data change occurs in the data storage center, triggering a synchronization task, and sending an instruction to the queue system; and enabling the queue system to send the instruction to the transfer cluster, and enabling the transfer cluster to execute file data synchronization according to the instruction. According to the method, the dependency relationship of the synchronous sub-tasks can be ensured, correct execution of the tasks is ensured, the cluster configuration management complexityis reduced, and task redo and custom interface triggering are supported.

Description

technical field [0001] The present invention relates to the technical field of distributed file synchronization, and more specifically refers to a distributed file synchronization method, device, computer equipment and storage medium. Background technique [0002] Now HDFS (Hadoop Distributed File System) file synchronization across computer rooms is realized by using Hadoop distcp (distributed data copy tool) through timing scripts; Hadoop distcp (distributed data copy tool) tool requires inter-cluster network communication, but cross-computer room cluster The inter-network is often disconnected, so a transfer cluster is needed for data transfer. As a result, a synchronization task contains multiple subtasks, and the execution of multiple subtasks has a sequential dependency. Only when the previous subtask succeeds will the next subtask be executed, which is triggered regularly. The method cannot guarantee the accurate execution of tasks in sequence, and the implementation ...

AI Technical Summary

Problems solved by technology

[0002] Now HDFS (Hadoop Distributed File System) file synchronization across computer rooms is realized by using Hadoop distcp (distributed data copy tool) through timing scripts; Hadoop distcp (distributed data copy tool) tool requires inter-cluster network communication, but cross-computer room cluster The inter-network is often disconnected, so a transfer cluster is needed for data transfer, which leads to a synchronization task containing multiple subtasks, and the execution of multiple subtasks has a sequential dependency. Only when the previous subtask succeeds will the next subtask be executed, which is triggered regularly The method cannot guarantee the accurate execution of tasks in sequence. The implementation of timing scripts has disadvantages such as complex cluster configuration management, no support for task redo, and no support for custom interface triggers.

Images