Incremental multi-replica data synchronization method and system

Abstract

The invention discloses an incremental multi-replica data synchronization method and system. The method includes the steps that a first operation record of a local replica and a second operation record of a far-end replica are acquired; whether the first operation record and the second operation record are consistent or not is judged; if not, synchronization operation is executed according to inconsistent content in the first operation record and the second operation record to ensure that the local replica and the far-end replica are consistent, and the synchronization progress is recorded. According to the data synchronization method and system, metadata is stored in a metadatabase, and as neither modification of content data nor modification of the metadata involves modification of the metadatabase, incremental synchronization can be achieved according to a modification log of the replica metadatabase; besides, the modification history of large files is deduced according to transaction logs of data, it is not needed to maintain extra log files, on the aspect of consistency, it is only needed to guarantee the consistency of the metadatabase and data files, and thus the implementation cost is greatly lowered.

Description

technical field [0001] The invention relates to the technical field of data synchronous storage, in particular to an incremental multi-copy data synchronization method and system. Background technique [0002] The rapid development of the Internet has produced a large number of pictures, documents and other files. These files are characterized by small size (0-100KB per move) and huge number (hundreds of millions). The traditional POSIX interface file system has been difficult to meet Demand, this is the industry's famous problem of massive small files. [0003] For the problem of massive small files, the common practice in the industry is to merge and store small files into a traditional POSIX file. In addition, in order to ensure the availability and fault tolerance of the system, these small files are usually stored in multiple copies. Multiple copies lead to a basic problem in distributed storage systems, that is, copy consistency. At present, the existing merged stora...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to propose an incremental multi-copy data synchronization method and system to solve the need for additional maintenance of the modification log for the operation records of small files in the prior art, in order to ensure metadata, content data, index information and modification The consistency of the log requires a more complicated rollback mechanism, which increases the cost of synchronization implementation

Images