A data cold backup method and system for log-structured storage engine

Abstract

The invention discloses a data cold backup method and system for a Log-structured storage engine. According to the method, based on the append storage characteristics of the Log-structured storage engine, a data automatic synchronization mechanism based on a file-level replaying mechanism is constructed, and a data cold backup solution for this type of storage engine is given with the data automatic synchronization mechanism being a core so as to realize continuous backup of data of the Log-structured storage engine with small resource expenditure and ensure that the backup data is available on any time node.

Description

technical field [0001] The invention relates to the technical field of data backup, in particular to a data cold backup method and system for a Log-structured storage engine. Background technique [0002] In recent years, with the vigorous development of Internet applications, a large amount of multimedia data such as video, audio, and pictures has been generated. In order to store these massive data, a large number of distributed NOSQL storage products have emerged, and a considerable part of them is based on Log. -structured mode storage engine, in view of its excellent read / write performance, such NOSQL storage products are deployed and applied by many Internet companies, and many companies use them to store PB or even EB-level data. For data security considerations, it is usually necessary to perform cold backup of data, that is, copy a copy of service data and place the copy in a non-service environment (that is, the backup copy is not served externally). When the data ...

AI Technical Summary

Problems solved by technology

But there are still some problems with this backup method

First of all, it is necessary to ensure that the full amount of data in the initial backup and the incremental data of subsequent backups are kept intact; secondly, when performing data recovery, the recovery work must be repeated and orderly based on the incremental copy. Errors in the order of the copies are likely to cause data loss, and even the problem of unusable data recovered; thirdly, since the recovery process involves many incremental copies, the recovery time is bound to be very long; finally, the incremental backup The problem of timeliness also exists. Although increasing the frequency of incremental backups can increase the timeliness of backup data, it will inevitably lead to an increase in incremental copies and backup resources, which will eventually bring more difficulties to the recovery work.

[0006] On the issue of cold backup of massive file data, the patent "A Data Backup Method" (Application No.: CN201510918534.7) related research, this patent performs data backup at the file level granularity, which is similar to the file synchronization software rsync, of course invented The author has optimized the comparison between the backup source and the backup target; however, the method proposed in the above patent does not control the order of file synchronization, and the Log-structured storage engine has strict requirements on the order of backup files. Any data Errors in the order of files or omissions of backup files will cause the storage engine to fail to start smoothly, resulting in fundamental damage to the stored data; in addition, according to the general garbage cleaning mechanism of the Log-structured storage engine, it is bound to be destroyed during the garbage cleaning process. Generate a large number of new data files and destroy a large number of invalid files at the same time; at this time, using a backup method similar to rsync will not be able to ensure the real-time availability of data files

[0007] To sum up, the current industry widely uses Log-structured storage engines to store massive data, but there is still a lack of better cold backup solutions for these data; the present invention is a solution researched and practiced to solve this problem

Images