Large file distributed backup method and device based on erasure correction technology

Abstract

The invention discloses a large file distributed backup method and device based on erasure correction technology, and the method comprises the steps that a backup proxy server gradually caches large file data into a memory according to the received large file backup request information of a client; the backup proxy service extracts the big file data pre-read into the memory according to the fixedblock size of each MB to obtain n source data blocks; the redundancy level of the erasure code is set as n + m, and m check data blocks are calculated based on the redundancy of the n source data blocks; metadata information disassembled into the data blocks is generated; the backup proxy service asynchronously and parallelly distributes the data blocks to n + m backup servers for distributed backup storage; and the backup proxy service monitors the backup data blocks of the backed-up large files. According to the method, the backup speed of the large file is increased, the check data block can provide secondary protection for the backup data with the minimum redundancy, and the reliability of the backup data is guaranteed while the backup storage space is saved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a large file distributed backup method and device based on erasure correction technology. Background technique [0002] File backup refers to the process of copying all or part of the file collection from the hard disk or array of the application host to other storage media in order to prevent file loss due to system operation errors or system failures. [0003] For file backup, the more mainstream backup system is basically single-point backup, that is, the backup target file is transferred to a designated storage medium through the network. The backup process is generally completed by a backup server, and the backup medium is generally a single Disk, or a single storage logical volume mounted on the backup server, the backup speed is greatly affected by the performance of the backup server and the quality of the backup media. Especially for large files, the backup method is ...

AI Technical Summary

Problems solved by technology

Especially for large files, the backup method is generally serial reading and then backup writing, which leads to a long backup time for a single large file

[0004] In addition, the backup mode does carry out secondary protection for the backed up data, but it happens from time to time that the backup cannot be restored due to the damage of the backup data

In the case of a clear need for production, it is generally only a mechanical backup of another backup data, which puts a lot of pressure on the storage capacity.

[0005] To sum up, there are certain bottlenecks in the performance of traditional backup methods, especially for large file backups, and lack of efficient and low-consumption methods for secondary protection of backup data

Images