A method for real-time updating of remote files based on content blocks

Abstract

The invention discloses a content-defined chunking remote file real-time updating method. The invention aims at providing a file increment updating method with low network transmission expenditure for a distributive storage system, so that the distributive storage system can support the random writing of a file at low expenditure. According to the technical scheme, the method comprises the following steps: storing an original file fold file in a chunking manner, calculating an abstract for each data chunk by adopting a Hash algorithm, calculating one fingerprint for the beginning and the ending of each data chunk by adopting a rabin-fingerprinting algorithm, contrasting the chunking information of the original file fold by adopting a content-defined chunking data chunking way, chunking a novel file Fnew, calculating the abstract for each data chunk of Fnew by adopting the Hash algorithm, comparing the abstracts of the Fold data chunk and the abstract of the Fnew data chunk, finding out the varied data chunk, deleting the data chunk to be deleted from the Fold, and adding the data chunk to be updated into Fold. By adopting the method, the random writing of the file can be supported by the distributive storage system at low expenditure.

Description

technical field [0001] The invention relates to a data update method in a distributed storage system, in particular to a data update method in a distributed storage system supporting wide area network level applications. Background technique [0002] With the rapid development of cloud storage, storage technology is undergoing revolutionary changes. Traditional file systems can no longer meet the needs of massive data storage, and distributed storage systems have emerged as the times require. Typical distributed storage systems include master-slave distributed file systems such as Google's GFS (Google File System), open source project Hadoop's HDFS, and flat ring-structured key-value storage systems such as Amazon's Dynamo, Facebook Cassandra et al. [0003] Distributed storage systems have shown unique advantages in terms of storage capacity, scalability, reliability, and performance. Therefore, they are more and more widely used in massive data processing and are gradual...

AI Technical Summary

Problems solved by technology

The existing method mainly sends the updated files back to the distributed storage system completely, which will increase the network transmission overhead and affect the IO network performance of the storage system, especially in the case of a large amount of IO access

[0005] How to solve the problem that the distributed storage system has high update overhead and generally does not support random writing of files is an important technical issue concerned by those skilled in the art. It can effectively reduce the network transmission overhead in the file update process and can be applied to distributed storage systems that support WAN-level applications. storage system, but the current mainstream GFS, HDFS and other distributed storage systems also have deficiencies. They only support file append operations, but do not support random file writes.

Images