A File Appending Method Based on Erasure Code

Abstract

The invention discloses an erasure code based file appending method and aims to reduce code computation overhead and network transmission overhead for file appending. According to the technical scheme, the method includes the steps: a distributed storage system composed of a control node and N storage nodes is established, with the control node being provided with a data reading-writing management program and each storage node being provided with a data reading-writing program; the data reading-writing management program reads original data blocks from the storage nodes where a last set of original data blocks locates, the data blocks AD are appended to the last set of original data blocks, redundant data blocks are deleted from the storage nodes where redundant data blocks corresponding to the last set of original data blocks locate, and the last set of k original data blocks of a data file is encoded and calculated again with a stored encoding coefficient so as to obtain appended redundant data blocks C1-Cm; the data reading-writing management program write newly-appended data blocks and the appended redundant data blocks C1-Cm into the different storage nodes, respectively. The erasure code based file appending method has the advantages that code computation overhead and network transmission overhead can be maximally reduced and file appending efficiency is effectively ensured.

Description

technical field [0001] The invention relates to a data fault-tolerant technology of a distributed storage system, in particular to a file appending method based on an erasure code. Background technique [0002] Data fault tolerance is one of the key technologies for distributed storage of massive data, and data fault tolerance is crucial to improving system availability and reliability. Currently, data fault-tolerant technologies mainly include fault-tolerant technologies based on replication and fault-tolerant technologies based on erasure codes. [0003] The fault-tolerant technology based on replication is simple and intuitive, and easy to implement and deploy. It needs to create several copies of the same size for each data file, and the storage space overhead is large. The error-tolerant technology based on erasure code divides the data file into multiple data blocks, and uses the erasure code algorithm to encode the original data block to obtain redundant data blocks,...

AI Technical Summary

Problems solved by technology

With the continuous increase of data files, the number of data blocks in the file shows a gradual increase trend, which makes the encoding calculation overhead and network transmission overhead of file appending more and more large, which affects the efficiency of file appending.

Images