An Adaptive Coding Storage Error Tolerance Method Based on Simple Regenerating Codes

Abstract

The invention discloses a self-adaptive encoding storage fault-tolerant method based on a simple regenerating code. The method comprises the following four main steps: 1) saving a file stored in the HDFS in a three-copy mode; 2) after one time slot, converting the file into a simple regenerating code fast code encoding; 3) after the file access is reduced, converting the file into the simple regenerating code compact code encoding; 4) possibly generating the condition that the file encoding states in the step 2) and step 3) are mutually converted along the dynamic attribute of the file in the running. Through the simultaneous adoption of the two encoding states of simple regenerating code fast code compact code and the fast code in the distributed file system HDFS, the saving expenditure and the repair cost are compromised; the self-adaptive encoding method based on the simple regenerating code is introduced into the HDFS so as to establish the self-adaptive encoding mechanism in combination with the file state and the system state; the integral saving efficiency of the distributed storage system is improved, and the repair cost is lowered.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to an adaptive encoding and storage fault-tolerant method based on simple regeneration codes. Background technique [0002] Since the use of erasure codes can achieve high reliability with low storage overhead, large-scale distributed storage systems have begun to use erasure codes. However, most online storage systems use only one erasure code, and the erasure code actually used in the system may be optimized in terms of reducing repair bandwidth overhead or reducing storage overhead. For example, RS coding is a widely used type of erasure code, which has been used in many storage systems such as Google's ColossusFS and FaceBook's HDFS-Raid. The RS code used in ColossusFS has a storage overhead of 1.5x, requiring 6 disk reads and network transfers to repair a data block. The storage overhead of the RS code used in HDFS-Raid is 1.4x, but repairing a data block requires reading o...

AI Technical Summary

Problems solved by technology

For cold data, since it is not accessed very frequently, it is less likely to cause degenerate reads due to data unavailability

Images