Cross-cluster traffic optimization method for single-point failure recovery of cluster storage system

Abstract

The invention relates to a cross-cluster traffic optimization method for single-point failure recovery of a cluster storage system, involving the cluster storage system. 1: Divide the original data into fixed-size data blocks, encode the data blocks to generate corresponding check blocks, and then distribute them to different nodes for storage according to constraints; 2: Find the cross-cluster repair for each stripe that minimizes the stripe Traffic repair scheme; 3: Initialize the unrepaired stripe set, perform several rounds of repairs, randomly select multiple stripes from the unrepaired stripe set in each round to form the current round of repaired stripe set, and obtain a single stripe repair according to step 2 Combining the schemes to obtain the initial multi-strip repair combination plan, using the greedy algorithm to schedule the multi-strip repair, and obtaining the multi-strip repair combination plan for each round through the replacement step and the exchange step; 4: Complete several rounds of repair, each round is based on the repair The scheme and coding rules are used to recover corresponding invalid stripes. Minimize and balance cross-cluster repair traffic, applicable to any erasure code, and widely applicable.

Description

technical field [0001] The invention relates to a cluster storage system, in particular to a cross-cluster traffic optimization method for single-point failure recovery of the cluster storage system. Background technique [0002] A large-scale cluster storage system usually consists of hundreds or even thousands of storage servers (also called nodes), and a large number of nodes makes the system have to solve common unexpected failures. In order to ensure the reliability of data, the existing system mainly adopts the method of storing redundant data in advance, and the two most representative schemes are "backup" and "erasure code". Backup is to copy data into n parts and store them in n different devices. When a device fails, select the backup data in the remaining unfailed devices to restore. The backup method is simple and effective, but it is easy to cause a huge storage overhead, and its storage overhead is n times that of the original data. Erasure codes take fixed-s...

AI Technical Summary

Problems solved by technology

In order to alleviate the impact of cross-cluster repair traffic, existing work mainly focuses on designing new cluster-aware erasure codes ([4] H.Hou, P.Lee, K.W.Shum, and Y.Hu. Rack-aware RegeneratingCodes for Data Centers.IEEE Transactions on Information Theory,65(8):4730–4745,2019; [5]Y.Hu,X.Li,M.Zhang,P.Lee,X.Zhang,P.Zhou,and D.Feng .Optimal RepairLayering for Erasure-Coded Data Centers:From Theory to Practice.ACMTransactions on Storage,13(4):33,2017.), maintain the same fault tolerance with less cross-cluster repair traffic, or propose a new repair order Scheduling methods to reduce cross-cluster repair traffic ([6]Z.Shen,J.Shu,andP.Lee.Reconsidering Single Failure Recovery in Clustered File Systems.InProc.of IEEE / IFIP DSN,2016.), but these works all choose Designate a dedicated node to repair all lost blocks, such a repair scheme will easily cause the bandwidth of the dedicated node to become the bottleneck of repair performance

Images