Cross-cluster traffic optimization method for single-point failure repair of cluster storage system

Abstract

The invention discloses a cross-cluster traffic optimization method for single-point failure repair of a cluster storage system, and relates to the cluster storage system. The method comprises the following steps: 1, dividing original data into data blocks in fixed sizes, encoding the data blocks to generate corresponding check blocks, and distributing the check blocks to different nodes for storage according to constraint conditions; 2, finding a repair scheme for minimizing the cross-cluster repair traffic of the strip for each strip; 3, initializing an unrepaired strip set, performing several rounds of repair, randomly selecting a plurality of strips from the unrepaired strip set in each round to form a current round of repaired strip set, obtaining an initial multi-strip repair combination scheme according to the single-strip repair scheme combination obtained in the step 2, scheduling repair of the plurality of strips by utilizing a greedy algorithm, and obtaining a multi-strip repair combination scheme in each round through a replacement step and an exchange step; 4, completing a plurality of rounds of repair, and performing corresponding failure strip recovery in each roundaccording to a repair scheme and a coding rule. The method minimizes and balances the cross-cluster repair traffic, is suitable for any erasure codes, and is wide in application range.

Description

technical field [0001] The invention relates to a cluster storage system, in particular to a cross-cluster traffic optimization method for single-point failure recovery of the cluster storage system. Background technique [0002] A large-scale cluster storage system usually consists of hundreds or even thousands of storage servers (also called nodes), and a large number of nodes makes the system have to solve common unexpected failures. In order to ensure the reliability of data, the existing system mainly adopts the method of storing redundant data in advance, and the two most representative schemes are "backup" and "erasure code". Backup is to copy data into n parts and store them in n different devices. When a device fails, select the backup data in the remaining unfailed devices to restore. The backup method is simple and effective, but it is easy to cause a huge storage overhead, and its storage overhead is n times that of the original data. Erasure codes take fixed-s...

AI Technical Summary

Problems solved by technology

In order to alleviate the impact of cross-cluster repair traffic, existing work mainly focuses on designing new cluster-aware erasure codes ([4] H.Hou, P.Lee, K.W.Shum, and Y.Hu. Rack-aware RegeneratingCodes for Data Centers.IEEE Transactions on Information Theory,65(8):4730–4745,2019; [5]Y.Hu,X.Li,M.Zhang,P.Lee,X.Zhang,P.Zhou,and D.Feng .Optimal RepairLayering for Erasure-Coded Data Centers:From Theory to Practice.ACMTransactions on Storage,13(4):33,2017.), maintain the same fault tolerance with less cross-cluster repair traffic, or propose a new repair order Scheduling methods to reduce cross-cluster repair traffic ([6]Z.Shen,J.Shu,andP.Lee.Reconsidering Single Failure Recovery in Clustered File Systems.InProc.of IEEE / IFIP DSN,2016.), but these works all choose Designate a dedicated node to repair all lost blocks, such a repair scheme will easily cause the bandwidth of the dedicated node to become the bottleneck of repair performance

Images