Construction method of local recoverable code

Abstract

The invention provides a construction method of a local recoverable code. The construction method comprises the following implementation steps of: initializing parameters; constructing a Vandermonde matrix and dividing the Vandermonde matrix; constructing an addition matrix L; constructing a generation matrix G; and obtaining a construction result of the local recoverable code. According to the method, one local check block and the global check block and the original data block which are linearly related to the local check block are deployed in each cluster, so that the advantage that only other nodes in the cluster need to be read when one node in one cluster is recovered is realized, and cross-cluster bandwidth is avoided; and the generation matrix is constructed in a mode of constructing the compression matrix and then constructing the addition matrix, so that the generated local check block can be expressed as a linear combination of the global check block and the original data block in the same cluster, the defect that all the original data blocks need to be read when the global check block is repaired is avoided, and the data recovery efficiency is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer coding and relates to a construction method of a local recoverable code, which can be used in a distributed storage system. Background Art

[0002] A distributed storage system is a data management technology that divides data into multiple blocks and stores them on different nodes to achieve load balancing and redundancy. The storage architecture of existing distributed storage systems usually adopts a hierarchical architecture, organizing storage nodes into clusters.

[0003] Erasure Coding (EC) is a data protection technology used to improve data reliability and storage efficiency in distributed storage systems. It divides data into multiple smaller data blocks through mathematical algorithms and generates a certain number of check blocks. These check blocks contain redundant information of the original data blocks, so that the system can reconstruct the lost data through the remaining data blocks and check blocks when some data blocks are lost, which significantly reduces the storage space requirements while ensuring data integrity.

[0004] Locally Recoverable Codes (LRC) are a special type of erasure code that is used to improve the efficiency of data recovery in distributed storage systems. The technical idea of ​​constructing LRC is to first construct a generator matrix, which includes three sub-matrices: the identity matrix, the generator matrix of the global check block, and the generator matrix of the local check block. Then, the generator matrix is ​​used to encode the original data to obtain a codeword with local recovery characteristics.

[0005] Existing methods for constructing local recoverable codes, such as a patent application with application publication number CN113258938A and titled “A method for constructing an erasure code for rapid repair of a single-node fault”, discloses a method for constructing an erasure code for rapid repair of a single-node fault, comprising: encoding in the horizontal direction and the vertical (longitudinal) direction respectively, firstly encoding in the horizontal direction to generate a horizontal local check and a global check to ensure the fault tolerance of the encoding; performing XOR encoding in the vertical direction to obtain a check, which solves the problem of needing to read all node data when restoring the original data block and the high recovery cost. However, since the global check block is not used when generating the local check block, and the cluster is not considered when deploying the codeword, a technical problem of low recovery efficiency is caused during recovery. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a method for constructing a local recoverable code to solve the technical problem of low data recovery efficiency in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Initialization parameters:

[0009] Initialize the local recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks and local check blocks to be deployed in each cluster is β, α and 1 respectively, where α≥1, Z≥1. First determine α and Z, calculate β=αz-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y = αZ, the yth global check block is u y , the zth local check block is v z , N=K+Y+Z=αZ 2 +Z, the nth code element is s n ,s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254}mod g(x), g(x) is any 8th-order irreducible polynomial;

[0010] (2) Construct the Vandermonde matrix and divide it:

[0011] Construct a Vandermonde matrix O of dimension (Y+1)×K and divide O into sub-Vandermonde matrices of dimension Y×K and a row vector l consisting of K elements;

[0012] (3) Constructing the sum matrix

[0013] The row vector l is expanded to obtain an expanded matrix L with a dimension of Z×K, and the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K

[0014] (4) Construct the generator matrix G:

[0015] Through the identity matrix I of dimension K×K K , and the sub-Vandermonde matrix Sum and Add Matrix Construct a generator matrix G with dimension N×K;

[0016] (5) Obtain the construction result of the local recoverable code:

[0017] The codeword S is calculated by the original data T and the generator matrix G, and each The original data blocks, A global check block and a local check block are deployed to each cluster to obtain a local recoverable code.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] (1) The present invention deploys a local check block and a global check block and an original data block linearly related to the local check block in each cluster, thereby achieving the advantage that when restoring a node in a cluster, only other nodes in the cluster need to be read, thus avoiding cross-cluster bandwidth and effectively improving data recovery efficiency.

[0020] (2) The present invention constructs a generation matrix by constructing a compression matrix and then reconstructing a sum matrix, so that the generated local check block can be expressed as a linear combination of the global check block and the original data block under the same cluster, avoiding the defect of needing to read all the original data blocks when repairing the global check block, and further improving the data recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the implementation of the present invention;

[0022] Figure 2 It is a layout diagram of locally recoverable codes in a cluster according to a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Reference Figure 1 , the present invention comprises the following steps:

[0025] Step 1) Initialize parameters:

[0026] Initialize the local recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks and local check blocks to be deployed in each cluster is β, α and 1 respectively, where α≥1, Z≥1. First determine α and Z, calculate β=αz-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y = αZ, the yth global check block is u y , the zth local check block is v z, N=K+Y+Z=αZ 2 +Z, the nth code element is s n ,s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254}mod g(x), g(x) is any 8th-order irreducible polynomial;

[0027] In this embodiment, α=2, Z=3, and the original data block is T={t1, t2, t3, t4, t5, t6, t7, t8, t9, t 10 ,t 11 ,t 12}, Y = 6, β = 4, the locally recoverable code S = {s1,s2,s3,s4,s5,s6,s7,s8,s9,s 10 ,s 11 ,s 12 ,s 13 ,s 14 ,s 15 ,s 16 ,s 17 ,s 18 ,s 19 ,s 20 ,s 21};

[0028] Step 2) Construct the Vandermonde matrix and partition it:

[0029] Construct a Vandermonde matrix O of dimension (Y+1)×K and divide O into sub-Vandermonde matrices of dimension Y×K and a row vector l consisting of K elements;

[0030] Vandermonde matrix O, and sub-Vandermonde matrix and row vector l, the expressions are:

[0031]

[0032] l=[o Y+1,1 Lo Y+1,k Lo Y+1,K ]

[0033] o y,k =2 (y-1)×(k-1) modg(x)

[0034] Among them, y,k is the element in the yth row and kth column of O, mod means modular operation, that is, the element is written in the form of a polynomial and then the remainder is obtained after division operation;

[0035] In this embodiment, the constructed 7×12-dimensional Vandermonde matrix O is:

[0036]

[0037] The sub-Vandermonde matrix g and vector l are:

[0038]

[0039] l=[1 64 20545143 96 3618170 80185 97]

[0040] Step 3) Construct the sum matrix

[0041] The row vector l is expanded to obtain an expanded matrix L with a dimension of Z×K, and the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K

[0042] The expansion matrix L and the contraction matrix The expressions are:

[0043]

[0044] Among them, l z,k represents the element in the zth row and kth column of L, h z,k express The element at row z and column k, represents the rounding up operation, and ∑ represents the summation operation;

[0045] The element in the zth row and kth column is obtained by replacing l z,k and h z,k They are written as 8-bit binary numbers and then XORed. The expression is:

[0046]

[0047] In this embodiment, the extended matrix L is:

[0048]

[0049] Compression Matrix for:

[0050]

[0051] Summation Matrix for:

[0052]

[0053] Step 4) Construct the generator matrix G:

[0054] Through the identity matrix I of dimension K×K K , and the sub-Vandermonde matrix Sum and Add Matrix Construct a generator matrix G with dimension N×K;

[0055] The generator matrix G is the identity matrix I K , sub-Vandermonde matrix Sum and Add Matrix The first K rows, the K+1th to K+Yth rows, and the K+Y+1th to K+Y+Zth rows are stacked vertically. The expression of the generator matrix G and the calculation formula of the number of rows N are:

[0056]

[0057] N=K+Y+Z

[0058] Among them, g n,k is the element in the nth row and kth column of G;

[0059] In this embodiment, the generator matrix G is:

[0060]

[0061] Step 5) Obtain the construction result of the local recoverable code:

[0062] The codeword S is calculated by the original data T and the generator matrix G, and each The original data blocks, A global check block and a local check block are deployed to each cluster to obtain a local recoverable code;

[0063] The codeword S is calculated by the original data T and the generator matrix G, where the nth code element s in the codeword S is n It is through t k g n,k and t k g n,k After being written as an 8-bit binary number, multiplication is performed, and all multiplication results are XORed. The calculation formula for S is:

[0064]

[0065] Among them, ∑ represents sum operation, mod represents modulo operation;

[0066] The original data block deployed in the zth cluster is {t (z-1)×β+1 ,t (z-1)×β+2 ,...,t (z-1)×β+β}, the global check block is {u (z-1)×α+1 ,u (z-1)×α+2 ,...,u (z-1)×α+α}, the local check block is v z ;

[0067] Assume that cluster z deploys global checksum block u α(z-1)+a Lost, it is known that it is the ath global check block in the cluster. There are data blocks {t β(z-1)+1 ,t β(z-1)+2 ,...,t β(z-1)+β}, global check block {u α(z-1)+1 ,u α(z-1)+2 ,...,u α(z-1)+a-1 ,u α(z-1)+a+1 ,...u α(z-1)+α} and local check block v z According to the construction method of the generator matrix G and the construction results of the local recoverable code, we can know that:

[0068]

[0069] So restore u α(z-1)+a Just need to calculate:

[0070]

[0071] That is, you only need to read other nodes under cluster z, and do not need to read other nodes across clusters;

[0072] The codeword obtained in this example is:

[0073] {s1,s2,s3,s4,s5,s6,s7,s8,s9,s 10 ,s 11 ,s 12 ,s 13 ,s 14 ,s 15 ,s 16 ,s 17 ,s 18 ,s 19 ,s 20 ,s 21}

[0074] ={t1,t2,t3,t4,t5,t6,t7,t8,t9,t 10 ,t 11 ,t 12,u1,u2,u3,u4,u5,u6,v1,v2,v3},

[0075] The layout of the local recoverable code in the cluster is as follows Figure 2 As shown, for cluster 1, place {t1, t2, t3, t4, u1, u2, v1}, for cluster 2, place {t5, t6, t7, t8, u3, u4, v2}, and for cluster 3, place {t9, t 10 ,t 11 ,t 12 ,u5,u6,v3}, assuming that the original data block t1 in cluster 1 is lost, according to the calculation formula of S, we can get v1=t1+64t2+205t3+45t4+u1+u2, so t1=v1+64t2+205t3+45t4+u1+u2, and it can be restored by reading other nodes of cluster 1.

Claims

1. A method for constructing a local recoverable code, characterized in that: The steps include: (1) Initialization parameters: Initialize the local recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks and local check blocks to be deployed in each cluster is β, α and 1 respectively, where α≥1, Z≥1. First determine α and Z, calculate β=αZ-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y = αZ, the yth global check block is u y , No. z The local check block is v z , N=K+Y+Z=αZ 2 +Z, the nth code element is s n ,s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254 }modg(x), g(x) is any 8th-order irreducible polynomial; (2) Construct the Vandermonde matrix and divide it: Construct a Vandermonde matrix O of dimension (Y+1)×K, and divide O into a sub-Vandermonde matrix g of dimension Y×K and a row vector l including K elements; (3) Constructing the sum matrix The row vector l is expanded to obtain an expanded matrix L with a dimension of Z×K, and the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K (4) Construct the generator matrix G: Through the identity matrix I of dimension K×K K , as well as the sub-Vandermonde matrix g and the sum matrix Construct a generator matrix G with dimension N×K; (5) Obtain the construction result of the local recoverable code: The codeword S is calculated by the original data T and the generator matrix G, and each The original data blocks, A global check block and a local check block are deployed to each cluster to obtain a local recoverable code.

2. The method according to claim 1, characterized in that: The Vandermonde matrix O described in step (2), and the sub-Vandermonde matrix and row vector l, the expressions are: o y,k =2 (y-1)×(k-1) modg(x) Among them, y,k is the element at the yth row and kth column of O. mod represents the modular operation, that is, the element is written in the form of a polynomial and then the remainder is taken after the division operation.

3. The method according to claim 1, characterized in that The expansion matrix L and the contraction matrix L described in step (3) The expressions are: Among them, l z,k represents the element in the zth row and kth column of L, h z,k express The element at row z and column k, represents the rounding up operation, and ∑ represents the summation operation.

4. The method according to claim 3, characterized in that The sum matrix of dimension Z×K described in step (3) is constructed in The element in the zth row and kth column is obtained by replacing l z,k and h z,k They are written as 8-bit binary numbers and then XORed. The expression is:

5. The method according to claim 1, characterized in that The generator matrix G described in step (4) is the identity matrix I K , sub-Vandermonde matrix Sum and Add Matrix The first K rows, the K+1th to K+Yth rows, and the K+Y+1th to K+Y+Zth rows are stacked vertically. The expression of the generator matrix G and the calculation formula of the number of rows N are: N=K+Y+Z Among them, g n,k is the element in the nth row and kth column of G.

6. The method according to claim 1, characterized in that The codeword S is calculated by using the original data T and the generator matrix G in step (5), wherein the nth code element s in the codeword S is n It is through t k g n,k and t k g n,k After being written as an 8-bit binary number, multiplication is performed, and all multiplication results are XORed. The calculation formula for S is: Among them, ∑ represents the sum operation, and mod represents the modulo operation.

7. The method according to claim 1, characterized in that In step (5), each The original data blocks, A global check block and a local check block are deployed to each cluster, where the original data block deployed in the zth cluster is {t (z-1)×β+1 ,t (z-1)×β+2 ,...,t (z-1)×β+β }, the global check block is {u (z-1)×α+1 ,u (z-1)×α+2 ,...,u (z-1)×α+α }, the local check block is v z .

Images