A method for constructing locally recoverable codes

Abstract

The present invention proposes a method for constructing a locally recoverable code, which includes the following steps: initializing parameters; constructing and partitioning a Vandermonde matrix; constructing a summation matrix L; constructing a generator matrix G; and obtaining the construction result of the locally recoverable code. The present invention deploys a local check block, a global check block linearly associated with it, and an original data block in each cluster. This achieves the advantage of only needing to read other nodes in a cluster when recovering a node in the cluster, thus avoiding cross-cluster bandwidth. Furthermore, the generator matrix is ​​constructed by constructing a compression matrix and then a summation matrix, so that the generated local check block can be represented as a linear combination of the global check block and the original data block in the same cluster. This avoids the defect of needing to read all original data blocks when repairing the global check block, effectively improving data recovery efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of computer coding and relates to a construction method of a local recoverable code, which can be used in a distributed storage system. Background Art

[0002] A distributed storage system is a data management technology that divides data into multiple blocks and stores them on different nodes to achieve load balancing and redundancy. Existing distributed storage systems typically use a hierarchical architecture, organizing storage nodes into clusters.

[0003] Erasure Coding (EC) is a data protection technology used to improve data reliability and storage efficiency in distributed storage systems. It uses a mathematical algorithm to split data into multiple smaller blocks and generate a certain number of parity blocks. These parity blocks contain redundant information about the original data blocks. This allows the system to reconstruct the lost data from the remaining data blocks and parity blocks if some data blocks are lost. This significantly reduces storage space requirements while ensuring data integrity.

[0004] Locally Recoverable Codes (LRC) are a special type of erasure code used to improve data recovery efficiency in distributed storage systems. The technical approach to constructing LRCs is to first construct a generator matrix, which consists of three sub-matrices: the identity matrix, the generator matrix for the global parity block, and the generator matrix for the local parity block. The generator matrix is ​​then used to encode the original data to produce a codeword with local recovery properties.

[0005] Existing methods for constructing locally recoverable codes, such as the patent application with application publication number CN113258938A, entitled "A method for constructing an erasure code for rapid repair of a single-node fault," disclose a method for constructing a single-node fault rapid repair erasure code, comprising: encoding in the horizontal and vertical (longitudinal) directions, first encoding in the horizontal direction to generate a horizontal local checksum and a global checksum to ensure the fault tolerance of the encoding; performing XOR encoding in the vertical direction to obtain a checksum, which solves the problem of needing to read all node data when recovering the original data block and the high recovery cost. However, since the global checksum is not used when generating the local checksum, and the cluster is not taken into consideration when deploying the codeword, a technical problem of low recovery efficiency occurs during recovery. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a method for constructing a local recoverable code to solve the technical problem of low data recovery efficiency in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Initialization parameters:

[0009] Initialize the locally recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks, and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks, and local check blocks to be deployed in each cluster is β, α, and 1, respectively. Among them, α≥1, Z≥1. First determine α and Z, and calculate β=αz-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y=αZ, the yth global check block is u y , the zth local check block is v z , N=K+Y+Z=αZ 2 +Z, the nth code element is s n , s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254}mod g(x), g(x) is any irreducible polynomial of power 8;

[0010] (2) Construct the Vandermonde matrix and divide it:

[0011] Construct a Vandermonde matrix O of dimension (Y+1)×K and divide O into sub-Vandermonde matrices of dimension Y×K and a row vector l consisting of K elements;

[0012] (3) Constructing the sum matrix

[0013] Expand the row vector l to obtain the expanded matrix L with dimension Z×K, and at the same time pass the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K

[0014] (4) Construct the generator matrix G:

[0015] Through the identity matrix I of dimension K×K K , and the sub-Vandermonde matrix Sum-addition matrix Construct a generator matrix G with dimension N×K;

[0016] (5) Obtain the construction result of the local recoverable code:

[0017] The code word S is calculated by the original data T and the generator matrix G, and each Original data blocks, A global check block and a local check block are deployed to each cluster to obtain a locally recoverable code.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] (1) The present invention deploys a local check block and a global check block and original data block linearly related to it in each cluster, achieving the advantage of only needing to read other nodes in the cluster when recovering a node in a cluster, avoiding cross-cluster bandwidth and effectively improving data recovery efficiency.

[0020] (2) The present invention constructs a generation matrix by constructing a compression matrix and then reconstructing a summation matrix, so that the generated local check block can be expressed as a linear combination of the global check block and the original data block under the same cluster, avoiding the defect of needing to read all the original data blocks when repairing the global check block, and further improving the data recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an implementation flow chart of the present invention;

[0022] Figure 2 This is a layout diagram of locally recoverable codes in a cluster according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Reference Figure 1 , the present invention comprises the following steps:

[0025] Step 1) Initialize parameters:

[0026] Initialize the locally recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks, and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks, and local check blocks to be deployed in each cluster is β, α, and 1, respectively. Among them, α≥1, Z≥1. First determine α and Z, and calculate β=αz-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y=αZ, the yth global check block is u y , the zth local check block is v z, N=K+Y+Z=αZ 2 +Z, the nth code element is s n , s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254}mod g(x), g(x) is any irreducible polynomial of power 8;

[0027] In this embodiment, α=2, Z=3, and the original data block is T={t1, t2, t3, t4, t5, t6, t7, t8, t9, t 10 ,t 11 ,t 12}, Y = 6, β = 4, the locally recoverable code S = {s1,s2,s3,s4,s5,s6,s7,s8,s9,s 10 ,s 11 ,s 12 ,s 13 ,s 14 ,s 15 ,s 16 ,s 17 ,s 18 ,s 19 ,s 20 ,s 21};

[0028] Step 2) Construct the Vandermonde matrix and partition it:

[0029] Construct a Vandermonde matrix O of dimension (Y+1)×K and divide O into sub-Vandermonde matrices of dimension Y×K and a row vector l consisting of K elements;

[0030] Vandermonde matrix O, and sub-Vandermonde matrix and row vector l, the expressions are:

[0031]

[0032] l=[o Y+1,1 Lo Y+1,k Lo Y+1,K ]

[0033] o y,k =2 (y-1)×(k-1) modg(x)

[0034] Among them, y,k is the element at row y and column k of O, mod represents the modular operation, which is to write the element in the form of a polynomial and then perform the division operation and take the remainder;

[0035] In this embodiment, the constructed 7×12-dimensional Vandermonde matrix O is:

[0036]

[0037] The sub-Vandermonde matrix g and vector l are:

[0038]

[0039] l=[1 64 20545143 96 3618170 80185 97]

[0040] Step 3) Construct the sum matrix

[0041] Expand the row vector l to obtain the expanded matrix L with dimension Z×K, and at the same time pass the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K

[0042] Expanded matrix L and contracted matrix The expressions are:

[0043]

[0044] Among them, l z,k represents the element in the zth row and kth column of L, h z,k express The element at row z and column k, represents the rounding up operation, and ∑ represents the summation operation;

[0045] The element in row z and column k in is obtained by replacing l z,k and h z,k Written as 8-bit binary numbers and then XORed. The expression is:

[0046]

[0047] In this embodiment, the extended matrix L is:

[0048]

[0049] Compression Matrix for:

[0050]

[0051] Summation Matrix for:

[0052]

[0053] Step 4) Construct the generator matrix G:

[0054] Through the identity matrix I of dimension K×K K , and the sub-Vandermonde matrix Sum-addition matrix Construct a generator matrix G with dimension N×K;

[0055] The generator matrix G is the identity matrix I K , sub-Vandermonde matrix Sum-addition matrix The first K rows, the K+1th to K+Yth rows, and the K+Y+1th to K+Y+Zth rows are stacked vertically. The expression of the generator matrix G and the calculation formula of the number of rows N are respectively:

[0056]

[0057] N=K+Y+Z

[0058] Among them, g n,k is the element in the nth row and kth column of G;

[0059] In this embodiment, the generator matrix G is:

[0060]

[0061] Step 5) Obtain the construction result of the local recoverable code:

[0062] The code word S is calculated by the original data T and the generator matrix G, and each Original data blocks, A global check block and a local check block are deployed to each cluster to obtain a local recoverable code;

[0063] The codeword S is calculated by the original data T and the generator matrix G, where the nth code element s in the codeword S is n It is through t k g n,k and t k g n,k After writing it as an 8-bit binary number, multiplication is performed, and all multiplication results are XORed. The calculation formula of S is:

[0064]

[0065] Among them, ∑ represents sum operation, mod represents modular operation;

[0066] The original data block deployed in the zth cluster is {t (z-1)×β+1 ,t (z-1)×β+2 ,...,t (z-1)×β+β}, the global check block is {u (z-1)×α+1 ,u (z-1)×α+2 ,...,u (z-1)×α+α}, the local check block is v z ;

[0067] Assume that cluster z deploys global check block u α(z-1)+a Lost, it is known that it is the ath global check block in the cluster, and there are data blocks {t β(z-1)+1 ,t β(z-1)+2 ,...,t β(z-1)+β}, global check block {u α(z-1)+1 ,u α(z-1)+2 ,...,u α(z-1)+a-1 ,u α(z-1)+a+1 ,...u α(z-1)+α} and local check block v z According to the construction method of the generator matrix G and the construction results of the locally recoverable code, we can know that:

[0068]

[0069] Therefore, restore u α(z-1)+a Just need to calculate:

[0070]

[0071] That is, you only need to read other nodes under cluster z, and do not need to read other nodes across clusters;

[0072] The codeword obtained in this example is:

[0073] {s1,s2,s3,s4,s5,s6,s7,s8,s9,s 10 ,s 11 ,s 12 ,s 13 ,s 14 ,s 15 ,s 16 ,s 17 ,s 18 ,s 19 ,s 20 ,s 21}

[0074] ={t1,t2,t3,t4,t5,t6,t7,t8,t9,t 10 ,t 11 ,t 12,u1,u2,u3,u4,u5,u6,v1,v2,v3},

[0075] The layout of the local recoverable code in the cluster is as follows Figure 2 As shown, for cluster 1, place {t1, t2, t3, t4, u1, u2, v1}, for cluster 2, place {t5, t6, t7, t8, u3, u4, v2}, for cluster 3, place {t9, t 10 ,t 11 ,t 12 ,u5,u6,v3}, assuming that the original data block t1 in cluster 1 is lost, according to the calculation formula of S, we can get v1=t1+64t2+205t3+45t4+u1+u2, so t1=v1+64t2+205t3+45t4+u1+u2, and it can be recovered by only reading other nodes of cluster 1.

Claims

1. A method for constructing a locally recoverable code, characterized in that: The steps include: (1) Initialization parameters: Initialize the locally recoverable code to be constructed, including N code elements consisting of K original data blocks, Y global check blocks, and Z local check blocks. The distributed storage system includes Z clusters. The number of original data blocks, global check blocks, and local check blocks to be deployed in each cluster is β, α, and 1, respectively. Among them, α≥1, Z≥1. First determine α and Z, and calculate β=αZ-α, K=βZ=αZ 2 -αZ, the kth original data block is t k , Y=αZ, the yth global check block is u y , No. z The local check block is v z , N=K+Y+Z=αZ 2 +Z, the nth code element is s n , s n ∈GF(2 8 )={0,2 0 ,2 1 ,2 2 ,...,2 254 }modg(x), g(x) is any irreducible polynomial of power 8; (2) Construct the Vandermonde matrix and divide it: Construct a Vandermonde matrix O of dimension (Y+1)×K, and divide O into a sub-Vandermonde matrix g of dimension Y×K and a row vector l including K elements; (3) Constructing the sum matrix Expand the row vector l to obtain the expanded matrix L with dimension Z×K, and at the same time pass the sub-Vandermonde matrix Construct a compressed matrix of dimension Z×K And by expanding the matrix L and compressing the matrix Construct a sum matrix of dimension Z×K (4) Construct the generator matrix G: Through the identity matrix I of dimension K×K K , as well as the sub-Vandermonde matrix g and the sum matrix Construct a generator matrix G with dimension N×K; (5) Obtain the construction result of the local recoverable code: The code word S is calculated by the original data T and the generator matrix G, and each Original data blocks, A global check block and a local check block are deployed to each cluster to obtain a locally recoverable code.

2. The method according to claim 1, characterized in that The Vandermonde matrix O described in step (2), and the sub-Vandermonde matrix and row vector l, the expressions are: o y,k =2 (y-1)×(k-1) modg(x) Among them, y,k is the element at the yth row and kth column of O, mod represents the modular operation, that is, the element is written in the form of a polynomial and then the remainder is obtained after the division operation.

3. The method according to claim 1, characterized in that The expansion matrix L and the contraction matrix described in step (3) The expressions are: Among them, l z,k represents the element in the zth row and kth column of L, h z,k express The element at row z and column k, represents the rounding up operation, and ∑ represents the summation operation.

4. The method according to claim 3, characterized in that The sum matrix of dimension Z×K constructed in step (3) in The element in row z and column k in is obtained by replacing l z,k and h z,k Written as 8-bit binary numbers and then XORed. The expression is:

5. The method according to claim 1, characterized in that The generator matrix G described in step (4) is the identity matrix I K , sub-Vandermonde matrix Sum-addition matrix The first K rows, the K+1th to K+Yth rows, and the K+Y+1th to K+Y+Zth rows are stacked vertically. The expression of the generator matrix G and the calculation formula of the number of rows N are respectively: N=K+Y+Z Among them, g n,k is the element in the nth row and kth column of G.

6. The method according to claim 1, characterized in that The codeword S is calculated by the original data T and the generator matrix G in step (5), wherein the nth code element s in the codeword S is n It is through t k g n,k and t k g n,k After writing it as an 8-bit binary number, multiplication is performed, and all multiplication results are XORed. The calculation formula of S is: Among them, ∑ represents the sum operation, and mod represents the modular operation.

7. The method according to claim 1, characterized in that In step (5), each Original data blocks, A global check block and a local check block are deployed to each cluster, where the original data block deployed in the zth cluster is {t (z-1)×β+1 ,t (z-1)×β+2 ,...,t (z-1)×β+β }, the global check block is {u (z-1)×α+1 ,u (z-1)×α+2 ,...,u (z-1)×α+α }, the local check block is v z .

Images