Decentralized data redundancy storage method and system based on erasure code

Abstract

The application discloses a kind of decentralized data redundancy storage method and system based on erasure code, and the specific steps of method are as follows: S1, file data is redundantly encoded;S2, load balancing of storage data is realized using chip interleaving storage technology;S3, data deinterleaving and data restoration;S4, decoding and repairing data.The application uses RS erasure code encoding and decoding and data interleaving technology, cooperates with cluster virtualization technology to realize safe, reliable, efficient data redundancy storage and repair scheme, greatly reduces the storage space occupation and the bandwidth required for repair.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed storage, and specifically relates to an erasure code-based redundant storage method for ubiquitous storage systems and a system thereof. Background Art

[0002] In recent years, with the rapid popularization and development of Internet applications, data and information have shown explosive growth. The sharp expansion of data, along with the increasing requirements for the security, reliability and privacy of data storage, has continuously exposed the problems and drawbacks of traditional storage systems. Taking the traditional centralized storage scheme represented by data centers as an example, it relies on a large number of storage servers to provide data storage services, which results in that the security and reliability of user data storage are highly correlated with the stability of the servers themselves and the anti-destruction capability of the data center. A single point of failure in the storage system will cause a large amount of data to be unavailable or even lost; meanwhile, with the development and commercialization of big data analysis and artificial intelligence, user personal data contains huge commercial value, and a large amount of data is stored on "oligarch" nodes, which cannot guarantee that user personal data will not be illegally stolen and used at all.

[0003] In order to solve the above problems, the current academic community and technology industry have proposed a variety of decentralized storage systems and solutions. Representative and successfully implemented solutions include Storj, MaidSafe, IPFS, etc. These decentralized storage systems are all committed to solving the problems of traditional centralized storage systems, such as single point of failure, data security and privacy, etc., and provide safer, more reliable and flexible data storage solutions through decentralized storage and encryption technologies. However, storage nodes in the decentralized storage mode are scattered in different regions. Compared with storage servers in data centers, decentralized storage nodes are more difficult to monitor and maintain, which further affects the stability of storage nodes. Therefore, in order to store data permanently, safely and reliably, appropriate data fault-tolerant technology is required to guarantee the security and availability of data storage.

[0004] In practical applications, most storage systems adopt the multi-copy backup method to guarantee the security and availability of data. However, this method often consumes a large amount of additional storage space, and is difficult to apply to some resource-constrained systems (for example, ubiquitous storage, edge computing, etc.). At present, another mainstream fault-tolerant technology meets the requirements of high data persistence and low redundancy through erasure codes. For example, the most commonly used RS (Reed-Solomon) erasure code algorithm divides the original data into k fragments through (n,k) erasure coding, and then encodes to obtain n code fragments (k<n) to guarantee data storage reliability. Compared with the traditional three-copy backup scheme, its storage overhead can be reduced by 53%, and the data storage reliability can be doubled.

[0005] Chinese patent application number 202210936132.X discloses a file fragmentation backup method based on IPFS: first, the source file is encoded; then, the encoded file is segmented; and finally, the segmented data blocks are distributed and stored on different storage nodes. This method aims to solve the problem of data information being spied on. While this method can improve data security and privacy, for decentralized storage systems with poor stability, data recovery in the event of data loss will incur additional communication and computational overhead, and the recovery time will also increase, making it difficult to practically apply to decentralized storage systems. Summary of the Invention

[0006] Based on the above-mentioned status quo in the field, the present invention aims to achieve reliable data storage and efficient repair in the event of unstable nodes within a storage network. It provides a data redundancy scheme in decentralized storage, which adopts RS erasure coding and decoding and data interleaving technology, combined with cluster virtualization technology to achieve a secure, reliable and efficient data redundancy storage and repair scheme. Compared with existing conventional multi-copy backup and erasure coding schemes, it greatly reduces storage space occupation and repair bandwidth requirements, while improving repair efficiency and data security.

[0007] The present invention adopts the following technical solution

[0008] The decentralized data redundancy storage method based on erasure coding has the following specific steps:

[0009] S1. Redundant encoding of file data;

[0010] S2. Utilize chip interleaving storage technology to achieve load balancing of stored data;

[0011] S3. Deinterleaving and restoring the data;

[0012] S4. Decode and repair the data.

[0013] This invention incorporates encoding / decoding and data interleaving techniques to achieve redundant storage of file data blocks, ultimately transforming data failures caused by sudden node anomalies into random failures.

[0014] The preferred embodiments of the specific steps of the present invention are described in detail below.

[0015] Preferably, S1. File data redundancy encoding

[0016] This invention extends the parameters based on RS erasure coding, and the parameters are as follows: The RS code, where:

[0017] parameter For storing the newly added threshold, When the number of available chips is below a threshold In such cases, data repair needs to be performed immediately. This method gives storage nodes time to recover and reduces unnecessary repair costs.

[0018] parameter This indicates the number of information chips obtained after segmenting the original data. (Parameter) This indicates the number of coded segments in the erasure coding stripe, which can tolerate any One chip failed;

[0019] parameter This indicates the number of nodes that can be accessed during data repair, i.e., the number of nodes that need to be repaired. Each node downloads data. .when This allows for optimal repair bandwidth overhead.

[0020] The encoding process is detailed as follows:

[0021] (1) In order to improve the computation efficiency of RS erasure codes, a Cauchy matrix is ​​constructed. Instead of the Vandermonde matrix, it reduces the computational complexity during decoding; where k is the number of data chips and r is the number of redundant chips. k+r together constitute the encoded chip set;

[0022] (2) By transforming the finite field, The elements in the field are converted into binary matrices, which, compared to the method described in Vandermonde RS erasure coding, eliminates the need to construct a logarithm / anti-logarithm table, making... It can support a large amount of Domain space computation is performed, effectively reducing computational complexity. The encoding matrix can be represented as:

[0023]

[0024] Where D represents the chunk of the original data. and All are the domains of Galois The elements in , Data chip and redundant chips It is a collection of one-dimensional arrays. Within Galois's finite field, any All elements on can be mapped to A binary field, and the original is identified using a binary matrix. The elements in. Therefore, in Generating matrix in After conversion, it becomes In the domain Binary matrix. Through field transformation, The multiplication operation in the program is programmed with logical AND. The addition operation has been changed to XOR. This greatly reduces computational complexity.

[0025] Preferably, S2. Chip interleaving storage technology

[0026] To address the unavoidable problem of sudden data failures in storage nodes due to network outages, equipment malfunctions, and other reasons, this invention proposes a chip load balancing scheme based on data interleaving technology. This scheme aims to achieve load balancing of stored data across the entire storage system and, simultaneously, transform sudden data failures caused by node failures into random failures.

[0027] To achieve the above objectives, an interleaving matrix based on pseudo-random numbers is used for random sorting of data blocks and sequence reconstruction. The specific implementation method is as follows:

[0028] (1) For a length of chip sequence Generate a pseudo-random number array of equal length, and rearrange the positions of the original code chips based on the pseudo-random number array to obtain a new code chip sequence. ;

[0029] (2) Constructing the interleaving matrix The rearranged chip sequence stream Write line by line The matrix buffer stores different data sequences that are written sequentially. When a write cycle ends, any unfilled portions of a row are padded with null bytes. The matrix depth is determined by the size of the data to be stored per unit time. The matrix width represents the number of available storage nodes in the current storage cycle. The design should not exceed the total number of storage nodes. This can be represented as:

[0030]

[0031] It should be noted that, in order to ensure that the same chip sequence is not stored in the same storage node, the matrix width is... This means that the number of available storage nodes per unit time is much greater than the number of data blocks after encoding; 'l' represents the number of data blocks processed (to be stored) per unit time. Meanwhile, in order to fully utilize storage node resources, It should be as close as possible to the maximum number of available storage nodes.

[0032] (3) After the writing cycle is completed, the output sequence is obtained by reading column by column. , The output sequence will then be displayed. They are stored sequentially into different storage nodes.

[0033] via matrix Process the data so that the number of output sequences equals the matrix width. ,and It can automatically adjust according to the node load, reducing the number of selections for heavily loaded nodes. Therefore, the data chips to be stored can ultimately be stored on all nodes with a balanced load. As the number of storage nodes in the storage system dynamically expands or shrinks, each storage procedure is a data load rebalancing process, effectively avoiding data skew.

[0034] Preferably, S3. Deinterleaving and data restoration

[0035] The deinterleaving process is achieved by constructing transpose matrix According to the sequence storage order and chip index identifier, the chip data stream is obtained from the corresponding storage node and written into the deinterleaving matrix. Among them, as shown below:

[0036]

[0037] Read the chip sequence column by column from left to right. Based on the pseudo-random number array ( Restore chip sequence The specific restoration method is as follows: The number at position 0 corresponds to the number 0 in the original chip sequence. Position, number The number at position 1 corresponds to the number in the original chip sequence. Number card in The location. Therefore, it is only necessary to... Chip data at each location moved to The original chip sequence can be obtained by finding the position of the number that matches the index number. .

[0038] Based on cluster virtualization technology, a unified interaction interface is provided to the external storage cluster to receive data. Subsequently, the cluster preprocesses the files (encoding, interleaving), and the interleaved data is stored in different storage nodes according to the sequence output by the interleaving matrix.

[0039] Preferably, S4. Data Decoding and Repair

[0040] Data deinterleaving and correct restoration of chip sequences Next, data decoding (repair) is performed. Since any submatrix of the Cauchy matrix is ​​a singular matrix, it has an inverse matrix. Therefore, the decoding process can be described as follows:

[0041]

[0042] in, .

[0043] For file data, when any data chip When a chip is lost or damaged, the repair system will determine whether the number of missing chips has reached the repair threshold. If the condition is met, the repair process is initiated; otherwise, the missing chip identifier information is recorded and the number of missing chips in the file is updated. The data repair process can be divided into the following steps:

[0044] (1) Obtain the missing chip identifier And the remaining valid chip data, and construct a decoding column vector based on the remaining chip data. ;

[0045] (2) By generating a matrix Decode the rows corresponding to the column vectors to obtain a square matrix. This leads to the decoding matrix ( (the inverse matrix) ;

[0046] (3) By decoding the matrix With the decoding vector Multiply to finally restore the data information. .

[0047] This invention also discloses a decentralized data redundancy storage system based on erasure coding. The technical solution based on the above method includes the following modules:

[0048] Encoding module: Performs redundant encoding on file data;

[0049] Load balancing module: Utilizes chip interleaving storage technology to achieve load balancing of stored data;

[0050] Deinterleaving and Restoration Module: Deinterleaves and restores data;

[0051] Decoding and Repair Module: Decodes and repairs data.

[0052] This invention provides a data redundancy storage scheme in decentralized storage. It adopts RS erasure coding and data interleaving technology, combined with cluster virtualization technology, to achieve a secure, reliable and efficient data redundancy storage and repair scheme, which greatly reduces storage space occupation and the bandwidth required for repair. Attached Figure Description

[0053] Figure 1 This is a flowchart of a preferred embodiment of the present invention for a decentralized data redundancy storage method based on erasure coding.

[0054] Figure 2 This invention is based on an interleaving diagram of a chip sequence of length 12.

[0055] Figure 3 This invention is a deinterleaving diagram based on a chip sequence of length 12.

[0056] Figure 4 This is a block diagram of a decentralized data redundancy storage system based on erasure coding, according to a preferred embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings.

[0058] A preferred embodiment of the present invention provides a decentralized data redundancy storage method based on erasure coding, comprising the following steps:

[0059] S1, Redundant encoding of file data;

[0060] S2 utilizes chip interleaving storage technology to achieve load balancing of stored data throughout the entire storage system;

[0061] S3, Deinterweaving and Data Restoration;

[0062] S4, Data Decoding and Repair.

[0063] The overall steps and flow of this embodiment are as follows: Figure 1 As shown, large files or raw data are erasure encoded into fragments. Chip interleaving storage technology is then used to randomly sort the data blocks and reconstruct the sequence. These processed chips are sent to nodes in the ubiquitous storage system for storage. When the original data is needed, deinterleaving and data restoration are performed first, and finally, the data is decoded and repaired to obtain the original data. The details of each step are as follows.

[0064] In this embodiment, step S1 is as follows:

[0065] This invention is based on RS erasure coding, adding a new parameter m for threshold storage, forming an RS code with parameters [n, k, d, m]. Here, k represents the number of information chips after the original data is segmented, n represents the number of coded chips in the erasure code stripe, and d represents the number of nodes accessible during data repair; that is, the repair node needs to connect to d nodes to download data. The newly added parameter m indicates that when the number of available chips is lower than the threshold m, data repair needs to be performed immediately, thus providing storage nodes with recovery tolerance time and avoiding unnecessary repair overhead.

[0066] The extended RS code can tolerate the failure of any nk chips, thus exhibiting strong error correction capabilities. Furthermore, when d=n-1, it achieves optimal repair bandwidth overhead.

[0067] The encoding process can be further refined as follows:

[0068] (1) In order to improve the computation efficiency of RS erasure codes, a Cauchy matrix is ​​constructed. Instead of the Vandermonde matrix, it reduces the computational complexity during decoding. Here, k is the number of data chips, r is the number of redundant chips, and k+r together constitute the set of encoded chips.

[0069] (2) By transforming the finite field, The elements in the field are converted into binary matrices. Compared to the method described in Vandermonde RS erasure coding, this method does not require the construction of a logarithm / anti-logarithm table, making... It can support a large amount of Domain space computation is performed, effectively reducing computational complexity. The encoding matrix can be represented as:

[0070]

[0071] Where D represents the chunk of the original data. and All are the domains of Galois The elements in , Data chip and redundant chips It is a collection of one-dimensional arrays. Within Galois's finite field, any All elements on can be mapped to A binary field, and the original is identified using a binary matrix. The elements in. Therefore, in Generating matrix in After conversion, it becomes In the domain Binary matrix. Through field transformation, The multiplication operation in the program is programmed with logical AND. The addition operation has been changed to XOR. This greatly reduces computational complexity.

[0072] like Figure 2 As shown, in this embodiment, step S2 is specifically as follows:

[0073] S21, firstly, an interleaving matrix based on pseudo-random numbers is used to randomly sort the data blocks. For a chip sequence of length 12... Generate a pseudo-random number array of equal length. , the original chip Based on pseudo-random number array Rearrange the positions to obtain a new chip sequence. ;

[0074] S22, Construct the interleaving matrix The chip sequence rearranged based on a pseudo-random array Write In the matrix buffer, when a write cycle ends, any unfilled portions of a row are padded with null bytes. The matrix width should be no greater than the number of storage nodes available per unit time. In this example, we take... It equals 12; The matrix depth is determined by the amount of data to be stored per unit time. It can be represented as:

[0075]

[0076] It should be noted that, in order to ensure that the same chip sequence is not stored in the same storage node, the matrix width is... This means that the number of available storage nodes per unit time is much greater than the number of data blocks after encoding; l represents the number of data blocks processed per unit time. Meanwhile, in order to fully utilize storage node resources, It should be as close as possible to the maximum number of available storage nodes.

[0077] S23, After the write cycle ends, read the output sequence column by column to obtain the output sequence. , The output sequence will then be... They are stored sequentially into different storage nodes.

[0078] Matrix T is used to process the data chips to be stored, and the data is stored by outputting a sequence number equal to the matrix width W. W can be adjusted automatically according to the node load, thereby reducing the number of selections for heavily loaded nodes. In this way, the data chips to be stored are ultimately stored evenly across all nodes. In the storage system, if the number of storage nodes dynamically expands or shrinks, a data load rebalancing operation is performed with each storage process, effectively preventing data skew. By dynamically adjusting the node load and redistributing data, the system can achieve data load balancing, ensuring a relatively balanced load on each node and improving system performance and efficiency.

[0079] In summary, by processing data through matrix T and dynamically adjusting node load, the storage system can achieve data load balancing, avoid data skew, and improve system reliability and performance.

[0080] like Figure 3 As shown, in this embodiment, step S3 specifically involves the following process:

[0081] S31, the deinterlacing process is achieved by constructing... transpose matrix According to the sequence storage order and chip index identifier, the chip data stream is obtained from the corresponding storage node and written into the deinterleaving matrix. Among them, as shown below:

[0082]

[0083] Read the chip sequence column by column from left to right. Based on the pseudo-random number array ( Restore chip sequence The specific restoration method is as follows: The number at position 0 corresponds to the number 0 in the original chip sequence. Position, number The number at position 1 corresponds to the number in the original chip sequence. Number card in The location. Therefore, it is only necessary to... Chip data at each location moved to The original chip sequence can be obtained by finding the position of the number that matches the index number. .

[0084] S32, based on cluster virtualization technology, provides a unified interaction interface to the external storage cluster and is responsible for receiving data. Subsequently, the cluster preprocesses the files (encoding, interleaving), and the interleaved data is stored in different storage nodes according to the sequence output by the interleaving matrix.

[0085] The application of interleaving technology ensures that each node stores only a portion of the original data; the failure of any node only affects that portion, effectively mitigating the impact of sudden failures on the storage system. Furthermore, interleaving technology provides a degree of fault tolerance, enabling data recovery from the fragments of other nodes even when multiple nodes fail. This enhances system reliability and fault tolerance, guaranteeing data security and integrity.

[0086] In this embodiment, step S4 includes the following steps:

[0087] S41, after data deinterleaving and correctly restoring the chip sequence X, data decoding and repair are performed. Since any submatrix of the Cauchy matrix is ​​a singular matrix, it has an inverse matrix. Therefore, the decoding process can be described as follows:

[0088]

[0089] ;

[0090] S42, for file data, when any data chip When a chip is lost or damaged, the repair system will determine whether the number of missing chips has reached the repair threshold. If the condition is met, the repair process is initiated; otherwise, the missing chip identifier information is recorded and the number of missing chips in the file is updated. The data repair process can be divided into the following steps:

[0091] (1) Obtain the missing chip identifier And the remaining valid chip data, and construct a decoding column vector based on the remaining chip data. ;

[0092] (2) By generating a matrix Decode the rows corresponding to the column vectors to obtain a square matrix. This leads to the decoding matrix ( (the inverse matrix) ;

[0093] (3) By decoding the matrix With the decoding vector Multiply to finally restore the data information. .

[0094] The advantage of Cauchy codes lies in the fact that the original data can be reconstructed from any k chips, and they can resist chip loss and corruption to a certain extent. Therefore, the repair process only starts after the repair threshold m is reached, avoiding unnecessary repair operations that would lead to performance loss. Meanwhile, in practical applications, issues related to data transmission, storage, and verification also need to be considered to ensure data integrity and reliability.

[0095] like Figure 4As shown, this embodiment discloses a decentralized data redundancy storage system based on erasure coding. Based on the above method embodiment, it includes the following modules:

[0096] Encoding module: Performs redundant encoding on file data;

[0097] Load balancing module: Utilizes chip interleaving storage technology to achieve load balancing of stored data;

[0098] Deinterleaving and Restoration Module: Deinterleaves and restores data;

[0099] Decoding and Repair Module: Decodes and repairs data.

[0100] Other aspects of this embodiment can be found in the embodiments described above.

[0101] In summary, this invention employs RS erasure coding and data interleaving technology, combined with cluster virtualization technology, to achieve a secure, reliable, and efficient data redundancy storage and repair solution. This greatly reduces storage space usage and the bandwidth required for repair, providing a data redundancy solution in decentralized storage.

[0102] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A decentralized data redundancy storage method based on erasure coding, characterized by: The specific steps are as follows: S1. Redundant encoding of file data; S2. Utilize chip interleaving storage technology to achieve load balancing of stored data; S3. Deinterleaving and restoring the data; S4. Decode and repair the data; In step S1, the parameters are expanded based on the RS erasure code to form the following parameters: The RS code, where: parameter For storing the newly added threshold, When the number of available chips is below the threshold If necessary, data repair will be performed; parameter This represents the number of information chips obtained after segmenting the original data. (Parameter) This represents the number of coded segments in an erasure coding stripe, which can tolerate arbitrary... One chip failed; parameter This indicates the number of nodes that can be accessed during data repair, i.e., the number of nodes that need to be repaired. Each node downloads data; ;when When this is achieved, the optimal repair bandwidth cost can be reached; The encoding process is as follows: (1) Constructing the Cauchy matrix Where r is the number of redundant chips, and k+r together constitute the set of encoded chips; (2) By transforming the finite field, The elements in the field are converted into a binary matrix, and the encoded matrix is ​​represented as follows: Where D represents the chunk of the original data. and All are the domains of Galois The elements in , Data chip and redundant chips It is a collection of one-dimensional arrays. Within Galois's finite field, any All elements on can be mapped to A binary field, and the original is identified using a binary matrix. Elements in; Step S2 is as follows: (1) For a length of chip sequence Generate a pseudo-random number array of equal length, and rearrange the positions of the original code chips based on the pseudo-random number array to obtain a new code chip sequence. ; (2) Constructing the interleaving matrix The rearranged chip sequence stream Write line by line The matrix buffer area allows different data sequences to be written sequentially. At the end of a write cycle, any unfilled portions of a row are padded with null bytes. The matrix depth is determined by the size of the data to be stored per unit time. The matrix width represents the number of available storage nodes within the current storage cycle. Not greater than the total number of storage nodes; matrix width This means that the number of available storage nodes per unit time is much greater than the number of data blocks after encoding; l represents the number of data blocks processed per unit time. (3) After the writing cycle is completed, the output sequence is obtained by reading column by column. , Then the output sequence They are stored sequentially into different storage nodes; In step S3, the deinterlacing process is achieved by constructing... transpose matrix According to the sequence storage order and chip index identifier, the chip data stream is obtained from the corresponding storage node and written into the deinterleaving matrix. Among them, as shown below: Read the chip sequence column by column from left to right. Based on the pseudo-random number array Restore chip sequence The specific restoration method is as follows: The number at position 0 corresponds to the number 0 in the original chip sequence. Position, number The number at position 1 corresponds to the number in the original chip sequence. Number card in Location; will Chip data at each location moved to The original chip sequence can be obtained by finding the position of the number that matches the index number. .

2. The decentralized data redundancy storage method based on erasure coding as described in claim 1, characterized in that, In step S4, after the data is deinterleaved and the chip sequence X is correctly restored, data decoding is performed. The decoding process is described as follows: Where D' is the decoded column vector, and the position index is consistent with the encoded chip; For file data, when any data chip When a chip is lost or damaged, determine whether the number of missing chips has reached the repair threshold. If the condition is met, the repair process is initiated; otherwise, the missing chip identifier information is recorded and the number of missing chips in the file is updated.

3. The decentralized data redundancy storage method based on erasure coding as described in claim 2, characterized in that, In step S4, the data repair process is as follows: (1) Obtain the missing chip identifier And the remaining valid chip data, and construct a decoding column vector based on the remaining chip data. ; (2) By generating a matrix Decode the rows corresponding to the column vectors to obtain a square matrix. This leads to the decoding matrix. ; (3) By decoding the matrix With the decoding vector Multiply to restore the data information .

4. A decentralized data redundancy storage system based on erasure coding, wherein the method described in any one of claims 1-3 is characterized in that, Includes the following modules: Encoding module: Performs redundant encoding on file data; Load balancing module: Utilizes chip interleaving storage technology to achieve load balancing of stored data; Deinterleaving and Restoration Module: Deinterleaves and restores data; Decoding and Repair Module: Decodes and repairs data.

Images