Cloud native high-availability object storage method based on erasure codes

Abstract

The invention particularly relates to a cloud native high-availability object storage method based on erasure codes. The cloud native high-availability object storage method based on erasure codes comprises the following steps: dividing original data into k data blocks, and generating m-k verification blocks; constructing an m * k matrix by using a Vandermonde matrix, wherein the m * k matrix is used for coding erasure codes; elements in the matrix are Byte values in a GF (28) finite field; the data blocks and the verification blocks are stored in a distributed mode, and when data needs to be read, the data blocks and the verification blocks are calculated through an inverse matrix of the Vandermonde matrix; when the number of the storage nodes is increased or decreased, the coefficient matrix is recalculated, and redistribution storage and dynamic capacity expansion are achieved. According to the cloud native high-availability object storage method based on the erasure codes, storage dynamic capacity expansion and data integrity protection are achieved, lost and damaged data can be recovered, the high availability of storage services is improved, the stability and continuity of core services are guaranteed, and the safety and high availability of a cloud native platform are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and particularly relates to a cloud-native highly available object storage method based on erasure codes. Background Art

[0002] During long-term operation, due to improper user operations and other abnormal reasons, node failures or disk damages may occur in the cloud-native environment. Many important services in the environment rely on the storage service. If there are deficiencies in the high availability of the storage service, problems such as abnormal service data recovery and data loss may occur, resulting in key services being unavailable or operating abnormally. Therefore, the cloud-native platform has put forward higher requirements for environmental high availability. As a key component of the cloud platform, the storage component provides storage services for various application services, and its high availability is very important.

[0003] Based on the above situation, the present invention proposes a cloud-native highly available object storage method based on erasure codes. Summary of the Invention

[0004] In order to make up for the defects of the prior art, the present invention provides a simple and efficient cloud-native highly available object storage method based on erasure codes.

[0005] The present invention is realized through the following technical solutions:

[0006] A cloud-native highly available object storage method based on erasure codes, characterized in that: an erasure code mechanism is used to ensure high availability, and the HighWayHash function is used to handle data corruption; it includes the following steps:

[0007] Step S1, parity separation

[0008] In an Erasure Code group, the original data is divided into k data blocks, and then m - k parity blocks are generated, where m is greater than k, to provide data integrity protection. This separation method is called parity separation;

[0009] Step S2, matrix modeling

[0010] A Vandermonde matrix is used to construct an m*k matrix for erasure code encoding; the elements in the matrix are Byte values in the finite field GF(2^8);

[0011] Step S3, storage distribution

[0012] Each data block and parity block are respectively stored on different nodes in the Erasure Code group to improve data reliability and ensure that the dispersed storage of data does not cause a single point of failure;

[0013] Step S4, Encoding Calculation

[0014] When reading data, use the inverse matrix of the Vandermonde matrix to calculate the necessary data blocks and parity blocks;

[0015] Step S5, Dynamic Scaling

[0016] Due to the advantages of Erasure Code, when the number of storage nodes increases or decreases, recalculate the coefficient matrix without having to replicate the entire dataset, enabling redistribution of storage and dynamic scaling.

[0017] In the said Step S2, use erasure code and checksum to protect data from hardware failures and silent data corruption; data can be recovered as long as the lost data is less than half of the total data; when the lost or corrupted data on a disk reaches half or more of the total data, the data cannot be recovered.

[0018] In the said Step S2, encode the original data through an erasure code algorithm to obtain redundant data, and store both the original data and the redundant data together for the purpose of fault tolerance; specifically as follows:

[0019] Separate n blocks of original data elements through parity separation calculation to obtain q blocks of redundant elements (parity blocks), totaling n + q blocks of elements; when any v blocks of elements in the original data and redundant data are in error, where v is less than or equal to q, combine the non - errored parity blocks and recover the original data through the HighWayHash function to obtain n blocks of original data elements.

[0020] In the said Step S3, use a distributed storage mode to automatically introduce erasure code, and adopt erasure code to guard against multiple node outages and bit rot.

[0021] In the said Step S3, the distributed storage mode requires at least 4 disks.

[0022] In the said Step S4, the encoding calculation is performed on any Erasure Code group nodes and supports simultaneous calculation on different Erasure Code group nodes; combine the calculated data blocks to generate the original data.

[0023] A cloud - native highly available object storage system based on erasure code, comprising:

[0024] A parity separation module, responsible for dividing the original data in an Erasure Code group into k data blocks, and then generating m - k parity blocks, where m is greater than k, to provide data integrity protection;

[0025] The matrix modeling module is responsible for constructing an m*k matrix using the Vandermonde matrix for erasure code encoding; the elements in the matrix are Byte values in the finite field GF(2^8).

[0026] The distributed storage module is responsible for storing each data block and parity block on different nodes in the Erasure Code group respectively to improve data reliability and ensure that the dispersed storage of data does not cause a single point of failure.

[0027] The encoding calculation module is responsible for calculating the necessary data blocks and parity blocks using the inverse matrix of the Vandermonde matrix when reading data.

[0028] The dynamic scaling module is responsible for taking advantage of Erasure Code to recalculate the coefficient matrix when the number of storage nodes increases or decreases, without having to replicate the entire dataset again, and realizing redistributed storage and dynamic expansion.

[0029] A cloud-native highly available object storage device based on erasure code, characterized in that: it includes a memory and a processor; the memory is used to store computer programs, and the processor is used to implement the above method steps when executing the computer programs.

[0030] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0031] The beneficial effects of the present invention are: the cloud-native highly available object storage method based on erasure code realizes dynamic storage expansion and data integrity protection, can recover lost and damaged data, improves the high availability performance of storage services, ensures the stability and continuity of core services, and improves the security and high availability capabilities of the cloud-native platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Attached Figure 1 is a schematic diagram of the cloud-native highly available object storage method based on erasure code of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0035] Erasure Code is a mathematical algorithm for recovering lost and damaged data. Currently, the application of erasure code technology in distributed storage systems mainly has three categories: array erasure codes (Array Code: RAID5, RAID6, etc.), Reed-Solomon (RS) erasure codes, and Low-Density Parity Check Code (LDPC) erasure codes. Erasure Code is a coding technology that can add m pieces of data to n pieces of original data and can restore the original data through any n pieces of the n + m pieces of data. That is, if any data less than or equal to m pieces fails, the remaining data can still be used to restore it.

[0036] The working principle of erasure code is different from that of RAID or replication. For example, RAID6 can not lose data when two disks are lost, while erasure code can still ensure data security when half of the disks are lost.

[0037] Moreover, erasure code acts at the object level and can restore one object at a time, while RAID acts at the volume level and the data recovery time is very long. Each object is encoded separately, and usually, there is no need to replace the hard disk or repair it once the storage service is deployed. The design goal of using erasure code in object storage is for performance and to use hardware acceleration as much as possible.

[0038] This cloud-native highly available object storage method based on erasure code uses the erasure code mechanism to ensure high availability and uses the HighWayHash function to handle data corruption; it includes the following steps:

[0039] Step S1, parity separation

[0040] In an Erasure Code group, the original data is divided into k data blocks, and then m - k check blocks are generated, where m is greater than k, to provide data integrity protection. This separation method is called parity separation;

[0041] Step S2, matrix modeling

[0042] Erasure code is a general-purpose Vandermonde matrix encoding method. This encoding method encodes by generating a matrix and applying the matrix to each data block; a Vandermonde matrix is used to construct an m*k matrix for erasure code encoding; the elements in the matrix are Byte values in the finite field GF(2^8).

[0043] Step S3, storage distribution

[0044] Each data block and parity block are stored on different nodes in the Erasure Code group respectively to improve data reliability and ensure that the dispersed storage of data does not cause a single point of failure.

[0045] Step S4, encoding calculation

[0046] When reading data, the inverse matrix of the Vandermonde matrix is used to calculate the necessary data blocks and parity blocks.

[0047] Step S5, dynamic scaling

[0048] Due to the advantages of Erasure Code, when the number of storage nodes increases or decreases, the coefficient matrix is recalculated without having to replicate the entire dataset again, realizing re-distributed storage and dynamic expansion.

[0049] In step S2, erasure code and checksum are used to protect data from hardware failures and silent data corruption; as long as the lost data is less than half of the total data, the data can be recovered; when the lost or damaged data on a disk reaches half or more of the total data, the data cannot be recovered.

[0050] By using the Vandermonde matrix for erasure code encoding, not only is data integrity protection and data recovery achieved, but also the occupation of storage space and the consumption of network bandwidth are reduced, realizing a highly reliable, high-performance and highly secure storage service, which is very suitable for use in cloud-native scenarios.

[0051] Through the calculation method of the check code, the lost and damaged data is verified and restored. On the one hand, through the calculation of the checksum of the data, the integrity of the data can be checked to check whether the data is damaged or changed, which is used in data transmission. On the other hand, by combining the calculation of the check code with the data, the lost and damaged data is restored, and redundancy can be reduced on the premise of ensuring reliability.

[0052] In step S2, the original data is encoded through the erasure code algorithm to obtain redundant data, and the original data and redundant data are stored together to achieve the purpose of fault tolerance; specifically as follows:

[0053] n original data elements are calculated by parity separation to obtain q redundant elements (check blocks), for a total of n + q elements; when any v elements in the original data and redundant data are in error, where v is less than or equal to q, the original data is recovered through the HighWayHash function in combination with the non - errored check blocks to obtain n original data elements.

[0054] Bit rot, also known as Data Rot or Silent Data Corruption, is a serious data loss problem for current hard - disk data. When data on a hard disk is corrupted, no error log may be generated.

[0055] The single - machine storage mode has a single - point failure. For a distributed storage with G hard disks, as long as G / 2 hard disks are available, the data is safely readable; at least G / 2 + 1 hard disks are available, the data is safely writable.

[0056] In step S3, the distributed storage mode automatically introduces erasure codes to prevent multiple node outages and bit rot.

[0057] In both distributed and single - machine modes, all write operations strictly adhere to read - after - write consistency.

[0058] In step S3, the distributed storage mode requires at least 4 disks.

[0059] In step S4, the encoding calculation is performed on any Erasure Code group node and supports simultaneous calculation on different Erasure Code group nodes; the calculated data blocks are combined to generate the original data.

[0060] The cloud - native highly - available object storage system based on erasure codes includes:

[0061] A parity separation module, responsible for dividing the original data in an Erasure Code group into k data blocks and then generating m - k check blocks, where m is greater than k, to provide data integrity protection;

[0062] A matrix modeling module, responsible for constructing an m*k matrix using a Vandermonde matrix for erasure - code encoding; the elements in the matrix are Byte values in the finite field GF(2^8);

[0063] A distributed storage module, responsible for storing each data block and check block on different nodes in the Erasure Code group respectively to improve data reliability and ensure that the dispersed storage of data does not cause a single - point failure;

[0064] An encoding calculation module, which is responsible for calculating the necessary data blocks and parity blocks using the inverse matrix of the Vandermonde matrix when reading data;

[0065] A dynamic scaling module, which is responsible for taking advantage of Erasure Code to recalculate the coefficient matrix when the number of storage nodes increases or decreases, without having to replicate the entire data set again, so as to achieve redistributed storage and dynamic expansion.

[0066] The cloud-native highly available object storage device based on erasure code includes a memory and a processor; the memory is used to store computer programs, and the processor is used to implement the above method steps when executing the computer programs.

[0067] A computer program is stored on the readable storage medium, and the above method steps are implemented when the computer program is executed by the processor.

[0068] The cloud-native highly available object storage method based on erasure code, the cloud-native highly available storage method and system based on erasure code support multiple CPU architectures such as x86 and arm64, realize dynamic expansion of storage and data integrity protection. When data is lost or damaged, replicas can be used for recovery, improving the high availability performance of storage services, ensuring the stability and continuity of core services, and enhancing the security and high availability capabilities of the cloud-native platform.

[0069] The above-described embodiments are only one of the specific implementation manners of the present invention, and the common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-native high-availability object storage method based on erasure coding, characterized by: Use erasure coding mechanism to ensure high availability and use HighWayHash function to handle data corruption; including the following steps: Step S1: Odd-even separation In an Erasure Code group, the original data is divided into k data blocks, and then mk check blocks are generated, where m is greater than k, to provide data integrity protection. This separation method is called parity separation. Step S2: Matrix modeling Use the Vandermonde matrix to construct an m*k matrix for erasure coding; the elements in the matrix are Byte values ​​over the GF(2^8) finite field; Step S3: Storage distribution Each data block and check block is stored on different nodes in the Erasure Code group to improve data reliability and ensure that distributed data storage does not cause single point failure; Step S4: Encoding calculation When data is to be read, the inverse of the Vandermonde matrix is ​​used to calculate the data block and check block; Step S5: Dynamic expansion and contraction When the number of storage nodes increases or decreases, the coefficient matrix is ​​recalculated without recopying the entire data set, thus achieving redistributed storage and dynamic capacity expansion.

2. The cloud-native high-availability object storage method based on erasure coding according to claim 1 is characterized in that: In step S2, erasure codes and checksums are used to protect data from hardware failures and silent data damage; data can be recovered as long as the lost data is less than half of the total data; when the lost or damaged data in the disk reaches half of the total data or more, the data cannot be recovered.

3. The cloud-native high-availability object storage method based on erasure coding according to claim 1 is characterized in that: In step S2, the original data is encoded by an erasure coding algorithm to obtain redundant data, and the original data and the redundant data are stored together to achieve the purpose of fault tolerance; the details are as follows: The n blocks of original data elements are calculated through parity separation to obtain q blocks of redundant elements as check blocks, totaling n+q blocks of elements; when any v block elements in the original data and redundant data are erroneous, v is less than or equal to q, and the original data is restored through the HighWayHash function in combination with the non-erroneous check blocks to obtain n blocks of original data elements.

4. The cloud-native high-availability object storage method based on erasure coding according to claim 1 is characterized in that: In step S3, erasure codes are automatically introduced using a distributed storage mode, and erasure codes are used to prevent multiple node crashes and bit decay.

5. The cloud-native high-availability object storage method based on erasure coding according to claim 4 is characterized in that: In step S3, the distributed storage mode requires at least 4 disks.

6. The cloud-native high-availability object storage method based on erasure coding according to claim 4 is characterized in that: In the step S4, the coding calculation is performed on any Erasure Code group node, and different Erasure Code group nodes are supported to perform calculations simultaneously; the calculated data blocks are combined to generate original data.

7. A cloud-native high-availability object storage system based on erasure coding, characterized by: include: The parity separation module is responsible for dividing the original data in an Erasure Code group into k data blocks, and then generating mk check blocks, where m is greater than k, to provide data integrity protection; The matrix modeling module is responsible for constructing an m*k matrix using the Vandermonde matrix for erasure coding; the elements in the matrix are byte values ​​over the GF(2^8) finite field; Distributed storage module, responsible for storing each data block and check block on different nodes in the Erasure Code group to improve data reliability and ensure that distributed data storage does not cause single point failure; The encoding calculation module is responsible for using the inverse matrix of the Vandermonde matrix to calculate the data block and the check block when reading data; The dynamic expansion and contraction module is responsible for recalculating the coefficient matrix when the number of storage nodes increases or decreases, without the need to re-copy the entire data set, thus achieving redistributed storage and dynamic expansion.

8. A cloud-native high-availability object storage device based on erasure coding, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method steps as claimed in any one of claims 1 to 6 when executing the computer program.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 6 are implemented.

Images