A method and system for updating erasure code data in a distributed hybrid storage system

Abstract

The present invention proposes a method and system for updating erasure code data in a distributed hybrid storage system, including constructing a generation matrix; determining whether the valid data in the performance layer constitutes a complete erasure code stripe; if so, reading the entire stripe data block of the performance layer, calculating and generating a new erasure code check block, overwriting the data block and the check block at one time and writing them to the capacity layer, and using a full-stripe update consistency repair strategy to repair them after a failure occurs; if not, reading the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions in the capacity layer, writing the calculated new erasure code check block and the new data block in a certain order to the capacity layer, and using a non-full-stripe update consistency repair strategy to repair them after a failure occurs. The present invention avoids the additional I / O operations required to maintain new and old versions of data on the HDD, improves storage resource utilization, and ensures that consistency and integrity can be quickly and efficiently restored when a write fails and possible data loss occurs.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a method and system for updating erasure code data in a distributed hybrid storage system. Background Art

[0002] A distributed storage system integrates storage resources from multiple servers into a single virtual storage pool. By distributing data across multiple nodes, this improves data access performance while enhancing the system's fault tolerance and availability. Users can dynamically allocate and return storage resources, enjoying high-performance and highly reliable storage services. Figure 1 This is a typical distributed hybrid storage system that combines the characteristics of different types of storage media (such as solid-state drives (SSDs) and mechanical hard disks (HDDs)) to achieve both high performance and high storage efficiency. Data written by users is preferentially stored in the high-speed SSD tier, known as the performance tier, ensuring high data throughput and low latency. Subsequently, background tasks migrate the data to the larger HDD tier, known as the capacity tier, optimizing overall storage costs. In this architecture, writes to the SSD tier are typically stored in multiple copies to ensure write speed, while the HDD tier uses efficient erasure coding technology to store data, significantly reducing storage space usage.

[0003] Erasure Coding (EC) is a data redundancy technology used to provide high data reliability while reducing storage overhead. It divides data into multiple data blocks and check blocks (collectively referred to as coding blocks). When any data block or check block is lost, the data can still be recovered through the remaining blocks. Compared with traditional multi-copy technology, erasure coding can achieve high fault tolerance with lower storage overhead. However, when updating data, erasure coding needs to update the relevant check blocks at the same time, which makes the data consistency problem more complicated. How to ensure the consistency of erasure code stripes when data is transferred from SSD to HDD, especially how to quickly restore consistency when a failure occurs, is a technical problem that needs to be solved urgently.

[0004] There are two main methods for solving the above consistency problem in the existing technology:

[0005] The first is based on the Multi-Version Parallel Write solution, which is used to solve the data consistency problem in erasure code storage. The advantage of this solution is that even if a power outage or node failure occurs during the update process, the system can still rely on the old version data to recover and ensure consistency because the old version stripe remains intact. At the same time, the generation and switching of the new version are controlled by metadata, which can effectively avoid the stripe inconsistency problem caused by partial writes. However, the multi-version parallel write solution also has a certain storage space and write performance overhead, because before the new version switch is completed, the data blocks and check blocks of the old and new versions need to be retained at the same time. In addition, the recovery of old version data depends on background cleanup tasks, which may increase management complexity.

[0006] The second approach is to update data by reading the original data from the HDD to the SSD, constructing a complete erasure coded (EC) stripe, and then writing the entire stripe. This solution has the advantages of simple logic. By fully overwriting the data, it eliminates inconsistencies caused by partial stripe updates and ensures the integrity of the stripe after the update. If a failure occurs during the update process, the system can rely on multiple copies of data on the SSD to reconstruct the complete stripe and complete the write, ensuring data security. However, this solution incurs additional I / O overhead by reading all data blocks and check blocks on the HDD, which can lead to performance bottlenecks, especially when the HDD has low random read performance. Secondly, the storage layer requires the SSD to provide sufficient cache space to store the complete stripe data, which places additional demands on SSD capacity and management. Finally, in scenarios with large-scale concurrent updates, excessive I / O operations may reduce overall system performance.

[0007] As storage scale expands, the existing technologies suffer from severe I / O amplification and low storage resource utilization, which seriously affect the performance and efficiency of erasure code storage systems. Striking a balance between efficient storage and data reliability has become an important technical challenge. Summary of the Invention

[0008] In view of this, the present invention proposes a method and system for updating erasure code data in a distributed hybrid storage system, which avoids the additional I / O operations required to maintain new and old versions of data on the HDD, improves storage resource utilization, and ensures that consistency can be restored quickly and efficiently in the event of write failure or data loss.

[0009] The present invention proposes a method for updating erasure code data in a distributed hybrid storage system, comprising:

[0010] Constructing a generator matrix according to the data block parameters and check block parameters of the erasure code;

[0011] Determine whether the valid data in the performance layer constitutes a complete erasure code stripe;

[0012] If so, read the entire stripe data block of the performance layer, and use the generator matrix to calculate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair it after a failure occurs;

[0013] If not, read the old data blocks and all check blocks at the corresponding positions of the performance layer's data to be updated and the capacity layer, and use the generation matrix to calculate the new erasure code check block. First write the calculated new erasure code check block to the capacity layer. After success, write the new data block to the capacity layer, and use the non-full stripe update consistency repair strategy to repair it after a failure occurs.

[0014] Exemplarily, the data block parameters include the number of data blocks, the check block parameters include the number of check blocks, and constructing a generator matrix based on the data block parameters and the check block parameters of the erasure code includes:

[0015] Constructing a first Vandermonde matrix and a second Vandermonde matrix, wherein the number of rows of the first Vandermonde matrix is ​​the number of the data blocks, and the number of columns is the sum of the number of the data blocks and the number of the check blocks, and the number of rows and columns of the second Vandermonde matrix are both the number of the data blocks;

[0016] Inverting the second Vandermonde matrix to obtain a second inverse Vandermonde matrix;

[0017] The generator matrix is ​​obtained by multiplying the first Vandermonde matrix and the second Vandermonde inverse matrix.

[0018] Exemplarily, the method of using a full stripe update consistency repair strategy to perform repair after a failure occurs includes:

[0019] Storing complete erasure coded stripes in multiple copies at the performance layer;

[0020] Read the full-stripe new data block from the performance layer, and use the full-stripe new data block and the generation matrix to calculate the new check block, and use the new data block and the new check block to overwrite the old data block and the old check block on the capacity layer to complete the full-stripe update consistency repair.

[0021] Exemplarily, reading the data to be updated in the performance layer and the old data blocks and all check blocks at corresponding positions in the capacity layer, and writing the calculated new erasure code check blocks and new data blocks into the capacity layer includes:

[0022] Read the new data block that needs to be updated from the performance layer, and simultaneously read the old data block and old check block at the corresponding position from the capacity layer;

[0023] New check blocks are calculated using the generator matrix, and the new check blocks are overwritten into the capacity layer. After all new check blocks are successfully written, new data blocks are overwritten and unmodified data blocks are retained.

[0024] Exemplarily, the repairing by adopting a non-full stripe update consistency repair strategy after a failure occurs includes:

[0025] Determine the failure type of the inconsistency in the partial stripe update, where the failure type includes a first failure type, a second failure type, and a third failure type;

[0026] If it is a first-class failure, the erasure code stripe is repaired to the latest version using all data blocks in the performance layer and updated data blocks and check blocks in the capacity layer that have not experienced any failures.

[0027] If it is a second type of failure, the erasure code stripe is repaired to the old version using data blocks and check blocks in the capacity layer that have not failed and have not been updated;

[0028] If it is a third type of failure, a repair matrix for consistency repair is constructed using the new data block, the updated new check block, and the generation matrix. The erasure code stripe is repaired to a consistent old version through the consistency repair matrix, and the old version of the erasure code stripe is further updated to the new version using the performance layer data.

[0029] The full stripe update consistency repair strategy and the non-full stripe update consistency repair strategy mentioned in the embodiments of the present invention are described in detail below.

[0030] First, the full-stripe update consistency repair strategy. For full-stripe updates, the SSD stores the complete EC stripe data blocks in multiple copies. During an update, the entire EC stripe data block is read from the SSD, the parity block is recalculated, and the data block and parity block are overwritten to the corresponding location on the HDD.

[0031] The second strategy involves repairing the consistency of partial stripe updates. If a partial stripe update fails due to a fault, only a portion of the stripe's data will be stored on the SSD, and the EC stripe on the HDD may be partially updated. The type of stripe inconsistency must be determined, and fault types include Type 1, Type 2, and Type 3.

[0032] Type 1: If it is a Type 1 failure, the erasure coded stripe is repaired to the latest version using all data blocks in the performance tier and surviving data blocks in the capacity tier.

[0033] Type 2: If it is the second type of failure, the erasure code stripe is repaired to the old version using the unupdated and non-failed data blocks and parity blocks in the capacity layer.

[0034] Type 3: If it is a type-3 fault, a repair matrix for consistency repair is constructed using the new data block and the newly updated parity block, and the erasure code stripe is repaired to the old version through the repair matrix for consistency repair;

[0035] Specifically, since it is required to write the parity block successfully before writing the data block during writing, when the inconsistency type belongs to this type, the data block must still not have been updated, while the parity block has been partially updated. Repairing this type of inconsistency requires the existence of data in the SSD replica layer. First, a transformed parity block is calculated using the non-faulty and updated parity blocks in the capacity layer, the data blocks in the performance layer, and the generation matrix; then, the transformed parity block and the non-faulty and non-updated parity blocks in the capacity layer are jointly used to form a transformed parity block list; then, the non-faulty and non-updated data blocks in the capacity layer, the transformed parity block list, and the repair matrix for consistency repair are used to calculate the complete old-version erasure code stripe; finally, the new data blocks in the performance layer are used to update the old-version erasure code stripe to obtain a complete new-version erasure code stripe, thereby completing the consistency repair. Hereinafter, the embodiments of the present invention will specifically describe the method using EC(k+m).

[0036] First, let the old coding blocks be D1, D2, D3,....., D k , P1, P2,...., P m , where D represents a data block and P represents a parity block, and the Roman numerals and k, m represent the serial numbers of the coding blocks. Let the generation matrix of the adopted EC coding be shown in the following formula, and the matrix is a fixed value and is generated by the method of step S101.

[0037] B 11 B 12 B 13 ...B 1k

[0038] B 21 B 22 B 23 ...B 2k ....

[0040] B m1 B m2 B m3 ...B mk

[0041] Assume that there are x new data blocks (x < k) in the SSD. After the update fails, y parity blocks are successfully updated (y < m). The data in the SSD and HDD are as follows:

[0042] SSD: D a ’, Db '、D c ',...D d '(a,b,c,d are numbers, apostrophes represent new data)

[0043] HDD: D1, D2, D3, ...., D k , P1', P2', ...., P m ' (The apostrophe in the checksum indicates that it may or may not have been updated).

[0044] Let P i ' is a new check block, then each new check block is composed of P i '=D a '*B ia +..+D d '*B id +..D u *B iu +..+D w *B iw Calculated, where a...d are the numbers of new data blocks in the SSD, and u...w are the numbers of old data blocks that have not been updated in the HDD.

[0045] By moving all the terms involving the new data block on the right side of the above formula to the left side, we can get P i *=P i '-(D a '*B ia +..+D d '*B id )=D u *B iu +..+D w *B iw , the new data blocks involved in the middle terms of this formula are located in the SSD and are known values. The coefficients of the generated matrix are fixed values, so P can be calculated. i *. All parity blocks are converted in this way, and the following matrix equation is finally obtained, where u..w are the numbers of the old data blocks that have not been updated in the HDD.

[0046]

[0047] The above formula represents a new generator matrix for RS encoding. If this generator matrix is ​​reversible, even if m coding blocks are lost, the lost coding blocks can be recovered using the surviving coding blocks. The data blocks obtained after decoding are old-version data blocks. To obtain new-version data blocks, simply update them using data from the SSD. The generator matrix provided by this embodiment of the present invention ensures that the generator matrix obtained by the consistency repair conversion in the previous paragraph is reversible even if m blocks are lost.

[0048] Exemplarily, repairing the erasure code stripe to a new version using the repair matrix of the consistency repair includes:

[0049] Calculate a conversion check block using the updated check blocks in the capacity layer that have not experienced a fault, the data blocks in the performance layer, and the generator matrix;

[0050] The conversion check block and the check blocks in the capacity layer that have not failed and have not been updated together form a conversion check block list;

[0051] Calculate a complete old version erasure code stripe using the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair;

[0052] The old version erasure code stripe is updated using the new data blocks in the performance layer to obtain a complete new version erasure code stripe, thereby completing consistency repair.

[0053] Exemplarily, the first type of failure is that the number of all data blocks in the performance layer and the data blocks and check blocks that have not failed and have been updated in the capacity layer is sufficient to repair the erasure code stripe; the second type of failure is that the number of data blocks and check blocks that have not failed and have not been updated in the capacity layer is sufficient to repair the erasure code stripe; the third type of failure is all other failures that cannot be classified into the first or second type of failures.

[0054] Exemplarily, the first Vandermonde matrix symbol is represented as:

[0055] A=V1(K,K+M)

[0056] The second Vandermonde matrix symbol is represented as:

[0057] B=V2(K,K)

[0058] The calculation formula of the generator matrix is:

[0059] C=A*B -1

[0060] Among them, V1 and V2 are Vandermonde matrices constructed using axioms, A is the first Vandermonde matrix, B is the second Vandermonde matrix, C is the generator matrix, K is the data block number parameter of the erasure code, and M is the check block number parameter of the erasure code.

[0061] For example, the written data is preferentially stored in the performance layer in the form of multiple copies, and is converted into an erasure code storage form when the data is transferred from the performance layer to the capacity layer.

[0062] The present invention proposes an erasure code data update system for a distributed hybrid storage system, comprising:

[0063] A generating unit, configured to construct a generating matrix according to data block parameters and check block parameters of the erasure code;

[0064] A judgment unit, used to judge whether the valid data of the performance layer constitutes a complete erasure code stripe;

[0065] A first repair unit is configured to read the entire stripe data block of the performance layer when the valid data of the performance layer can form a complete erasure coded stripe, calculate and generate a new erasure coded check block, overwrite the data block and the check block to the capacity layer at one time, and perform repair using a full stripe update consistency repair strategy after a failure occurs;

[0066] The second repair unit is used to read the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer when the valid data in the performance layer cannot form a complete erasure code stripe, and use the said generation matrix to calculate the new erasure code check block, first write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success, and adopt the non-full stripe update consistency repair strategy to perform repair after a failure occurs.

[0067] The present invention proposes a method and system for updating erasure code data in a distributed hybrid storage system, including constructing a generation matrix based on the data block parameters and check block parameters of the erasure code; determining whether the valid data in the performance layer constitutes a complete erasure code stripe; if so, reading the entire stripe data block of the performance layer, and calculating and generating a new erasure code check block, overwriting the data block and the check block at one time and writing them to the capacity layer, and using a full stripe update consistency repair strategy to repair them after a failure occurs; if not, reading the old data block and all check blocks at the corresponding position of the performance layer data to be updated and the capacity layer, writing the calculated new erasure code check block and the new data block to the capacity layer, and using a non-full stripe update consistency repair strategy to repair them after a failure occurs. The technical solution of the present invention avoids the additional I / O operations required to maintain new and old versions of data on the HDD, improves storage resource utilization, and ensures that consistency can be quickly and efficiently restored in the event of write failure or data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0069] Figure 1 A schematic diagram of a typical distributed hybrid storage system provided by an embodiment of the present invention;

[0070] Figure 2Schematic diagram of partially updating EC data on HDD with SSD data according to an embodiment of the present invention;

[0071] Figure 3 A schematic flow chart of a method for updating erasure code data in a distributed hybrid storage system provided by an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of a generator matrix calculation method provided in an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of the erasure code data update process provided by an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of the full stripe update consistency repair process provided by an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of the non-full stripe update consistency repair process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0077] Erasure Coding (EC) is a data redundancy technology used to provide high data reliability while reducing storage overhead. It divides data into multiple data blocks and check blocks (collectively referred to as coding blocks). When any data block or check block is lost, the data can still be recovered through the remaining blocks. The typical form of erasure coding is the k+m mode, where k represents the number of data blocks and m represents the number of check blocks. For example, EC(4+2) means there are 4 data blocks and 2 check blocks. Compared with traditional multi-copy technology, erasure coding can achieve high fault tolerance with lower storage overhead. However, when updating data, erasure coding needs to update the relevant check blocks at the same time, which makes the data consistency problem more complicated.

[0078] like Figure 2As shown in the figure, assume that in a distributed hybrid storage system, data in the capacity layer is stored in a 4+2 erasure-coded (EC) format, with four data blocks (D1, D2, D3, D4) and two parity blocks (P1, P2) forming a stripe. During data migration from SSDs to HDDs, some data blocks (such as D1 and D2) may need to be updated. In this case, the parity blocks (P1 and P2) must be updated synchronously to maintain stripe consistency. A failure during the update process, such as a power outage or node failure, may result in some data blocks and parity blocks (such as D1', D2', and P1') being updated, while parity block P2 remains at the old version. In this case, the data blocks and parity blocks within the stripe do not match, and both the old and new versions contain fewer than k copies. This makes it impossible for the system to recover lost data blocks using the parity blocks, potentially leading to the risk of data loss. Ensuring the consistency of erasure-coded stripes during data migration from SSDs to HDDs, particularly how to quickly restore consistency in the event of a failure, is an urgent technical challenge.

[0079] The embodiments of the present invention can effectively reduce implementation complexity by optimizing the update process, reducing unnecessary I / O operations, and improving storage resource utilization. Figure 1 , the embodiment of this patent is based on the typical architecture of a distributed hybrid storage system, including a performance layer (SSD) and a capacity layer (HDD). The data written by the user is first stored in the SSD, and high performance and data reliability are guaranteed through multiple copies of storage. In the background task, the data will be converted into an erasure code (EC) storage format during the process of sinking from the SSD to the HDD. It should be noted that the embodiment of this patent only solves the consistency problems that may be encountered in the process of data sinking from the SSD to the HDD, and does not discuss the writing process on the user side, the reading process of EC data, etc.

[0080] Example 1

[0081] Figure 3 A schematic flow chart of a method for updating erasure code data in a distributed hybrid storage system provided by an embodiment of the present invention, the method specifically includes:

[0082] Step S101, constructing a generator matrix according to data block parameters and check block parameters of the erasure code, where the data block parameters include the number of data blocks, and the check block parameters include the number of check blocks;

[0083] Specifically, the embodiment of the present invention generates the first Vandermonde matrix and the second Vandermonde matrix according to the k and m parameters of EC, referring to Figure 4 The generator matrix is ​​obtained by the regular process, and the generator matrix is ​​used to calculate the check block in the future. The first Vandermonde matrix symbol is represented as:

[0084] A=V1(K,K+M)

[0085] The second Vandermonde matrix symbol is:

[0086] B=V2(K,K)

[0087] The calculation formula of the generated matrix is:

[0088] C=A*B -1

[0089] V1 and V2 are Vandermonde matrices constructed using axioms, A is the first Vandermonde matrix, B is the second Vandermonde matrix, C is the generator matrix, K is the erasure code's parameter for the number of data blocks, and M is the erasure code's parameter for the number of parity blocks. The first Vandermonde matrix has rows equal to the number of data blocks, and columns equal to the sum of the number of data blocks and the number of parity blocks. The second Vandermonde matrix has rows and columns equal to the number of data blocks. By inverting the second Vandermonde matrix to obtain the second inverse Vandermonde matrix, the generator matrix can be calculated based on the first and second inverse Vandermonde matrices.

[0090] Step S102, determining whether the performance layer valid data constitutes a complete erasure code stripe;

[0091] If yes, then execute step S1031 to read the entire stripe data block of the performance layer, and use the generator matrix to calculate the new erasure code check block, overwrite the data block and check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair it after a failure occurs;

[0092] Specifically, the entire stripe's data blocks are read from the SSD, a new EC parity block is calculated, and all data blocks and parity blocks are written to the corresponding storage locations on the HDD at once. If a failure occurs during the write process, the system will perform a repair according to the steps described in "Full Stripe Update Consistency Repair" to ensure data consistency.

[0093] If not, execute step S1032, then read the old data blocks and all check blocks at the corresponding positions of the performance layer's data to be updated and the capacity layer, and use the generation matrix to calculate the new erasure code check block, first write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success, and use the non-full stripe update consistency repair strategy to repair it after a failure occurs.

[0094] Specifically, if the valid data on the SSD cannot form a complete EC stripe, the new data that needs to be updated is read from the SSD, and the old data blocks and all check blocks at the corresponding position are read from the HDD. A new check block is generated by combining the old and new data. Subsequently, the new check block is overwritten and written to the HDD first, and after ensuring that all check blocks are written successfully, the new data block is overwritten. For data blocks that have not been modified, there is no need to rewrite them. If a failure occurs during the writing process, the system will perform repairs according to the step strategy described in "Non-full stripe update consistency repair" to ensure stripe consistency. Please refer to Figure 5 Erasure code data update process.

[0095] When the data on the SSD does not form a complete stripe, the check block needs to be updated first and then the data block. Compared with existing technical solutions, on the one hand, there is no need to maintain multiple versions of new and old data on the HDD, and the new data can directly overwrite the old data. On the other hand, no data needs to be written to the SSD, and there is no write amplification problem. Through the method and process of the embodiment of the present invention, in most cases, the data can be written in parallel without the need for consistency repair.

[0096] The full stripe update consistency repair strategy and the non-full stripe update consistency repair strategy mentioned in the embodiments of the present invention are described in detail below.

[0097] The first is the full stripe update consistency repair strategy. Figure 6 For full-stripe updates, the SSD stores the complete EC stripe data blocks in multiple copies. When updating, you only need to read the complete EC stripe data blocks from the SSD, recalculate the parity blocks, and overwrite the data blocks and parity blocks to the corresponding locations on the HDD.

[0098] The second is the non-full stripe update consistency repair strategy. Figure 7 If a partial stripe update fails due to a fault, only part of the stripe data is stored in the SSD, and the EC stripe in the HDD may be partially updated. The type of stripe inconsistency needs to be determined. Fault types include Type 1, Type 2, and Type 3.

[0099] Type 1: If it is a Type 1 failure, the erasure coded stripe is repaired to the latest version using all data blocks in the performance tier and surviving data blocks in the capacity tier.

[0100] Specifically, this type means that the EC stripe can be repaired to the latest version using surviving new-version code blocks. Surviving new-version code blocks refer to all data blocks in the SSD replica layer, updated, unfailed parity blocks in the HDD EC layer, and unfailed data blocks in the HDD EC layer that are not related to this I / O. This type is guaranteed to apply if all parity blocks have been successfully updated.

[0101] Type 2: If it is the second type of failure, the erasure code stripe is repaired to the old version using the unupdated and non-failed data blocks and parity blocks in the capacity layer.

[0102] Specifically, this type means that the EC stripe can be repaired to the old version using surviving old version code blocks. Here, surviving old version code blocks refer to the data blocks and parity blocks in the HDD EC layer that have not been updated and have not failed.

[0103] Type 3: If it is the third type of failure, a repair matrix for consistency repair is constructed using the new data block and the updated new parity block, and the erasure code stripe is repaired to the old version using the repair matrix for consistency repair;

[0104] Specifically, since writing requires that the check block be written successfully before writing the data block, when the inconsistency type belongs to this type, the data block must still not be updated, and the check block is partially updated. Repairing this type of inconsistency requires that the data in the SSD replica layer must exist. First, the conversion check block is calculated using the check blocks in the capacity layer that have not failed and have been updated, the data blocks in the performance layer, and the generation matrix; then, the conversion check block and the check blocks in the capacity layer that have not failed and have not been updated are used to form a conversion check block list; then, the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair are used to calculate a complete old version erasure code stripe; finally, the old version erasure code stripe is updated using the new data block in the performance layer to obtain a complete new version erasure code stripe, thereby completing the consistency repair. The following embodiment of the present invention uses EC(k+m) to illustrate the specific method.

[0105] First, let the old code blocks be D1, D2, D3, ....., D k , P1, P2, ...., P m , where D represents the data block, P represents the check block, and the Roman numerals k and m represent the sequence numbers of the coding blocks. Assume that the generator matrix of the EC code used is as shown below. The matrix is ​​a fixed value and is generated using the method in step S101.

[0106] B 11 B 12 B 13 ...B1k

[0107] B 21 B 22 B 23 ...B 2k ....

[0109] B m1 B m2 B m3 ...B mk

[0110] Assume there are x new data blocks in the SSD (x < k). After an update failure, y parity blocks are successfully updated (y < m). The data in the SSD and HDD are as follows:

[0111] SSD: D a ’, D b ’, D c ’,...D d ’ (a, b, c, d are numbers, and the apostrophe represents new data)

[0112] HDD: D1, D2, D3,....., D k , P1’, P2’,...., P m ’ (The apostrophe for the parity block means it may or may not have been updated).

[0113] Let P i ’ be the new parity block. Then each new parity block is calculated by P i ’ = D a ’ * B ia +.. + D d ’ * B id +..D u * B iu +.. + D w * B iw It is calculated that, where a...d are the numbers of the new data blocks in the SSD, and u...w are the numbers of the old data blocks in the HDD that have not been updated.

[0114] Move all the terms involving new data blocks on the right side of the above formula to the left side, then we can get P i * = P i ’ - (D a ’ * B ia +.. + D d ’ * B id ) = D u * B iu +.. + D w * B iw, the new data blocks involved in the middle terms of this formula are located in the SSD and are known values. The coefficients of the generated matrix are fixed values, so P can be calculated. i *. All parity blocks are converted in this way, and the following matrix equation is finally obtained, where u..w are the numbers of the old data blocks that have not been updated in the HDD.

[0115]

[0116] The above formula represents a new generator matrix for RS encoding. If this generator matrix is ​​reversible, even if m coding blocks are lost, the lost coding blocks can be recovered using the surviving coding blocks. The data blocks obtained after decoding are old-version data blocks. To obtain new-version data blocks, simply update them using data from the SSD. The generator matrix provided by this embodiment of the present invention ensures that the generator matrix obtained by the consistency repair conversion in the previous paragraph is reversible even if m blocks are lost.

[0117] An embodiment of the present invention proposes a method for repairing data consistency in a hybrid storage system after a failure during erasure code writing causes data inconsistency. This method constructs a consistency repair generator matrix based on conditions such as whether the data is alive and whether the data is a new or old version. This generator matrix is ​​then used to recover lost data and repair inconsistent data. Advantageous effects of this embodiment of the present invention include:

[0118] (1) I / O overhead is significantly reduced. The additional I / O operations required to maintain the old and new versions of data on the HDD are avoided, and new data blocks and check blocks can be directly overwritten, reducing the burden of metadata management and the recycling of old versions of data. In addition, there is no need to write any data to the SSD, which greatly alleviates the problem of I / O amplification and reduces resource usage and performance consumption during data migration.

[0119] (2) Improved storage resource utilization. By fully leveraging the performance characteristics of SSDs and HDDs, the need to write multiple copies to the SSDs is avoided. Furthermore, through an efficient erasure code update process, the storage resource utilization of SSDs and HDDs is significantly improved. Furthermore, there is no need to retain old versions of data and other metadata on the HDDs and SSDs, further improving the overall efficiency of the system's storage resources.

[0120] (3) The repair process is efficient and independent of the underlying system. The overall process design is concise, clearly defining the processing paths for full-stripe updates and partial-stripe updates, and avoiding the need to manage and maintain complex multi-version data. During the consistency repair process, efficient repair based on erasure codes is achieved through matrix conversion calculations, reducing dependence on the underlying storage system. This makes it more operational in actual deployments and reduces implementation difficulty and maintenance costs.

[0121] (4) Improved system reliability. In both full-stripe and partial-stripe update scenarios, the embodiments of the present invention provide a clear fault repair process to ensure that consistency can be quickly and efficiently restored in the event of a write failure or data loss. In particular, for partial-stripe update scenarios, by designing a reversible consistency repair matrix and mathematical transformations, data blocks and check blocks can still be recovered even in the event of a partial update that causes inconsistencies and the loss of m coding blocks, effectively improving system reliability.

[0122] Example 2

[0123] A distributed hybrid storage system erasure code data update system, comprising:

[0124] A generating unit, configured to construct a generating matrix according to data block parameters and check block parameters of the erasure code;

[0125] A judgment unit, used to judge whether the valid data of the performance layer constitutes a complete erasure code stripe;

[0126] The first repair unit is configured to read the entire stripe data block of the performance layer when the valid data of the performance layer can form a complete erasure coded stripe, calculate and generate a new erasure coded check block, overwrite the data block and check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to perform repair after a failure occurs;

[0127] The second repair unit is used to read the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions in the capacity layer when the valid data in the performance layer cannot form a complete erasure code stripe, write the calculated new erasure code check blocks and new data blocks to the capacity layer, and use a non-full stripe update consistency repair strategy for repair after a failure occurs.

[0128] It can be understood that the above-mentioned erasure code data updating system of a distributed hybrid storage system corresponds to the erasure code data updating method of a distributed hybrid storage system in Example 1. Any optional items in Example 1 are also applicable to this embodiment and will not be described in detail here.

[0129] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A method for updating erasure code data in a distributed hybrid storage system, characterized in that: include: Constructing a generator matrix according to the data block parameters and check block parameters of the erasure code; Determine whether the valid data in the performance layer constitutes a complete erasure code stripe; If so, read the entire stripe data block of the performance layer, and use the generator matrix to calculate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair it after a failure occurs; If not, read the old data blocks and all check blocks at the corresponding positions of the data to be updated in the performance layer and the capacity layer, and use the generation matrix to calculate the new erasure code check block. The calculated new erasure code check block is first written to the capacity layer. After success, the new data block is written to the capacity layer, and after a failure occurs, a non-full stripe update consistency repair strategy is used to repair it; First, the calculated new erasure code check block is written to the capacity layer. After success, the new data block is written to the capacity layer. Specifically, it includes: If the valid data on the SSD cannot form a complete EC stripe, the new data that needs to be updated is read from the SSD, and the old data blocks and all parity blocks at the corresponding position are read from the HDD at the same time; a new parity block is calculated based on the new and old data; then, the new parity block is overwritten and written to the HDD first, and after ensuring that all parity blocks are written successfully, the new data block is overwritten; for data blocks that have not been modified, there is no need to rewrite them.

2. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The data block parameters include the number of data blocks, the check block parameters include the number of check blocks, and constructing a generator matrix based on the data block parameters and the check block parameters of the erasure code includes: Constructing a first Vandermonde matrix and a second Vandermonde matrix, wherein the number of rows of the first Vandermonde matrix is ​​the number of the data blocks, and the number of columns is the sum of the number of the data blocks and the number of the check blocks, and the number of rows and columns of the second Vandermonde matrix are both the number of the data blocks; Inverting the second Vandermonde matrix to obtain a second inverse Vandermonde matrix; The generator matrix is ​​obtained by multiplying the first Vandermonde matrix and the second Vandermonde inverse matrix.

3. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The method of using a full stripe update consistency repair strategy to repair a failure after the failure occurs includes: Storing complete erasure coded stripes in multiple copies at the performance layer; Read the full-stripe new data block from the performance layer, and use the full-stripe new data block and the generation matrix to calculate the new check block, and use the new data block and the new check block to overwrite the old data block and the old check block on the capacity layer to complete the full-stripe update consistency repair.

4. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, wherein: The step of reading the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions in the capacity layer, and writing the calculated new erasure code check blocks and the new data blocks into the capacity layer comprises: Read the new data block that needs to be updated from the performance layer, and simultaneously read the old data block and old check block at the corresponding position from the capacity layer; New check blocks are calculated using the generator matrix, and the new check blocks are overwritten into the capacity layer. After all new check blocks are successfully written, new data blocks are overwritten and unmodified data blocks are retained.

5. The method for updating erasure code data in a distributed hybrid storage system according to claim 4, characterized in that: The repair strategy using a non-full stripe update consistency repair strategy after a failure occurs includes: Determine the failure type of the inconsistency in the partial stripe update, where the failure type includes a first failure type, a second failure type, and a third failure type; If it is a first-class failure, the erasure code stripe is repaired to the latest version using all data blocks in the performance layer and updated data blocks and check blocks in the capacity layer that have not experienced any failures. If it is a second type of failure, the erasure code stripe is repaired to the old version using data blocks and check blocks in the capacity layer that have not failed and have not been updated; If it is a third type of failure, a repair matrix for consistency repair is constructed using the new data block, the updated new check block, and the generation matrix. The erasure code stripe is repaired to a consistent old version through the consistency repair matrix, and the old version of the erasure code stripe is further updated to the new version using the performance layer data.

6. The method for updating erasure code data in a distributed hybrid storage system according to claim 5, characterized in that: Repairing the erasure code stripe to a new version using the repair matrix of the consistency repair includes: Calculate a conversion check block using the updated check blocks in the capacity layer that have not experienced a fault, the data blocks in the performance layer, and the generator matrix; The conversion check block and the check blocks in the capacity layer that have not failed and have not been updated together form a conversion check block list; Calculate a complete old version erasure code stripe using the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair; The old version erasure code stripe is updated using the new data blocks in the performance layer to obtain a complete new version erasure code stripe, thereby completing consistency repair.

7. The method for updating erasure code data in a distributed hybrid storage system according to claim 6, characterized in that: The repair matrix expression for consistency repair is as follows: ； in, is an element in the conversion check block list, u, w are the numbers of the old data blocks that have not been updated in the capacity layer, B is the generation matrix, D is the data block, k is the total number of old data block numbers in the capacity layer, and i is the element number of the repair matrix.

8. The method for updating erasure code data in a distributed hybrid storage system according to claim 7, characterized in that: The first type of failure is that the number of all data blocks in the performance layer and the data blocks and parity blocks that have not failed and have been updated in the capacity layer is sufficient to repair the erasure code stripe. The second type of failure is that the number of data blocks and parity blocks that have not failed and have not been updated in the capacity layer is sufficient to repair the erasure code stripe. The third type of failure is all other failures that cannot be classified into the first or second type of failures.

9. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, wherein: The written data is stored in the performance layer in a multi-copy format and is converted into erasure coded storage as the data is transferred from the performance layer to the capacity layer.

Images