Erasure code data updating method and system of distributed hybrid storage system

Abstract

The invention provides an erasure code data updating method and system of a distributed hybrid storage system. The erasure code data updating method comprises the steps of constructing a generation matrix; judging whether the effective data of the performance layer form a complete erasure code strip or not; if yes, reading a whole-strip data block of a performance layer, calculating and generating a new erasure code check block, writing the data block and the check block into a capacity layer in a one-time coverage manner, and repairing by adopting a whole-strip updating consistency repairing strategy after a fault occurs; and if not, reading the data to be updated of the performance layer and the old data blocks and all the check blocks at the corresponding positions of the capacity layer, writing the calculated new erasure code check blocks and new data blocks into the capacity layer according to a certain sequence, and repairing by adopting a non-full stripe updating consistency repair strategy after the fault occurs. According to the method, extra I / O operation required for maintaining new and old version data on the HDD is avoided, the utilization rate of storage resources is improved, and it is ensured that consistency and integrity can be quickly and efficiently recovered when write-in fails and possible data is lost due to superposition.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a method and system for updating erasure code data in a distributed hybrid storage system. Background Art

[0002] A distributed storage system is a system architecture that integrates storage resources on multiple servers into a virtual storage pool. By distributing data on multiple nodes, it not only improves data access performance, but also enhances the system's fault tolerance and availability. Users can dynamically allocate and return storage resources and enjoy high-performance and high-reliability storage services. Figure 1 It is a typical distributed hybrid storage system that combines the characteristics of different types of storage media (such as solid-state drives (SSDs) and mechanical hard disks (HDDs)) to achieve high performance and high storage efficiency at the same time. The data written by users will be stored preferentially in the high-speed SSD layer, called the performance layer, to ensure high data throughput and low latency; then, the background task will migrate the data to the larger HDD layer, called the capacity layer, to optimize the overall storage cost. In this architecture, writes to the SSD layer are usually stored in multiple copies to ensure write speed, while the HDD layer uses efficient erasure coding technology to store data to significantly reduce storage space occupancy.

[0003] Erasure Coding (EC) is a data redundancy technology used to provide high data reliability while reducing storage overhead. It divides data into multiple data blocks and check blocks (collectively referred to as coding blocks). When any data block or check block is lost, the data can still be restored through the remaining blocks. Compared with traditional multi-copy technology, erasure coding can achieve high fault tolerance with lower storage overhead. However, when updating data, erasure coding needs to update related check blocks at the same time, which makes the data consistency problem more complicated. How to ensure the consistency of erasure code stripes in the process of data sinking from SSD to HDD, especially how to quickly restore consistency when a failure occurs, is a technical problem that needs to be solved urgently.

[0004] There are two main methods for solving the above consistency problem in the prior art:

[0005] The first is based on the multi-version parallel write (Multi-Version Writes) solution, which is used to solve the data consistency problem in erasure code storage. The advantage of this solution is that even if a power outage or node failure occurs during the update process, since the old version stripe remains intact, the system can still rely on the old version data to recover and ensure consistency. At the same time, the generation and switching of the new version are controlled by metadata, which can effectively avoid the stripe inconsistency problem caused by partial writing. However, the multi-version parallel write solution also has a certain storage space and write performance overhead, because before the new version switch is completed, the data blocks and check blocks of the old and new versions need to be retained at the same time. In addition, the recovery of the old version data depends on the background cleanup task, which may increase the management complexity.

[0006] The second method is to update data by reading the original data from the HDD to the SSD, assembling a complete erasure code (EC) stripe, and then writing the entire stripe. The advantage of this solution is that the logic is simple, and the inconsistency problem caused by partial stripe updates is eliminated by full coverage, and the integrity of the stripe after the update is guaranteed. If a failure occurs during the update process, the system can rely on multiple copies of data on the SSD to reconstruct the complete stripe and complete the write to ensure data security. However, this solution will increase additional I / O overhead due to reading all data blocks and check blocks of the HDD, especially when the random read performance of the HDD is low, which may cause performance bottlenecks. Secondly, the storage layer requires the SSD to provide sufficient cache space to save the data of the complete stripe, which puts additional requirements on the capacity and management of the SSD. Finally, in the scenario of large-scale concurrent updates, the overall system performance may be reduced due to too many I / O operations.

[0007] As storage scale expands, the problems of severe I / O amplification and low storage resource utilization in existing technologies have seriously affected the performance and efficiency of erasure code storage systems. How to strike a balance between efficient storage and data reliability has become an important technical challenge. Summary of the invention

[0008] In view of this, the present invention proposes a method and system for updating erasure code data in a distributed hybrid storage system, which avoids the additional I / O operations required to maintain new and old versions of data on the HDD, improves the utilization of storage resources, and ensures that consistency can be restored quickly and efficiently when writing fails or data is lost.

[0009] The present invention proposes a method for updating erasure code data in a distributed hybrid storage system, comprising:

[0010] Constructing a generator matrix according to the data block parameters and the check block parameters of the erasure code;

[0011] Determine whether the valid data in the performance layer constitutes a complete erasure code stripe;

[0012] If yes, read the entire stripe data block of the performance layer, and use the generation matrix to calculate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair them after a failure occurs;

[0013] If not, read the old data blocks and all check blocks at the corresponding positions of the performance layer's data to be updated and the capacity layer, and use the generation matrix to calculate the new erasure code check block. First write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success. After a failure occurs, use the non-full-stripe update consistency repair strategy to repair it.

[0014] Exemplarily, the data block parameters include the number of data blocks, the check block parameters include the number of check blocks, and constructing a generator matrix according to the data block parameters and the check block parameters of the erasure code includes:

[0015] Constructing a first Vandermonde matrix and a second Vandermonde matrix, wherein the number of rows of the first Vandermonde matrix is ​​the number of data blocks, the number of columns is the sum of the number of data blocks and the number of check blocks, and the number of rows and columns of the second Vandermonde matrix are both the number of data blocks;

[0016] Inverting the second Vandermonde matrix to obtain a second Vandermonde inverse matrix;

[0017] The generator matrix is ​​obtained by multiplying the first Vandermonde matrix and the second Vandermonde inverse matrix.

[0018] Exemplarily, the repairing by using a full stripe update consistency repair strategy after a failure occurs includes:

[0019] Storing complete erasure coded stripes in multiple copies at the performance layer;

[0020] The full-stripe new data block is read from the performance layer, and the new full-stripe new data block and the generation matrix are used to calculate the new check block, and the new data block and the new check block are used to overwrite the old data block and the old check block on the capacity layer to complete the full-stripe update consistency repair.

[0021] Exemplarily, reading the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer, and writing the calculated new erasure code check blocks and the new data blocks into the capacity layer includes:

[0022] Reading new data blocks that need to be updated from the performance layer, and synchronously reading old data blocks and old check blocks at corresponding positions from the capacity layer;

[0023] The generation matrix is ​​used to calculate a new check block, and the new check block is overwritten and written into the capacity layer. After all new check blocks are written successfully, new data blocks are overwritten and written, and unmodified data blocks are retained.

[0024] Exemplarily, the repairing by using a non-full stripe update consistency repair strategy after a failure occurs includes:

[0025] Determine the failure type of the non-full stripe update inconsistency, where the failure type includes a first failure type, a second failure type, and a third failure type;

[0026] If it is a first type of failure, the erasure code stripe is repaired to a new version using all data blocks in the performance layer and data blocks and check blocks that have not failed and have been updated in the capacity layer;

[0027] If it is a second type of failure, the erasure code stripe is repaired to the old version by using data blocks and check blocks in the capacity layer that have not failed and have not been updated;

[0028] If it is a third type of failure, a repair matrix for consistency repair is constructed using the new data block, the updated new check block and the generation matrix. The erasure code stripe is repaired to a consistent old version through the consistency repair repair matrix, and the old version of the erasure code stripe is further updated to the new version using the performance layer data.

[0029] The full stripe update consistency repair strategy and the non-full stripe update consistency repair strategy mentioned in the embodiments of the present invention are described in detail below.

[0030] The first is the full stripe update consistency repair strategy. For full stripe updates, the SSD stores the complete EC stripe data blocks in multiple copies. When updating, you only need to read the complete EC stripe data blocks from the SSD, recalculate the check blocks, and overwrite the data blocks and check blocks together to the corresponding locations on the HDD.

[0031] The second is the consistency repair strategy for non-full stripe updates. If a non-full stripe update fails due to a fault, only part of the data in the stripe will be in the SSD, and the EC stripe in the HDD may be partially updated. It is necessary to determine the type of stripe inconsistency, which includes the first type of fault, the second type of fault, and the third type of fault.

[0032] Type 1: If it is the first type of failure, the erasure code stripe is repaired to the new version using all data blocks in the performance layer and data blocks that have not failed in the capacity layer;

[0033] Type 2: If it is the second type of failure, the erasure code stripe is repaired to the old version using the data blocks and check blocks in the capacity layer that have not been updated and have not failed;

[0034] Type 3: If it is a third - type fault, a repair matrix for consistency repair is constructed using the new data block and the newly updated parity block, and the erasure - code stripe is repaired to the old version through the repair matrix for consistency repair;

[0035] Specifically, since it is required to write the data block after the parity block is successfully written during writing, when the inconsistency type belongs to this kind, the data block must still not be updated, while the parity block is partially updated. Repairing this type of inconsistency requires the existence of data in the SSD replica layer. First, a transformed parity block is calculated using the non - faulty and updated parity blocks in the capacity layer, the data blocks in the performance layer, and the generating matrix; then, the transformed parity block and the non - faulty and non - updated parity blocks in the capacity layer are used to jointly form a transformed parity - block list; then, using the non - faulty and non - updated data blocks in the capacity layer, the transformed parity - block list, and the repair matrix for consistency repair, a complete old - version erasure - code stripe is calculated; finally, the new data block in the performance layer is used to update the old - version erasure - code stripe to obtain a complete new - version erasure - code stripe, thereby completing the consistency repair. Hereinafter, the embodiments of the present invention will specifically elaborate the method using EC(k + m).

[0036] First, let the old encoding blocks be D1, D2, D3,....., D k , P1, P2,...., P m , where D represents a data block, P represents a parity block, and Roman numerals and k, m represent the serial numbers of the encoding blocks. Let the generating matrix of the adopted EC encoding be shown as the following formula, and the matrix is a fixed value, generated using the method of step S101.

[0037] B 11 B 12 B 13 ...B 1k

[0038] B 21 B 22 B 23 ...B 2k ....

[0040] B m1 B m2 B m3 ...B mk

[0041] Suppose there are x new data blocks (x < k) in the SSD. After the update fails, y parity blocks are successfully updated (y < m). The data in the SSD and HDD are as follows:

[0042] SSD: D a ’, Db '、D c ',...D d '(a,b,c,d are numbers, apostrophes represent new data)

[0043] HDD: D1, D2, D3, ....., D k , P1', P2', ...., P m ' (The apostrophe in the checksum indicates that it may or may not be updated).

[0044] Let P i ' is a new check block, then each new check block consists of P i '=D a '*B ia +..+D d '*B id +..D u *B iu +..+D w *B iw It is calculated that, where a...d is the number of the new data block in the SSD, and u...w is the number of the old data block that has not been updated in the HDD.

[0045] By moving all the terms involving the new data block on the right side of the above formula to the left side, we can get P i *=P i '-(D a '*B ia +..+D d '*B id )=D u *B iu +..+D w *B iw , the new data blocks involved in the intermediate terms of this formula are located in the SSD and are known values. The coefficients of the generated matrix are fixed values, so P can be calculated i *. All check blocks are converted in this way, and finally the following matrix equation is obtained, where u..w is the number of the old data block that has not been updated in the HDD.

[0046]

[0047] The above formula represents a new RS coding generator matrix. If the generator matrix is ​​reversible, when m coding blocks are lost, the lost coding blocks can be restored through the surviving coding blocks. The data block obtained after decoding is the old version of the data block. It only needs to be updated with the data in the SSD to obtain the new version of the data block. The generator matrix provided by the embodiment of the present invention can ensure that the generator matrix obtained by the consistency repair conversion in the previous paragraph is reversible when m blocks are lost.

[0048] Exemplarily, repairing the erasure code stripe to a new version by using the repair matrix of the consistency repair includes:

[0049] The conversion check block is calculated by using the check blocks in the capacity layer that have not failed and have been updated, the data blocks in the performance layer and the generation matrix;

[0050] The conversion check block and the check blocks in the capacity layer that have not failed and have not been updated together form a conversion check block list;

[0051] Using the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair, a complete old version erasure code stripe is calculated;

[0052] The old version erasure code stripe is updated using the new data block in the performance layer to obtain a complete new version erasure code stripe, thereby completing consistency repair.

[0053] Exemplarily, the first type of failure is that the number of all data blocks in the performance layer and the data blocks and check blocks that have not failed and have been updated in the capacity layer is sufficient to repair the erasure code stripe, the second type of failure is that the number of data blocks and check blocks that have not failed and have not been updated in the capacity layer is sufficient to repair the erasure code stripe, and the third type of failure is all other failures that cannot be classified into the first or second type of failures.

[0054] Exemplarily, the first Vandermonde matrix symbol is represented as:

[0055] A=V1(K,K+M)

[0056] The second Vandermonde matrix symbol is expressed as:

[0057] B=V2(K,K)

[0058] The generator matrix calculation formula is:

[0059] C=A*B -1

[0060] Among them, V1 and V2 are Vandermonde matrices constructed using axioms, A is the first Vandermonde matrix, B is the second Vandermonde matrix, C is the generator matrix, K is the data block number parameter of the erasure code, and M is the check block number parameter of the erasure code.

[0061] Exemplarily, the written data is preferentially stored in the performance layer in the form of multiple copies, and is converted into an erasure code storage form in the process of data sinking from the performance layer to the capacity layer.

[0062] The present invention proposes an erasure code data update system for a distributed hybrid storage system, comprising:

[0063] A generating unit, used for constructing a generating matrix according to data block parameters and check block parameters of the erasure code;

[0064] A judgment unit, used to judge whether the valid data of the performance layer constitutes a complete erasure code stripe;

[0065] A first repair unit is used to read the entire stripe data block of the performance layer when the valid data of the performance layer can form a complete erasure code stripe, calculate and generate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair after a failure occurs;

[0066] The second repair unit is used to read the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer when the valid data in the performance layer cannot form a complete erasure code stripe, and use the generation matrix to calculate the new erasure code check block, first write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success, and use a non-full stripe update consistency repair strategy to perform repair after a failure occurs.

[0067] The present invention proposes a method and system for updating erasure code data in a distributed hybrid storage system, including constructing a generation matrix according to the data block parameters and check block parameters of the erasure code; judging whether the valid data of the performance layer constitutes a complete erasure code stripe; if so, reading the entire stripe data block of the performance layer, and calculating and generating a new erasure code check block, overwriting the data block and the check block at one time and writing them to the capacity layer, and using a full stripe update consistency repair strategy to repair them after a failure occurs; if not, reading the old data blocks and all check blocks at the corresponding positions of the performance layer data to be updated and the capacity layer, writing the calculated new erasure code check block and the new data block to the capacity layer, and using a non-full stripe update consistency repair strategy to repair them after a failure occurs. The technical solution of the present invention avoids the additional I / O operations required to maintain the new and old versions of data on the HDD, improves the utilization rate of storage resources, and ensures that consistency can be restored quickly and efficiently when writing fails or data is lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0069] Figure 1 A schematic diagram of a typical distributed hybrid storage system provided by an embodiment of the present invention;

[0070] Figure 2A schematic diagram of partially updating EC data on a HDD with SSD data provided by an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a method for updating erasure code data in a distributed hybrid storage system provided by an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of a method for calculating a generator matrix provided in an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of an erasure code data update process provided by an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of a full stripe update consistency repair process provided by an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of a non-full stripe update consistency repair process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0077] Erasure Coding (EC) is a data redundancy technology used to provide high data reliability while reducing storage overhead. It divides the data into multiple data blocks and check blocks (collectively referred to as coding blocks). When any data block or check block is lost, the data can still be restored through the remaining blocks. The typical form of erasure coding is the k+m mode, where k represents the number of data blocks and m represents the number of check blocks. For example, EC (4+2) means there are 4 data blocks and 2 check blocks. Compared with traditional multi-copy technology, erasure coding can achieve high fault tolerance with lower storage overhead. However, when updating data, erasure coding needs to update the relevant check blocks at the same time, which makes the data consistency problem more complicated.

[0078] like Figure 2As shown in the figure, assume that in a distributed hybrid storage system, the data in the capacity layer is stored in the form of 4+2 erasure codes (EC), where 4 data blocks (D1, D2, D3, D4) and 2 check blocks (P1, P2) together form a stripe. In the process of data sinking from SSD to HDD, some data blocks (such as D1 and D2) may need to be updated. At this time, the check blocks (P1 and P2) must be updated synchronously to maintain the consistency of the stripe. If a failure occurs during the update process, such as power failure or node failure, some data blocks and check blocks (such as D1', D2' and P1') may be updated, while the check block P2 is still the old version. In this case, the data blocks in the stripe do not match the check block version, and the new and old versions are less than k copies, resulting in the system being unable to recover the lost data blocks through the check blocks, which may further cause the risk of data loss. How to ensure the consistency of the erasure code stripe in the process of data sinking from SSD to HDD, especially to quickly restore the consistency when a failure occurs, is a technical problem that needs to be solved urgently.

[0079] The embodiments of the present invention can effectively reduce the implementation complexity by optimizing the update process, reducing unnecessary I / O operations, and improving the utilization of storage resources. Figure 1 , the embodiment of this patent is based on the typical architecture of a distributed hybrid storage system, including a performance layer (SSD) and a capacity layer (HDD). The data written by the user is first stored in the SSD, and high performance and data reliability are guaranteed through multiple copies of storage. In the background task, the data will be converted into an erasure code (EC) storage format during the process of sinking from the SSD to the HDD. It should be noted that the embodiment of this patent only solves the consistency problems that may be encountered in the process of data sinking from the SSD to the HDD, and does not discuss the writing process on the user side, the reading process of EC data, etc.

[0080] Example 1

[0081] Figure 3 A schematic flow chart of a method for updating erasure code data in a distributed hybrid storage system provided by an embodiment of the present invention, the method specifically includes:

[0082] Step S101, constructing a generator matrix according to data block parameters and check block parameters of the erasure code, where the data block parameters include the number of data blocks, and the check block parameters include the number of check blocks;

[0083] Specifically, the embodiment of the present invention generates a first Vandermonde matrix and a second Vandermonde matrix according to the k and m parameters of EC, referring to Figure 4 The generator matrix is ​​obtained by the regular process, and the generator matrix is ​​used to calculate the check block later. The first Vandermonde matrix symbol is represented as:

[0084] A=V1(K,K+M)

[0085] The second Vandermonde matrix symbol is:

[0086] B=V2(K,K)

[0087] The calculation formula of the generated matrix is:

[0088] C=A*B -1

[0089] Among them, V1 and V2 are Vandermonde matrices constructed using axioms, A is the first Vandermonde matrix, B is the second Vandermonde matrix, C is the generator matrix, K is the data block number parameter of the erasure code, and M is the check block number parameter of the erasure code. The number of rows of the first Vandermonde matrix is ​​the number of data blocks, and the number of columns is the sum of the number of data blocks and the number of check blocks. The number of rows and columns of the second Vandermonde matrix are both the number of data blocks; the second Vandermonde inverse matrix is ​​obtained by inverting the second Vandermonde matrix, and the generator matrix can be calculated according to the first Vandermonde matrix and the second Vandermonde inverse matrix.

[0090] Step S102, determining whether the valid data of the performance layer constitutes a complete erasure code stripe;

[0091] If yes, execute step S1031, read the entire stripe data block of the performance layer, and use the generation matrix to calculate the new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair them after a failure occurs;

[0092] Specifically, the data blocks of the entire stripe are read from the SSD, a new EC check block is calculated, and all data blocks and check blocks are overwritten and written to the corresponding storage location on the HDD at one time. During the writing process, if a failure occurs, the system will repair it according to the step strategy described in "Full Stripe Update Consistency Repair" to ensure data consistency.

[0093] If not, execute step S1032, then read the old data blocks and all check blocks at the corresponding positions of the performance layer data to be updated and the capacity layer, and use the generation matrix to calculate the new erasure code check block, first write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success, and use the non-full stripe update consistency repair strategy to repair it after a failure occurs.

[0094] Specifically, if the valid data on the SSD cannot form a complete EC stripe, the new data that needs to be updated is read from the SSD, and the old data blocks and all check blocks at the corresponding position are read from the HDD. New check blocks are calculated based on the new and old data. Subsequently, the new check blocks are overwritten on the HDD, and after ensuring that all check blocks are written successfully, the new data blocks are overwritten. There is no need to rewrite data blocks that have not been modified. If a failure occurs during the writing process, the system will perform repairs according to the step strategy described in "Non-full stripe update consistency repair" to ensure stripe consistency. Please refer to Figure 5 The erasure code data update process.

[0095] When the data on the SSD does not form a complete stripe, it is necessary to update the check block first and then the data block. Compared with the existing technical solutions, on the one hand, it is not necessary to maintain multiple versions of new and old data on the HDD, and the new data can directly overwrite the old data. On the other hand, it is not necessary to write any data to the SSD, and there is no write amplification problem. Through the method flow of the embodiment of the present invention, in most cases, the data can be written without consistency repair.

[0096] The full stripe update consistency repair strategy and the non-full stripe update consistency repair strategy mentioned in the embodiments of the present invention are described in detail below.

[0097] The first is the full stripe update consistency repair strategy. Figure 6 For full stripe updates, the SSD stores the complete EC stripe data blocks in multiple copies. When updating, you only need to read the complete EC stripe data blocks from the SSD, recalculate the check blocks, and overwrite the data blocks and check blocks to the corresponding locations on the HDD.

[0098] The second is the non-full stripe update consistency repair strategy. Figure 7 If the non-full stripe update fails due to a fault, only part of the data in the stripe will be in the SSD, and the EC stripe in the HDD may be partially updated. It is necessary to determine the type of stripe inconsistency, which includes the first type of fault, the second type of fault, and the third type of fault.

[0099] Type 1: If it is the first type of failure, the erasure code stripe is repaired to the new version using all data blocks in the performance layer and data blocks that have not failed in the capacity layer;

[0100] Specifically, this type means that the EC stripe can be repaired to the new version using the surviving new version coding blocks. The surviving new coding blocks here refer to all data blocks in the SSD replica layer, the updated and non-faulty check blocks in the HDD EC layer, and the non-faulty data blocks in the HDD EC layer that are not related to this I / O. If all check blocks have been successfully updated, it must belong to this type.

[0101] Type 2: If it is the second type of failure, the erasure code stripe is repaired to the old version using the data blocks and check blocks in the capacity layer that have not been updated and have not failed;

[0102] Specifically, this type means that the EC stripe can be repaired to the old version using the surviving old version coding blocks. The surviving old coding blocks here refer to the data blocks and check blocks in the HDD EC layer that have not been updated and have not failed.

[0103] Type 3: If it is a third type of failure, a repair matrix for consistency repair is constructed using the new data block and the updated new check block, and the erasure code stripe is repaired to the old version through the repair matrix for consistency repair;

[0104] Specifically, since it is required to write the check block successfully before writing the data block, when the inconsistency type belongs to this type, the data block must not be updated, and the check block is partially updated. Repairing this type of inconsistency requires that the data in the SSD replica layer must exist. First, the conversion check block is calculated using the check blocks in the capacity layer that have not failed and have been updated, the data blocks in the performance layer, and the generation matrix; then, the conversion check block and the check blocks in the capacity layer that have not failed and have not been updated are used together to form a conversion check block list; then, the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair are used to calculate a complete old version of the erasure code stripe; finally, the old version of the erasure code stripe is updated using the new data block in the performance layer to obtain a complete new version of the erasure code stripe, thereby completing the consistency repair. The following embodiment of the present invention uses EC(k+m) to explain the specific method.

[0105] First, let the old code blocks be D1, D2, D3, ....., D k , P1, P2, ...., P m , where D represents data block, P represents check block, Roman numerals and k, m represent the serial number of coding block. Assume that the generator matrix of EC coding is as shown below, the matrix is ​​a fixed value, and is generated by the method of step S101.

[0106] B 11 B 12 B 13 ...B1k

[0107] B 21 B 22 B 23 ...B 2k ....

[0109] B m1 B m2 B m3 ...B mk

[0110] Suppose there are x new data blocks in the SSD (x < k). After an update failure, y parity blocks are successfully updated (y < m). The data in the SSD and HDD are as follows:

[0111] SSD: D a ’, D b ’, D c ’,...D d ’ (a, b, c, d are numbers, and the apostrophe represents new data)

[0112] HDD: D1, D2, D3,....., D k , P1’, P2’,...., P m ’ (The apostrophe of the parity block means it may or may not have been updated).

[0113] Let P i ’ be the new parity block. Then each new parity block is calculated by P i ’ = D a ’ * B ia +.. + D d ’ * B id +.. D u * B iu +.. + D w * B iw It is calculated that, where a...d are the numbers of the new data blocks in the SSD, and u...w are the numbers of the old data blocks in the HDD that have not been updated.

[0114] Move all the terms involving new data blocks on the right side of the above formula to the left side, then we can get P i * = P i ’ - (D a ’ * B ia +.. + D d ’ * B id ) = D u * B iu +.. + D w * B iw, the new data blocks involved in the intermediate terms of this formula are located in the SSD and are known values. The coefficients of the generated matrix are fixed values, so P can be calculated i *. All check blocks are converted in this way, and finally the following matrix equation is obtained, where u..w is the number of the old data block that has not been updated in the HDD.

[0115]

[0116] The above formula represents a new RS coding generator matrix. If the generator matrix is ​​reversible, when m coding blocks are lost, the lost coding blocks can be restored through the surviving coding blocks. The data block obtained after decoding is the old version of the data block. It only needs to be updated with the data in the SSD to obtain the new version of the data block. The generator matrix provided by the embodiment of the present invention can ensure that the generator matrix obtained by the consistency repair conversion in the previous paragraph is reversible when m blocks are lost.

[0117] The embodiment of the present invention proposes a method for repairing data consistency in a hybrid storage system after a failure during writing of erasure codes causes data inconsistency. A consistency repair generation matrix can be constructed based on conditions such as whether the data is alive and whether the data is a new version or an old version, and the generation matrix can be used to retrieve lost data and repair inconsistent data. The beneficial effects of the embodiment of the present invention include:

[0118] (1) I / O overhead is significantly reduced. The additional I / O operations required to maintain the old and new versions of data on the HDD are avoided, and new data blocks and check blocks can be directly overwritten, reducing the burden of metadata management and recycling old versions of data. In addition, there is no need to write any data to the SSD, which greatly alleviates the problem of I / O amplification and reduces resource usage and performance consumption during data migration.

[0119] (2) Improved storage resource utilization. The performance characteristics of SSDs and HDDs are fully utilized to avoid writing multiple copies to SSDs. The efficient erasure code update process significantly improves the storage resource utilization of SSDs and HDDs. At the same time, there is no need to retain old versions of data and other metadata in HDDs and SSDs, further improving the overall efficiency of system storage resources.

[0120] (3) The repair process is efficient and does not rely on the underlying system. The overall process design is simple, and the processing paths for full-stripe updates and non-full-stripe updates are clearly defined, avoiding the need to manage and maintain complex multi-version data. In the consistency repair process, efficient repair based on erasure codes is achieved through matrix conversion calculations, reducing dependence on the underlying storage system, making it more operational in actual deployment, and reducing implementation difficulty and maintenance costs.

[0121] (4) Improved system reliability. In both full-stripe and non-full-stripe update scenarios, the embodiments of the present invention provide a clear fault repair process to ensure that consistency can be quickly and efficiently restored when write failures or data loss occur. In particular, for non-full-stripe update scenarios, by designing a reversible consistency repair generation matrix and mathematical transformation, it is ensured that data blocks and check blocks can still be restored in the event of a partial update that causes inconsistency and the loss of m coding blocks, thereby effectively improving the reliability of the system.

[0122] Example 2

[0123] A distributed hybrid storage system erasure code data update system, comprising:

[0124] A generating unit, used for constructing a generating matrix according to data block parameters and check block parameters of the erasure code;

[0125] A judgment unit, used to judge whether the valid data of the performance layer constitutes a complete erasure code stripe;

[0126] The first repair unit is used to read the entire stripe data block of the performance layer when the valid data of the performance layer can form a complete erasure code stripe, calculate and generate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair after a failure occurs;

[0127] The second repair unit is used to read the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer when the valid data in the performance layer cannot form a complete erasure code stripe, write the calculated new erasure code check blocks and new data blocks to the capacity layer, and use a non-full stripe update consistency repair strategy for repair after a failure occurs.

[0128] It can be understood that the above-mentioned erasure code data updating system of a distributed hybrid storage system corresponds to the erasure code data updating method of a distributed hybrid storage system in Embodiment 1. Any optional items in Embodiment 1 are also applicable to this embodiment and will not be described in detail here.

[0129] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for updating erasure code data in a distributed hybrid storage system, characterized in that: include: Constructing a generator matrix according to the data block parameters and the check block parameters of the erasure code; Determine whether the valid data in the performance layer constitutes a complete erasure code stripe; If yes, read the entire stripe data block of the performance layer, and use the generation matrix to calculate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair them after a failure occurs; If not, read the old data blocks and all check blocks at the corresponding positions of the performance layer's data to be updated and the capacity layer, and use the generation matrix to calculate the new erasure code check block. First write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success. After a failure occurs, use the non-full-stripe update consistency repair strategy to repair it.

2. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The data block parameters include the number of data blocks, the check block parameters include the number of check blocks, and constructing a generator matrix according to the data block parameters and the check block parameters of the erasure code includes: Constructing a first Vandermonde matrix and a second Vandermonde matrix, wherein the number of rows of the first Vandermonde matrix is ​​the number of data blocks, the number of columns is the sum of the number of data blocks and the number of check blocks, and the number of rows and columns of the second Vandermonde matrix are both the number of data blocks; Inverting the second Vandermonde matrix to obtain a second Vandermonde inverse matrix; The generator matrix is ​​obtained by multiplying the first Vandermonde matrix and the second Vandermonde inverse matrix.

3. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The method of using a full stripe update consistency repair strategy to repair a failure after the failure occurs includes: Storing complete erasure coded stripes in multiple copies at the performance layer; The full-stripe new data block is read from the performance layer, and the new full-stripe new data block and the generation matrix are used to calculate the new check block, and the new data block and the new check block are used to overwrite the old data block and the old check block on the capacity layer to complete the full-stripe update consistency repair.

4. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The step of reading the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer, and writing the calculated new erasure code check blocks and the new data blocks into the capacity layer comprises: Reading new data blocks that need to be updated from the performance layer, and synchronously reading old data blocks and old check blocks at corresponding positions from the capacity layer; The generation matrix is ​​used to calculate a new check block, and the new check block is overwritten and written into the capacity layer. After all new check blocks are written successfully, new data blocks are overwritten and written, and unmodified data blocks are retained.

5. The method for updating erasure code data in a distributed hybrid storage system according to claim 4, characterized in that: The repairing by using a non-full stripe update consistency repair strategy after a failure occurs includes: Determine the failure type of the non-full stripe update inconsistency, where the failure type includes a first failure type, a second failure type, and a third failure type; If it is a first type of failure, the erasure code stripe is repaired to a new version using all data blocks in the performance layer and data blocks and check blocks that have not failed and have been updated in the capacity layer; If it is a second type of failure, the erasure code stripe is repaired to the old version by using data blocks and check blocks in the capacity layer that have not failed and have not been updated; If it is a third type of failure, a repair matrix for consistency repair is constructed using the new data block, the updated new check block and the generation matrix. The erasure code stripe is repaired to a consistent old version through the consistency repair repair matrix, and the old version of the erasure code stripe is further updated to the new version using the performance layer data.

6. The method for updating erasure code data in a distributed hybrid storage system according to claim 5, characterized in that: Repairing the erasure code stripe to a new version by using the repair matrix of the consistency repair includes: The conversion check block is calculated by using the check blocks in the capacity layer that have not failed and have been updated, the data blocks in the performance layer and the generation matrix; The conversion check block and the check blocks in the capacity layer that have not failed and have not been updated together form a conversion check block list; Using the data blocks in the capacity layer that have not failed and have not been updated, the conversion check block list, and the repair matrix for consistency repair, a complete old version erasure code stripe is calculated; The old version erasure code stripe is updated using the new data block in the performance layer to obtain a complete new version erasure code stripe, thereby completing consistency repair.

7. The method for updating erasure code data in a distributed hybrid storage system according to claim 6, characterized in that: The repair matrix expression for consistency repair is as follows: in, is an element in the conversion check block list, u, w are the numbers of the old data blocks that have not been updated in the capacity layer, B is the generation matrix, D is the data block, k is the total number of old data block numbers in the capacity layer, and i is the element number of the repair matrix.

8. The method for updating erasure code data in a distributed hybrid storage system according to claim 7, characterized in that: The first type of failure is that the number of data blocks and check blocks that have not failed and have been updated in the performance layer and the capacity layer is sufficient to repair the erasure code stripe; the second type of failure is that the number of data blocks and check blocks that have not failed and have not been updated in the capacity layer is sufficient to repair the erasure code stripe; the third type of failure is all other failures that cannot be classified into the first or second type of failures.

9. The method for updating erasure code data in a distributed hybrid storage system according to claim 1, characterized in that: The written data is stored in the performance layer in the form of multiple copies, and is converted into erasure code storage when the data is transferred from the performance layer to the capacity layer.

10. A distributed hybrid storage system erasure code data update system, characterized in that: include: A generating unit, used for constructing a generating matrix according to data block parameters and check block parameters of the erasure code; A judgment unit, used to judge whether the valid data of the performance layer constitutes a complete erasure code stripe; A first repair unit is used to read the entire stripe data block of the performance layer when the valid data of the performance layer can form a complete erasure code stripe, calculate and generate a new erasure code check block, overwrite the data block and the check block to the capacity layer at one time, and use the full stripe update consistency repair strategy to repair after a failure occurs; The second repair unit is used to read the data to be updated in the performance layer and the old data blocks and all check blocks at the corresponding positions of the capacity layer when the valid data in the performance layer cannot form a complete erasure code stripe, and use the generation matrix to calculate the new erasure code check block, first write the calculated new erasure code check block to the capacity layer, and then write the new data block to the capacity layer after success, and use a non-full stripe update consistency repair strategy to perform repair after a failure occurs.

Images