A file length negotiation method for EC replica hybrid redundant storage
Through the file length negotiation method, the problem of CPU overhead and data recovery difficulties in EC storage mode when writing data is solved, and the determination of effective data length and data consistency negotiation are realized, which simplifies the process and improves reliability.
Patent Information
- Application Number
- CN202311722353.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-14
AI Technical Summary
EC storage method causes excessive CPU overhead when data is written, and it is difficult to determine the effective data length when node downtime or disk data is lost, resulting in difficulty in data recovery.
The file length negotiation method is adopted, and the valid data length is determined by setting prerequisites, IO writes in multiple times, and the EC chunk is sealed after the write is completed. The LC_len and prev_lc_len are sorted and traversed by shard checking to determine the effective data length.
It effectively avoids data loss and overhead caused by traditional consistency protocols, simplifies the data consistency negotiation process, and improves the reliability of data recovery.
Smart Images

Figure CN117851124B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of cloud computing and storage, and in particular to a file length negotiation method for EC replica hybrid redundant storage. Background Art
[0002] EC storage is an erasure code storage method, which is a cross-node data protection method that can group data into multiple units and generate additional redundant units for storage so that it can be restored through redundant data when some data is lost or damaged. However, when using EC storage, each time data is written, the corresponding check unit will be recalculated, causing excessive CPU overhead. In addition, when encountering node crashes, disk data loss, etc., it is impossible to determine the effective data length that has been written to the system, making data recovery difficult. Summary of the invention
[0003] The object of the present invention is to provide a file length negotiation method for EC replica hybrid redundant storage to solve the problems raised in the above background technology.
[0004] To achieve the above object, the present invention provides the following technical solution: a file length negotiation method for EC replica hybrid redundant storage, the negotiation method comprising the following steps;
[0005] The first step is to set the prerequisites, which include the chunk data writing method, the information carried by the parameters when writing chunk data, the chunk data writing failure processing, the data writing success conditions and the negotiation participation shard number setting;
[0006] The second step is to write the chunk data in multiple IO operations, where the chunk data writing includes data slice writing and verification data writing;
[0007] The third step is to seal the EC chunk after writing is completed and negotiate its chunk length. The length negotiation is performed by sorting the given lc_len and prev_lc_len and traversing the shards to check and finally determine the valid data length.
[0008] Preferably, in the first step, the chunk data is written only by appending and when a write offset error occurs, writing is prohibited and an error is returned. The information carried by the parameters when writing chunk data includes the target logical length, the logical length of the node record data, and the logical length of the last IO.
[0009] Preferably, the chunk data write failure processing in the first step includes allowing the current chunk to be written again after an IO fails during the chunk writing process, and the data write success condition includes returning success if all nodes have successfully written data, and the write operation fails if any node fails to write.
[0010] Preferably, the negotiation of setting the number of participating shards in the first step includes negotiating that the number of participating shards is greater than or equal to the number of data shards in the EC policy.
[0011] Preferably, the specific operation of performing IO writing of chunk data in multiple times in the second step includes the following steps:
[0012] A1, divide the original data into several chunks of equal or unequal lengths, and encode each chunk;
[0013] A2, select a storage node that contains all data information after encoding, use this node as the data writing node, and perform the first writing;
[0014] A3, stores or writes the encoded chunk data to other storage nodes to prevent single point failure;
[0015] A4, performs data verification and calculation;
[0016] A5,periodically checks each storage node and repairs or re-encodes data when errors are found.
[0017] Preferably, in the third step, a seal operation is performed on the EC chunk after writing is completed, which specifically includes the following steps:
[0018] B1, verifying data integrity, wherein the integrity verification method includes calculating a checksum and a hash value;
[0019] B2, determining a seal mode, wherein the seal mode includes marking as read-only, writing a protection flag, freezing data, and forcing a read-only mode;
[0020] B3, update metadata, the metadata includes the location, status and attributes of the data, and after the metadata is updated, the status of the EC chunk is reflected as sealed;
[0021] B4, data protection and backup, data protection and backup of the sealed EC chunk.
[0022] Preferably, the chunk length negotiation operation in the third step specifically includes the following steps:
[0023] C1, get the lc_len and prev_lc_len of all current shards and sort them from large to small;
[0024] C2, take out the first length in the sequence, traverse all nodes, and determine whether the current data of the node meets the length requirement. When all node data meet the length requirement, the length is the final negotiated length. If there is a node data that does not meet the length requirement, select the next length requirement in the sequence for judgment again. If it still does not meet the requirement, repeat the operation until all node data meet the selected length requirement.
[0025] C3: If all lengths cannot be met, the final negotiated length output is 0.
[0026] Preferably, the sorting operation in C1 includes the following steps:
[0027] D1, obtain lc_len and prev_lc_len of all shards;
[0028] D2, create data structures and use appropriate data structures to store the lc_len and prev_lc_len values of each shard;
[0029] D3, select a sorting algorithm to sort in descending order;
[0030] D4, obtain the sorting results.
[0031] Preferably, the operation of traversing all nodes in C2 includes the following steps:
[0032] E1, obtain the sorted sequence structure;
[0033] E2, determining a traversal form, wherein the traversal form includes linear traversal, recursive traversal and breadth-first traversal;
[0034] E3, perform traversal operation;
[0035] E4, setting a termination condition, wherein the termination condition includes that all node data meet the selection length requirement;
[0036] E5, releasing resources, wherein releasing resources includes closing file handles and reclaiming memory.
[0037] Preferably, whether the current data of the node in C2 meets the length requirement refers to whether the node has all the data that the node should have under the length.
[0038] Technical effects and advantages of the present invention:
[0039] The present invention utilizes the logical length information carried by each IO, identifies the target logical length of the IO through lc_len, and determines the final chunk length through negotiation. The consistency negotiation can be completed through the information carried by each IO, effectively avoiding data loss and the overhead caused by using traditional consistency protocols. The technology is simple and reliable. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic diagram of the overall method flow of the present invention.
[0041] Figure 2 This is a schematic diagram of data storage in each node after the first IO writing is completed in the present invention.
[0042] Figure 3 This is a schematic diagram of data storage in each node after the second IO writing is completed in the present invention.
[0043] Figure 4 This is a schematic diagram of data storage in each node after the third IO write failure of the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] The present invention provides Figure 1-4 A file length negotiation method for EC replica hybrid redundant storage is shown, and the negotiation method includes the following steps:
[0046] The first step is to set the prerequisites, which include the chunk data writing method, the information carried by the parameters when writing chunk data, the chunk data writing failure processing, the data writing success conditions, and the negotiation and participation in the number of shards.
[0047] The second step is to write the chunk data in multiple IO operations. Chunk data writing includes data slice writing and verification data writing.
[0048] The third step is to seal the EC chunk after writing is completed and negotiate its chunk length. The length negotiation is done by sorting the given lc_len and prev_lc_len and traversing the shards to check and finally determine the effective data length.
[0049] In the first step, chunk data can only be written by appending, and when the write offset is wrong, writing is prohibited and an error is returned. When writing chunk data, the information carried by the parameters includes the target logical length, the logical length of the node record data, and the logical length of the last IO.
[0050] Furthermore, the target logical length refers to the specific length to which the data is compressed, which is usually used to control the size of data transmission or the compression algorithm, while the logical length of the node recording data refers to the length to which the node converts the data into a logical length and records it when recording or storing data, which is used to measure the size of the data or the storage space occupied. The logical length of the last IO refers to the logical length of the data read or written in the last I / O operation, which is used to indicate the size of the data during transmission, storage or processing.
[0051] In the first step, the chunk data writing failure processing includes allowing the current chunk to be written again after an IO fails during the chunk writing process. The data writing success condition includes the successful return of data writing success on all nodes. If any node fails to write, the write operation fails.
[0052] The negotiation of the number of participating shards in the first step includes negotiating that the number of participating shards is greater than or equal to the number of data shards in the EC policy.
[0053] Furthermore, negotiating that the number of participating shards is greater than or equal to the number of data shards in the EC policy means dividing the data into multiple data shards and generating a certain number of redundant shards to achieve data recovery when some shards are lost or damaged. The number of data shards and the number of redundant shards are set to meet fault tolerance requirements and avoid insufficient nodes participating in error correction encoding and decoding operations in actual data recovery operations.
[0054] In the second step, the specific operation of writing chunk data in multiple IOs includes the following steps:
[0055] A1, divide the original data into several chunks of equal or unequal lengths, and encode each chunk;
[0056] A2, select a storage node that contains all data information after encoding, use this node as the data writing node, and perform the first writing;
[0057] A3, stores or writes the encoded chunk data to other storage nodes to prevent single point failure;
[0058] A4, performs data verification and calculation;
[0059] A5,periodically checks each storage node and repairs or re-encodes data when errors are found.
[0060] Embodiment 1:
[0061] Taking EC4+2 and a unit size of 4K as an example, the possible situation of writing chunk data in multiple IOs is described, and the changes of lc_len and prev_lc_len are explained;
[0062] S1, write 0K~12K data, all written successfully;
[0063] Write 0K~12K slices to nodes 1~3 respectively, and then write verification data to nodes 5~6. After IO is completed, each node stores data as follows Figure 2 As shown, further, 0K~12K data is divided into three parts and written into nodes 1, 2, and 3 respectively, and then the same copy data is written into check nodes 5 and 6 respectively. At this time, the lc_len recorded in nodes 1, 2, 3, 5, and 6 is 12K.
[0064] S2, write 12K~14K data, write successfully;
[0065] 12K~14K total 2K data will be written into nodes 4, 5, and 6 respectively. The structure after writing is as follows Figure 3 As shown, at this time, the lc_len of nodes 4, 5, and 6 is updated to 14K, the prev_lc_len of nodes 5 and 6 is updated to the previous logical length of 12K, and the other shards remain unchanged;
[0066] S3, writes 14K to 23K of data, of which nodes 1 and 5 fail to write;
[0067] 14K~23K, a total of 9K data, will be written to nodes 1, 2, 4, 5, and 6 respectively. The IO of nodes 1 and 5 will fail. The final result is as follows Figure 4 As shown, the write failure status of 1 and 5 remains unchanged, node 2 writes 20K to 23K of data and updates lc_len, node 4 writes 14K to 16K of data and updates lc_len, and after shard 6 is full of 16K, it first calculates the verification data of 0K to 16K, deletes the original copy data, writes the verification data, and then writes the copy data of 16K to 23K that is less than 16K, and updates lc_len.
[0068] In the third step, after writing is completed, the EC chunk is sealed, which includes the following steps:
[0069] B1, verify data integrity. The integrity verification method includes calculating the checksum and hash value;
[0070] Furthermore, if any data corruption or error is found, it needs to be repaired or re-encoded accordingly.
[0071] B2, determine the seal mode, which includes marking as read-only, writing protection flag, freezing data and forcing read-only mode;
[0072] Furthermore, marking as read-only means that the chunk cannot be modified after it is sealed, writing the protection flag means setting a protection flag to prevent further modification of the data, and the protection flag is a special flag bit or attribute used to indicate that the chunk has been sealed and no write operations are allowed, freezing data means "freezing" or "locking" the data to prevent any modification to it, and forcing read-only mode means putting the entire EC storage system in read-only mode, so that all write operations are prohibited, including modifications to existing data.
[0073] B3, update metadata, which includes the location, status and attributes of the data. After the metadata is updated, it reflects that the status of the ECchunk is archived;
[0074] B4, data protection and backup, data protection and backup of the sealed EC chunk.
[0075] Furthermore, data protection and backup include copying data to other storage nodes or storage media to improve data reliability and recoverability.
[0076] The chunk length negotiation operation in the third step specifically includes the following steps:
[0077] C1, get the lc_len and prev_lc_len of all current shards and sort them from large to small;
[0078] C2, take out the first length in the sequence, traverse all nodes, and determine whether the current data of the node meets the length requirement. When all node data meet the length requirement, the length is the final negotiated length. If there is a node data that does not meet the length requirement, select the next length requirement in the sequence for judgment again. If it still does not meet the requirement, repeat the operation until all node data meet the selected length requirement.
[0079] C3: If all lengths cannot be met, the final negotiated length output is 0.
[0080] The sorting operation in C1 includes the following steps;
[0081] D1, obtain lc_len and prev_lc_len of all shards;
[0082] D2, create data structures and use appropriate data structures to store the lc_len and prev_lc_len values of each shard;
[0083] D3, select a sorting algorithm to sort in descending order;
[0084] D4, obtain the sorting results.
[0085] The operation of traversing all nodes in C2 includes the following steps:
[0086] E1, obtain the sorted sequence structure;
[0087] E2, determine the traversal form, which includes linear traversal, recursive traversal and breadth-first traversal;
[0088] E3, perform traversal operation;
[0089] E4, set the termination condition, which includes that all node data meet the selection length requirement;
[0090] E5, release resources. Releasing resources includes closing file handles and reclaiming memory.
[0091] Whether the current data of the node in C2 meets the length requirement refers to whether the node has all the data that the node should have under the length.
[0092] Further, according to Figure 4 As shown, for a length of 23K, node 1 should have 8K data with offsets of 0K~4K and 16K~20K if data is not lost, but node 1 only stores 4K data with offsets of 0K~4K, so node 1 does not meet the length of 23K. Similarly, for node 2, it meets the length of 23K and any length between 0K~23K. Node 5 should store 4K of verification data with offsets of 0K~16K, plus the original copy of the data with offsets of 16K~23K, totaling 11K data, but currently there is only the original copy of the data with offsets of 0K~14K, totaling 14K data, so node 5 also does not meet the length of 23K.
[0093] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A file length negotiation method for EC replica hybrid redundant storage, characterized in that: The negotiation method comprises the following steps: The first step is to set the prerequisites, which include the chunk data writing method, the information carried by the parameters when writing chunk data, the chunk data writing failure processing, the data writing success conditions and the negotiation participation shard number setting; The second step is to write the chunk data in multiple IO operations, where the chunk data writing includes data slice writing and verification data writing; The third step is to perform a seal operation on the EC chunk after writing is completed, and negotiate its chunk length. The length negotiation is performed by sorting the given lc_len and prev_lc_len and traversing the shards to check, and finally determine the valid data length; In the first step, the chunk data is written in a way that it can only be written in an append write mode and when a write offset error occurs, writing is prohibited and an error is returned. The information carried by the parameters when writing chunk data includes the target logical length, the logical length of the node record data, and the logical length of the last IO; The specific operation of writing the chunk data in multiple IOs in the second step includes the following steps: A1, divide the original data into several chunks of equal or unequal lengths, and encode each chunk; A2, select a storage node that contains all data information after encoding, use this node as the data writing node, and perform the first writing; A3, stores or writes the encoded chunk data to other storage nodes to prevent single point failure; A4, performs data verification and calculation; A5, periodically checks each storage node and repairs or re-encodes data when errors are found; The chunk length negotiation operation in the third step specifically includes the following steps: C1, get the lc_len and prev_lc_len of all current shards and sort them from large to small; C2, take out the first length in the sequence, traverse all nodes, and determine whether the current data of the node meets the length requirement. When all node data meet the length requirement, the length is the final negotiated length. If there is a node data that does not meet the length requirement, select the next length requirement in the sequence for judgment again. If it still does not meet the requirement, repeat the operation until all node data meet the selected length requirement. C3: If all lengths cannot be met, the final negotiated length output is 0; In the second step, the specific operation of writing chunk data in multiple IOs includes the following steps: A1, divide the original data into several chunks of equal or unequal lengths, and encode each chunk; A2, select a storage node that contains all data information after encoding, use this node as the data writing node, and perform the first writing; A3, stores or writes the encoded chunk data to other storage nodes to prevent single point failure; A4, performs data verification and calculation; A5,periodically checks each storage node and repairs or re-encodes data when errors are found.
2. According to claim 1, a file length negotiation method for EC replica hybrid redundant storage is characterized in that: The chunk data write failure processing in the first step includes allowing the current chunk to be written again after an IO fails during the chunk writing process. The data write success condition includes returning success if all nodes have written data successfully. If any node fails to write, the write operation fails.
3. According to claim 1, a file length negotiation method for EC replica hybrid redundant storage is characterized in that: The negotiation of the number of participating shards in the first step includes the negotiation that the number of participating shards is greater than or equal to the number of data shards in the EC policy.
4. According to claim 1, a file length negotiation method for EC replica hybrid redundant storage is characterized in that: In the third step, after writing is completed, the EC chunk is sealed, which specifically includes the following steps: B1, verifying data integrity, wherein the integrity verification method includes calculating a checksum and a hash value; B2, determining a seal mode, wherein the seal mode includes marking as read-only, writing a protection flag, freezing data, and forcing a read-only mode; B3, updating metadata, where the metadata includes the location, status and attributes of the data, and after the metadata is updated, the status of the ECchunk is reflected as being sealed; B4, data protection and backup, data protection and backup of the sealed EC chunk.
5. The file length negotiation method for EC replica hybrid redundant storage according to claim 1, characterized in that: The chunk length negotiation operation in the third step specifically includes the following steps: C1, get the lc_len and prev_lc_len of all current shards and sort them from large to small; C2, take out the first length in the sequence, traverse all nodes, and determine whether the current data of the node meets the length requirement. When all node data meet the length requirement, the length is the final negotiated length. If there is a node data that does not meet the length requirement, select the next length requirement in the sequence for judgment again. If it still does not meet the requirement, repeat the operation until all node data meet the selected length requirement. C3: If all lengths cannot be met, the final negotiated length output is 0.
6. A file length negotiation method for EC replica hybrid redundant storage according to claim 5, characterized in that: The operation of traversing all nodes in C2 includes the following steps: E1, obtain the sorted sequence structure; E2, determining a traversal form, wherein the traversal form includes linear traversal, recursive traversal and breadth-first traversal; E3, perform traversal operation; E4, setting a termination condition, wherein the termination condition includes that all node data meet the selection length requirement; E5, releasing resources, wherein releasing resources includes closing file handles and reclaiming memory.
7. A file length negotiation method for EC replica hybrid redundant storage according to claim 6, characterized in that: Whether the current data of the node in C2 meets the length requirement refers to whether the node has all the data that the node should have under the length.
