Data redundancy protection method for adaptive switching of erasure codes and multiple copies
By adaptively switching between erasure coding and multi-replica modes and selecting appropriate protection strategies based on data segment types, the problem of low storage and update efficiency in existing technologies is solved, achieving efficient data redundancy protection.
Patent Information
- Application Number
- CN202511911618.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-24
AI Technical Summary
Existing single data redundancy protection strategies have low storage and update efficiency. Multi-copy technology has low storage efficiency and high cost, while erasure coding has high computational complexity in append write and update scenarios.
An adaptive switching method between erasure coding and multiple replicas is adopted. Based on the length of the data segment to be processed and the operation type, it is determined whether it is a standard data segment. If it is, erasure coding mode is used for encoding; otherwise, multiple replica mode is used for redundancy. Combining the advantages of erasure coding and multiple replicas, storage utilization is improved and operation complexity is reduced.
While ensuring high storage efficiency, the adaptive switching strategy reduces the complexity of frequent operations, improves the efficiency of data updates, and avoids high CPU computing costs.
Smart Images
Figure CN121918758A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed storage technology, and in particular to a data redundancy protection method using erasure coding and adaptive switching of multiple replicas. Background Technology
[0002] Distributed Storage Systems (DSS) are a technological field resulting from the combination of storage and network technologies. They connect dispersed storage nodes through network communication to build a unified storage space with high reliability and availability, capable of storing massive amounts of data. Data redundancy, as a crucial means of ensuring data reliability and availability, is widely used in various distributed storage systems. Data redundancy protection strategies can be divided into two main categories: replication and erasure codes. Replication schemes copy the original data file into n copies and distribute them to different network nodes; erasure codes first divide the original data into k original blocks, then encode these original blocks into (m=nk) encoded blocks, resulting in a total of n data blocks, which are then forwarded to different network nodes. Replication technology is widely used in numerous distributed storage systems due to its simplicity and ease of use; while erasure codes, due to their technological advantages and design flexibility, have also received increasing attention from industry and academia and have seen some applications, but the following problems still exist:
[0003] Multiple copy technology is relatively simple and efficient for appending and updating files, but because it stores multiple copies of the same data, it suffers from low storage efficiency and high storage costs, making it suitable for small files and small-scale storage. Erasure coding has higher storage efficiency and is suitable for large file storage. However, erasure coding involves more operations in append and update scenarios: reading, encoding, and writing data, requiring more complex calculations and more data transfer, resulting in higher costs. Summary of the Invention
[0004] The purpose of this application is to provide a data redundancy protection method using erasure coding and adaptive switching of multiple replicas, to solve the technical problems of low storage and update efficiency in existing single data redundancy protection strategies. The various technical effects of the preferred solutions among the many technical solutions provided in this application are detailed below.
[0005] To achieve the above objectives, this application provides the following technical solutions:
[0006] This application provides a data redundancy protection method using erasure coding and adaptive switching of multiple replicas, comprising: dividing a file to be processed into several data segments of a preset length, wherein data segments reaching the preset length are standard data segments, and data segments not reaching the preset length are non-standard data segments; determining the data segment to be encoded based on the operation type of each data segment, wherein the operation type includes append write operation and update operation, determining whether the data segment to be encoded is the standard data segment, if it is the standard data segment, encoding it using erasure coding mode, and if it is the non-standard data segment, using multiple replica mode for redundancy, thereby obtaining a data block list; assembling the data in the data block list to obtain an assembly package, sending the assembly package to a storage pool based on a storage node list, and storing the data in the assembly package in the storage nodes corresponding to the storage pool.
[0007] In some embodiments, if the data segment to be processed is the standard data segment and the operation type is the append write operation, the method includes: determining whether the data segment to be encoded is full and becomes the standard data segment based on the write offset and length of the data to be written; if it is full, encoding is performed according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is not full, a data block list including 1 original block and m replica blocks is constructed by data copying.
[0008] In some embodiments, if the data segment to be processed is the non-standard data segment, and the operation type is the append write operation, the method includes: combining the data of the data segment to be processed with the data to be written to obtain the data segment to be encoded; determining whether the data segment to be encoded is the standard data segment; if it is the standard data segment, encoding the standard data segment according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is the non-standard data segment, constructing a data block list including 1 original block and m replica blocks by data replication.
[0009] In some embodiments, if the data segment to be processed is the standard data segment, and the operation type is the update operation, the method includes: obtaining a new original block list based on the starting position and update length of the data to be updated in the data segment to be processed; reading the corresponding old original block list from the storage nodes based on the storage node list; performing difference calculation and encoding based on the new original block list and the old original block list to obtain an coded block increment list; and writing the new original block list and the coded block increment list together into multiple storage nodes.
[0010] In some embodiments, if the data segment to be processed is the non-standard data segment, and the operation type is the update operation, the method includes: combining the data of the data segment to be processed with the data to be updated to obtain the data segment to be encoded; determining whether the data segment to be encoded is the standard data segment; if it is the standard data segment, encoding the standard data segment according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is the non-standard data segment, constructing a data block list including 1 original block and m replica blocks by data replication.
[0011] In some embodiments, the operation type further includes a read operation. If the data segment to be processed is the standard data segment, the operation type is the read operation. The method includes: querying the read cache table for a cache record based on the file identifier and the data segment identifier; if the cache record exists, reading it directly; if the cache record does not exist, reading at least k raw blocks from the storage node list, decoding them to obtain the standard data segment, and reading data from the standard data segment.
[0012] In some embodiments, if the data segment to be processed is the non-standard data segment and the operation type is the read operation, the method includes: querying the read cache table for a cache record based on the file identifier and the data segment identifier; if the cache record exists and the data to be read is within the cache range, then reading directly; if the cache record exists but the data to be read exceeds the cache range, then reading at least one data block from the storage node list to obtain the non-standard data segment, and then reading data from the non-standard data segment; if the cache record does not exist, then reading at least one data block of the corresponding range data from the storage node list based on the offset and length of the read requirement.
[0013] In some embodiments, if the append write operation fails, the method further includes: comparing the number of storage nodes that successfully wrote the data with the number of original blocks; if the number of storage nodes that successfully wrote the data is less than the number of original blocks, then notifying the storage nodes corresponding to the successful write, server failure, and timeout to delete the data; if the number of storage nodes that successfully wrote the data is greater than or equal to the number of original blocks, then recording the exception information in the exception record table as a basis for asynchronous repair.
[0014] In some embodiments, if the update operation fails, the method further includes: comparing the number of storage nodes that successfully wrote the data with the number of original blocks; if the number of storage nodes that successfully wrote the data is less than the number of original blocks, then notifying the storage nodes corresponding to the successful write, server failure, and timeout to perform a rollback repair operation; if the number of storage nodes that successfully wrote the data is greater than or equal to the number of original blocks, then determining whether it is the non-standard data segment; if not, then using the erasure coding mode to perform encoding calculation repair; if so, then using the multi-replica mode to perform replication repair.
[0015] In some embodiments, the method further includes: dividing the file namespace into multiple hash subspaces, mapping the data segment of each file to the corresponding hash subspace through hash calculation, each hash subspace corresponding to a storage node list, and a plurality of hash subspaces and corresponding storage node lists constituting a hash subspace mapping table; calculating a hash identifier based on the file identifier and data segment identifier of the file to be processed, and obtaining the storage node list corresponding to the hash identifier from the hash subspace mapping table.
[0016] Implementing one of the above-described technical solutions of this application has the following advantages or beneficial effects: In this application, the file to be processed is first divided into multiple data segments according to a preset length. It is then determined whether the data segment to be encoded is a standard data segment or a non-standard data segment. If it is a standard data segment, it is processed using erasure coding mode; if it is a non-standard data segment, it is processed using multi-copy mode. In this case, this application uses erasure coding mode for standard data segments that do not require frequent operations, which can improve storage utilization. For non-standard data segments that require frequent operations, multi-copy mode is used to avoid high CPU computation costs caused by frequent encoding and decoding operations, thus reducing operational complexity. The embodiments of this application integrate the advantages of multi-copy and encoding through adaptive switching between erasure coding mode and multi-copy mode, overcoming their respective disadvantages and ensuring efficient data updates under the premise of high storage efficiency. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0018] Figure 1 This is a flowchart illustrating the data redundancy protection method of erasure coding and adaptive switching of multiple copies according to an embodiment of this application.
[0019] Figure 2 This is a schematic diagram of the overall system architecture of an embodiment of this application.
[0020] Figure 3 This is a schematic diagram of the data encoding and storage process according to an embodiment of this application.
[0021] Figure 4 This is a schematic diagram of the hash subspace mapping process in an embodiment of this application.
[0022] Figure 5 This is a schematic diagram of the data operation types in an embodiment of this application.
[0023] Figure 6 This is a flowchart illustrating the append-write operation of a standard data segment according to an embodiment of this application.
[0024] Figure 7 This is a flowchart illustrating the appending and writing operation of a non-standard data segment according to an embodiment of this application.
[0025] Figure 8 This is a flowchart illustrating the update operation of a standard data segment according to an embodiment of this application.
[0026] Figure 9 This is a flowchart illustrating the update operation of a non-standard data segment according to an embodiment of this application.
[0027] Figure 10 This is a flowchart illustrating the reading operation of a standard data segment according to an embodiment of this application.
[0028] Figure 11 This is a flowchart illustrating the reading operation of a non-standard data segment according to an embodiment of this application. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this application clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments and depict various exemplary embodiments that may be adopted to implement this application. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of this application disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of this application.
[0030] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0031] To illustrate the technical solutions described in this application, specific embodiments are provided below, showing only the parts related to the embodiments of this application.
[0032] The erasure coding and multi-replica adaptive switching data redundancy protection method involved in the embodiments of this application can be applied to scenarios where data is stored on several storage nodes of a distributed storage system, and can improve storage efficiency while ensuring maximum storage utilization.
[0033] like Figure 1 As shown, this application provides a data redundancy protection method with erasure coding and adaptive switching of multiple replicas, including the following steps (steps S1 to S3):
[0034] S1. Divide the file to be processed into several data segments according to a preset length. The data segments that reach the preset length are standard data segments, and the data segments that do not reach the preset length are non-standard data segments.
[0035] Specifically, each file includes a file identifier (fileid), each file can be divided into multiple data segments, each data segment includes a data segment identifier (segmentid), and each data segment can be further divided into multiple data blocks, each data block also includes a corresponding block identifier (blockid). The preset length (segmentsize) refers to a pre-set data segment length. When dividing a file into several data segments to be processed, there may be a non-standard data segment at the end that differs from the preset length.
[0036] S2. Based on the operation type of each data segment to be processed, including append and update operations, determine the data segment to be encoded, and determine whether the data segment to be encoded is a standard data segment. If it is a standard data segment, use erasure coding mode for encoding; if it is a non-standard data segment, use multi-replica mode for redundancy, and obtain a list of data blocks. For example... Figure 2 As shown, data segments can be processed based on the erasure coding mode and the multiple copy mode of the encoding module.
[0037] Specifically, such as Figure 3 As shown, an append operation can refer to adding a standard data segment or a non-standard data segment to the end of a standard data segment, or appending data to the end of a non-standard data segment; an update operation can refer to updating data within a standard data segment, or updating data within a non-standard data segment. Further, depending on the data segment to be processed, the append operation of a standard data segment in this embodiment can be represented as PUT_SEGMENT, the append operation of a non-standard data segment can be represented as AWRITE_SEGMENT, the update operation of a standard data segment can be represented as UPDATE_SEGMENT, and the update operation of a non-standard data segment can be represented as FUPDATE_SEGMENT.
[0038] In some embodiments, the data redundancy protection method with erasure coding and adaptive switching of multiple replicas may further include: calculating the position of the data to be written in the file to be processed, as well as the starting offset and length of the data segment to be written.
[0039] In some embodiments, such as Figure 4As shown, the data redundancy protection method using erasure coding and adaptive multi-replica switching can further include: dividing the file namespace into multiple hash subspaces, mapping the data segments of each file to the corresponding hash subspace through hash calculation, with each hash subspace corresponding to a list of storage nodes, and several hash subspaces and their corresponding lists of storage nodes constituting a hash subspace mapping table; calculating a hash identifier (VgId) based on the file identifier and data segment identifier of the file to be processed, and obtaining the list of storage nodes (osdList) corresponding to the hash identifier from the hash subspace mapping table (VgMap). Specifically, the list of storage nodes corresponding to each hash subspace is an ordered list of storage node identifiers, containing the number of nodes equal to the number of data blocks in the encoding strategy, not exceeding the total number of storage nodes in the storage pool. The storage node order in the list of storage nodes corresponding to each hash subspace is different, and the storage nodes sequentially store a data block from the encoding strategy. Storage nodes are logical storage nodes, distinguished by storage node identifiers, and a storage host can contain one or more storage nodes. Therefore, before encoding the data segment to be encoded, the corresponding list of storage nodes can be obtained, and the data can be stored on the corresponding storage nodes of the storage pool according to the list of storage nodes and the list of data blocks. For example, the first data block of the data block list can be stored on the first storage node of the storage node list.
[0040] Specifically, in erasure coding mode, the number of data blocks should be the same as the number of storage nodes corresponding to the hash subspace, and each storage node can be used to store one data block, such as... Figure 5 As shown, the data segment to be processed can be divided into k original blocks and m encoded blocks / encoded redundancy blocks according to preset encoding parameters. Both original blocks and encoded blocks are data blocks. In multi-replica mode, the first storage node in the storage node list is selected to place the data of the original data segment as the original block, and the subsequent m storage nodes each place the data of the m data segment replicas as replica blocks.
[0041] For non-standard data segments, appending and updating operations are frequent, and the encoding cost is high. Therefore, this application adopts a multi-replica mode to store them across several storage nodes. For the entire file to be processed, most of the data is stored using erasure coding, which has high storage utilization. The small amount of frequently accessed data is stored using a simple and easy-to-use multi-replica method, which can ensure high storage utilization while avoiding time-consuming operations on a small amount of data.
[0042] Furthermore, erasure coding mode and multi-replica mode differ in the number of storage nodes storing the data. Erasure coding mode selects all storage nodes in the storage node list, while multi-replica mode selects 1+m storage nodes. With this configuration, both modes can correctly retrieve stored data even if any m storage nodes fail.
[0043] In some embodiments, such as Figure 6 As shown, if the data segment to be processed is a standard data segment and the operation type is an append write operation, the data redundancy protection method of erasure coding and multi-replica adaptive switching can include: determining whether the data segment to be encoded is full and becomes a standard data segment based on the write offset and length of the data to be written; if it is full, encoding is performed according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is not full, a data block list including 1 original block and m replica blocks is constructed by data replication.
[0044] Specifically, during system installation and deployment, detailed information about the storage pool and its corresponding encoding strategy can be stored in a metadata cluster for read by other modules on the server and client sides. To ensure high performance, this embodiment of the application preferentially uses a distributed in-memory database. It is understood that other methods of implementing the metadata storage cluster can also be used, and no limitation is made here.
[0045] In some embodiments, encoding is performed according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks. This may include: obtaining the encoding strategy and preset encoding parameters corresponding to the storage pool from the metadata cluster; instantiating an encoding module according to the encoding strategy and preset encoding parameters, wherein the preset encoding parameters include a preset length, the number of original data nodes k, and the number of encoded blocks m; dividing the standard data segment into k original blocks according to the number of original data nodes, and then using the encoding module to perform calculations to obtain m encoded blocks. Finally, these k+m data blocks can be stored in storage nodes respectively.
[0046] Accordingly, if non-standard data segments are encoded in a multi-replica mode, one original block and m replica blocks can be obtained. The (k-1)th storage nodes between the first and kth storage nodes can be left unused for storing data. The first storage node stores the original data segment, corresponding to one original block. The kth to (k+m-1)th storage nodes can be used to store replicas of the data segment, corresponding to replica blocks. All of these encoded blocks, original blocks, and replica blocks are data blocks within the data block list.
[0047] In some embodiments, such as Figure 7As shown, if the data segment to be processed is a non-standard data segment and the operation type is an append write operation, the data redundancy protection method of erasure coding and multi-replica adaptive switching may include: combining the data of the data segment to be processed with the data to be written to obtain the data segment to be encoded, and determining whether the data segment to be encoded is a standard data segment; if it is a standard data segment, then encoding the standard data segment according to the preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is a non-standard data segment, then constructing a data block list including 1 original block and m replica blocks by data replication.
[0048] Specifically, the data to be written can be written into a new data segment according to the offset and length to obtain the data segment to be written. The offset at the end of the data segment to be written is used to determine whether it is full. If it is not full, it is put into the write cache list. If it is full, the old data in the data segment to be processed is read and copied into the current data segment, and then merged with the data to be written to form a standard data segment.
[0049] Furthermore, erasure coding can be used to encode standard data segments into a list of data blocks and write it into the acquired list of storage nodes. When non-standard data segments that are not fully written to the write cache are finally written to disk, the data only contains the offset and length of the append write. A multi-replica mode can be adopted to write one original data segment and m data segment replicas into the list of storage nodes. The first storage node in the list of storage nodes stores the data of the original data segment, the middle k-1 storage nodes do not store data, and the last m storage nodes store the data segment replicas. Then, the data is sent to the server through the communication module.
[0050] In some embodiments, if an append write operation fails, the data redundancy protection method for erasure coding and adaptive switching of multiple replicas may further include: comparing the number of storage nodes that successfully wrote the data with the number of original blocks; if the number of storage nodes that successfully wrote the data is less than the number of original blocks, then notifying the storage nodes corresponding to the successful write, server failure, and timeout to delete the data; if the number of storage nodes that successfully wrote the data is greater than or equal to the number of original blocks, then recording the exception information in the exception record table as a basis for asynchronous repair.
[0051] Specifically, in erasure coding mode, the number of original blocks is k; in multi-replica mode, the number of original blocks is 1. For data blocks that are successfully written, a deletion message request is sent to the server. For data blocks that fail to be written, if the server fails or times out and it is uncertain whether to write, a deletion message request is also sent. If the connection is abnormal, no action is taken.
[0052] If the number of storage nodes that successfully writes is greater than or equal to the number of original blocks but less than the total number of storage nodes, an exception message will be recorded in the exception table, and an exception message notification will be sent to the storage nodes that timed out. Finally, the records in the exception record table will be repaired by a separate repair component.
[0053] In some embodiments, such as Figure 8 As shown, if the data segment to be processed is a standard data segment and the operation type is an update operation, the data redundancy protection method of erasure coding and multi-replica adaptive switching can include: obtaining a new original block list based on the starting position and update length of the data to be updated in the data segment to be processed; reading the corresponding old original block list from the storage nodes based on the storage node list; performing difference calculation and encoding based on the new original block list and the old original block list to obtain a coded block increment list; and writing the new original block list and the coded block increment list together into multiple storage nodes. Specifically, the new original block list can be written to the first k storage nodes, and the coded block increment list can be written to the last m storage nodes. The new original block list can include one or more data blocks, with a maximum of k data blocks.
[0054] In some embodiments, such as Figure 9 As shown, if the data segment to be processed is a non-standard data segment and the operation type is an update operation, the data redundancy protection method of erasure coding and multi-replica adaptive switching may include: combining the data of the data segment to be processed with the data to be updated to obtain the data segment to be encoded; determining whether the data segment to be encoded is a standard data segment; if it is a standard data segment, encoding the standard data segment according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is a non-standard data segment, constructing a data block list including 1 original block and m replica blocks by data replication.
[0055] Specifically, based on the offset and length of the data to be updated within the data segment, a new data segment is written to it. The system then checks if the segment is full; if not, it's added to the write cache list. If the segment is full, the old data is read from the storage node and copied to the new segment to assemble a standard data segment. When committing the data to disk, if it's a standard data segment, it's encoded and sent to the storage node list; if it's a non-standard data segment from the write cache list, it's written to the storage node list in a multi-replica mode. The methods for encoding standard data segments using erasure coding or non-standard data segments using multi-replica mode can be found in the append write operation description above, and will not be repeated here.
[0056] In some embodiments, if an update operation fails, the data redundancy protection of erasure coding and multi-replica adaptive switching may further include: comparing the number of storage nodes that have successfully written data with the number of original blocks; if the number of storage nodes that have successfully written data is less than the number of original blocks, then notifying the storage nodes corresponding to successful writing, server failure, and timeout to perform rollback repair operations; if the number of storage nodes that have successfully written data is greater than or equal to the number of original blocks, then determining whether it is a non-standard data segment; if not, then using erasure coding mode for encoding calculation and repair; if so, then using multi-replica mode for replication repair.
[0057] Specifically, exception handling for update operations can distinguish between standard data segments, non-standard data segments, and special data segments. Special data segments can refer to non-standard data segments being converted to standard data segments. For updates to standard or special data segments, if the number of storage nodes that successfully updated is less than the original number of blocks k, data version rollback can be performed on storage nodes that successfully updated, timed out, or failed to respond to the server. If data rollback fails, an exception notification is sent to the server and recorded in the exception table.
[0058] If the number of successfully updated storage nodes is greater than or equal to the original number of blocks k, but less than the total number of nodes, a full data repair is performed on the anomalies in the non-standard data segments. The repair method involves directly obtaining the data for full repair from the successfully updated storage nodes and then attempting to write it to the previously unsuccessfully updated storage nodes.
[0059] For both standard and special data segments in erasure coding mode, the standard data segment processing method can be adopted. Based on the list of failed block identifiers, other data blocks are retrieved, and the encoding module calculates the list of data blocks to be repaired using the encoding algorithm. Then, an attempt is made to write these blocks to the corresponding failed update storage nodes. If anomalies persist after data repair, they are recorded in an anomaly table, and timeout anomalies are notified to the server node. Subsequent data repair is then performed by a separate repair module.
[0060] In some embodiments, data redundancy protection with erasure coding and adaptive multi-copy switching may further include: dividing the file to be processed into several data segments according to the file size and a preset length; and decomposing the file data to be read into read data segments according to the file offset and data length of the data to be read. For each data segment to be read, it is also necessary to calculate the offset and read length of each data segment to be read in order to perform the next data reading operation.
[0061] In some embodiments, such as Figure 10As shown, the operation type can also include read operations. If the data segment to be processed is a standard data segment and the operation type is a read operation, the data redundancy protection method of erasure coding and adaptive switching of multiple replicas can include: querying the read cache table for a cached record based on the file identifier and data segment identifier; if a cached record exists, it is read directly; if no cached record exists, at least k raw blocks are read from the storage node list, decoded to obtain a standard data segment, and data is read from the standard data segment. The obtained standard data segment can then be added to the read cache table.
[0062] In some embodiments, such as Figure 11 As shown, if the data segment to be processed is a non-standard data segment and the operation type is a read operation, the data redundancy protection method of erasure coding and multi-replica adaptive switching can include: querying the read cache table for a cache record based on the file identifier and data segment identifier; if a cache record exists and the data to be read is within the cache range, then read directly; if a cache record exists but the data to be read exceeds the cache range, then at least one data block is read from the storage node list to obtain the non-standard data segment, and then data is read from the non-standard data segment; if no cache record exists, then at least one data block of the corresponding range of data is read from the storage node list according to the offset and length of the read requirement. The obtained non-standard data segment or the currently read range of data can then be added to the read cache table. Furthermore, when the data to be read exceeds the cache range, the full data of the non-standard data segment can be read. Since the non-standard data segment is written to the storage node list in multi-replica mode during writing, it can be read preferentially from the storage node storing the replica blocks during reading, successfully reading at least one data block and obtaining the required data from it.
[0063] S3. Assemble the data in the data block list to obtain an assembled package. Based on the storage node list, send the assembled package to the storage pool, and store the data in the assembled package in the corresponding storage node of the storage pool. The communication module assembles each data block in the data block list into a message packet based on a preset message format, and adds the message packet to the bidirectional queue of messages to be sent in the connection object.
[0064] Specifically, the communication module first checks whether a TCP connection has been established with the storage node to be written. After initiating and confirming the establishment of all connections with all storage nodes, the communication module assembles the data and message type information of each data block in the data block list into a message packet according to a preset message format and adds the message to the bidirectional queue of messages to be sent in the connection object. In each connection instance, there are separate message receiving and sending threads that continuously read and send messages from the receive queue and send queue, respectively, to achieve asynchronous data transmission. If the number of messages waiting to receive a reply equals the size of the data block list, it indicates that the data segment has been successfully sent.
[0065] In this application, the file to be processed is first divided into multiple data segments according to a preset length. It is then determined whether the data segment to be encoded is a standard or non-standard data segment. If it is a standard data segment, it is processed using erasure coding mode; if it is a non-standard data segment, it is processed using multi-copy mode. In this case, this application uses erasure coding mode for standard data segments that do not require frequent operations to improve storage utilization, while using multi-copy mode for non-standard data segments that require frequent operations. This avoids the high CPU computation cost caused by frequent encoding and decoding operations, thus reducing operational complexity. This application's embodiments combine the advantages of multi-copy and encoding by adaptively switching between erasure coding mode and multi-copy mode, overcoming their respective disadvantages and ensuring efficient data updates under the premise of high storage efficiency.
[0066] Those skilled in the art will understand that all or part of the features / steps of the above-described method embodiments can be implemented by methods, data processing systems, or computer programs. These features may be implemented without hardware, entirely in software, or in a combination of hardware and software. The aforementioned computer program may be stored in one or more computer-readable storage media. When the computer program is executed (e.g., by a processor), it performs the steps of the data redundancy protection method embodiments of erasure coding and adaptive switching of multiple copies described above.
[0067] The aforementioned storage media capable of storing program code include: static hard disks, solid-state hard disks, random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), optical storage devices, magnetic storage devices, flash memory, magnetic disks or optical disks and / or combinations thereof, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.
[0068] This application also provides a processing device embodiment, including one or more processors and a memory; wherein the memory is used to store one or more computer programs, and the one or more processors are used to execute the one or more computer programs stored in the memory, so that the processors execute the features / steps of the data redundancy protection method embodiment of erasure coding and multiple copy adaptive switching described above.
[0069] The above description is merely a preferred embodiment of this application. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of this application. Furthermore, under the teachings of this application, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of this application. Therefore, this application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this application.
Claims
1. A data redundancy protection method using erasure coding and adaptive switching of multiple replicas, characterized in that, include: The file to be processed is divided into several data segments according to a preset length. The data segments that reach the preset length are standard data segments, and the data segments that do not reach the preset length are non-standard data segments. Based on the operation type of each data segment to be processed, including append write operation and update operation, the data segment to be encoded is determined, and it is determined whether the data segment to be encoded is the standard data segment. If it is the standard data segment, it is encoded using erasure coding mode. If it is the non-standard data segment, it is redundantly encoded using multi-replica mode, and a list of data blocks is obtained. The data in the data block list is assembled into an assembly package. The assembly package is sent to the storage pool based on the storage node list, and the data in the assembly package is stored in the storage node corresponding to the storage pool.
2. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If the data segment to be processed is the standard data segment, and the operation type is the append write operation, the method includes: determining whether the data segment to be encoded is full and becomes the standard data segment based on the write offset and length of the data to be written; if it is full, encoding is performed according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is not full, a data block list including 1 original block and m replica blocks is constructed by data copying.
3. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If the data segment to be processed is a non-standard data segment, and the operation type is an append write operation, the method includes: combining the data of the data segment to be processed with the data to be written to obtain the data segment to be encoded; determining whether the data segment to be encoded is a standard data segment; if it is a standard data segment, encoding the standard data segment according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is a non-standard data segment, constructing a data block list including 1 original block and m replica blocks by data replication.
4. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If the data segment to be processed is the standard data segment, and the operation type is the update operation, the method includes: obtaining a new original block list based on the starting position and update length of the data to be updated in the data segment to be processed; reading the corresponding old original block list from the storage nodes based on the storage node list; performing difference calculation and encoding based on the new original block list and the old original block list to obtain an coded block increment list; and writing the new original block list and the coded block increment list together into multiple storage nodes.
5. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If the data segment to be processed is a non-standard data segment, and the operation type is the update operation, the method includes: combining the data of the data segment to be processed with the data to be updated to obtain the data segment to be encoded; determining whether the data segment to be encoded is a standard data segment; if it is a standard data segment, encoding the standard data segment according to preset encoding parameters to obtain a data block list including k original blocks and m encoded blocks; if it is a non-standard data segment, constructing a data block list including 1 original block and m replica blocks by data replication.
6. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, The operation type also includes a read operation. If the data segment to be processed is the standard data segment, the operation type is the read operation. The method includes: querying the read cache table for a cache record based on the file identifier and the data segment identifier; if the cache record exists, reading it directly; if the cache record does not exist, reading at least k raw blocks from the storage node list, decoding them to obtain the standard data segment, and reading data from the standard data segment.
7. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 6, characterized in that, If the data segment to be processed is the non-standard data segment, and the operation type is the read operation, the method includes: querying the read cache table for a cache record based on the file identifier and the data segment identifier; if the cache record exists and the data to be read is within the cache range, then reading directly; if the cache record exists but the data to be read exceeds the cache range, then reading at least one data block from the storage node list to obtain the non-standard data segment, and then reading data from the non-standard data segment; if there is no cache record, then reading at least one data block of the corresponding range data from the storage node list based on the offset and length of the read requirement.
8. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If an exception occurs during the append write operation, the method further includes: The number of storage nodes that successfully wrote data is compared with the number of original blocks. If the number of storage nodes that successfully wrote data is less than the number of original blocks, the storage nodes corresponding to the successful write, server failure, and timeout are notified to delete the data. If the number of storage nodes that successfully wrote data is greater than or equal to the number of original blocks, the exception information is recorded in the exception record table as a basis for asynchronous repair.
9. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, If the update operation encounters an error, the method further includes: The number of storage nodes that successfully wrote data is compared with the number of original blocks. If the number of storage nodes that successfully wrote data is less than the number of original blocks, the storage nodes corresponding to the successful write, server failure, and timeout are notified to perform rollback repair operations. If the number of storage nodes that successfully wrote data is greater than or equal to the number of original blocks, it is determined whether the data segment is a non-standard data segment. If not, the erasure coding mode is used for encoding calculation and repair. If so, the multi-replica mode is used for replication repair.
10. The data redundancy protection method with erasure coding and adaptive switching of multiple replicas according to claim 1, characterized in that, The method further includes: dividing the file namespace into multiple hash subspaces, mapping the data segment of each file to the corresponding hash subspace through hash calculation, with each hash subspace corresponding to a storage node list, and several hash subspaces and their corresponding storage node lists constituting a hash subspace mapping table; calculating a hash identifier based on the file identifier and data segment identifier of the file to be processed, and obtaining the storage node list corresponding to the hash identifier from the hash subspace mapping table.