Data storage method and apparatus based on erasure code

Abstract

The embodiment of the application provides a data storage method and device based on an erasure code, which comprises the following steps: a network card unit acquires first data to be stored, and divides the first data into a plurality of original data blocks. The network card unit performs erasure code processing on the plurality of original data blocks, and obtains at least one check data block corresponding to the plurality of original data blocks. The network card unit sends the plurality of original data blocks and the at least one check data block to a second storage server, so that the second storage server stores the original data blocks and the check data blocks. After the network card unit receives the first data to be stored, the network card unit directly performs data segmentation, erasure code encoding, and data transmission after the erasure code encoding is completed, so that the bandwidth cost of the storage server in the erasure code processing process can be effectively reduced.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data storage method and apparatus based on erasure coding. Background Technology

[0002] Erasure coding is a forward error correction technique with good fault tolerance and security. Distributed storage systems typically use erasure coding to achieve data redundancy and improve storage reliability.

[0003] Currently, storage systems typically include a network interface card (NIC), system memory, a CPU, and storage units. In existing storage systems implementing erasure coding, the NIC first receives user data and then transmits it to system memory via a data transmission link. The CPU then retrieves the data from system memory, performs erasure coding, and writes the resulting erasure codewords back to system memory. Finally, the NIC retrieves the erasure codewords from system memory and writes them to the storage units.

[0004] However, because multiple data transfers need to be performed within the system memory via data transfer links, this can lead to significant system bandwidth overhead and may cause a bandwidth bottleneck in the storage system. Summary of the Invention

[0005] This application provides a data storage method and apparatus based on erasure coding to overcome the problem of high system bandwidth overhead during erasure coding processing of storage servers.

[0006] In a first aspect, embodiments of this application provide a data storage method based on erasure coding, comprising:

[0007] The network interface card unit acquires the first data to be stored and divides the first data into multiple raw data blocks;

[0008] The network interface card unit performs erasure coding processing on the plurality of original data blocks to obtain at least one check data block corresponding to the plurality of original data blocks;

[0009] The network interface card unit sends the plurality of raw data blocks and the at least one check data block to the second storage server, so that the second storage server stores the raw data blocks and the check data block.

[0010] Secondly, embodiments of this application provide a data storage device based on erasure coding, comprising:

[0011] The acquisition module is used to acquire the first data to be stored and divide the first data into multiple raw data blocks;

[0012] The processing module is used to perform erasure coding processing on the plurality of original data blocks to obtain at least one check data block corresponding to the plurality of original data blocks;

[0013] The transmission module is used to send the plurality of original data blocks and the at least one verification data block to the second storage server, so that the second storage server stores the original data blocks and the verification data block.

[0014] Thirdly, embodiments of this application provide a data storage device based on erasure coding, comprising:

[0015] Memory, used to store programs;

[0016] A processor for executing the program stored in the memory, wherein, when the program is executed, the processor is configured to perform the method described in the first aspect above and any of the various possible designs of the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect above and any of the various possible designs of the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above and any of the various possible designs of the first aspect.

[0019] The erasure coding-based data storage method and apparatus provided in this application embodiment, after receiving the first data to be stored through the network interface card (NIC) unit, directly performs data segmentation and erasure coding encoding by the NIC unit. After completing the erasure coding encoding, the NIC unit also directly transmits the data. This ensures that all erasure coding-related processing is directly completed by the NIC unit, avoiding the problem of large bandwidth overhead caused by multiple data transmissions through the PCIe link inside the storage server. Therefore, it can effectively reduce the bandwidth overhead inside the storage server during erasure coding processing. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the architecture of a distributed storage system provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram illustrating the implementation of erasure coding provided in an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating the erasure coding process provided in an embodiment of this application.

[0024] Figure 4 A flowchart illustrating the erasure coding-based data storage method provided in this application embodiment;

[0025] Figure 5 The flow of the erasure coding-based data storage method provided in the embodiments of this application Figure 2 ;

[0026] Figure 6 A schematic diagram illustrating the implementation of determining data slices in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of matrix processing for erasure coding provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram illustrating the implementation of determining column elements in the generator matrix according to an embodiment of this application;

[0029] Figure 9 A schematic diagram illustrating the implementation of determining the intermediate matrix provided in an embodiment of this application;

[0030] Figure 10 A schematic diagram illustrating the implementation of determining the verification vector provided in an embodiment of this application;

[0031] Figure 11 A schematic diagram illustrating the implementation of determining the verification data block provided in an embodiment of this application;

[0032] Figure 12 Implementation diagram of the erasure coding processing circuit provided in the embodiments of this application Figure 1 ;

[0033] Figure 13 This is a schematic diagram of the erasure coding processing circuit provided in the embodiments of this application. Figure 2 ;

[0034] Figure 14 This is a schematic diagram illustrating the execution process of erasure coding provided in the embodiments of this application;

[0035] Figure 15 A schematic diagram of the structure of a data storage device based on erasure coding provided in an embodiment of this application;

[0036] Figure 16 This is a schematic diagram of the hardware structure of a data storage device based on erasure coding provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] To better understand the technical solution of this application, the relevant technologies involved in this application will be further described in detail below.

[0039] Distributed storage systems can achieve data storage with more flexible scalability and larger storage scale; for example, they can be combined with... Figure 1 To understand distributed storage systems. Figure 1 This is a schematic diagram of the architecture of a distributed storage system provided in an embodiment of this application.

[0040] like Figure 1 As shown, a distributed storage system may include multiple storage servers and a storage cluster switch.

[0041] Distributed storage systems employ a decentralized networking approach, where each storage server can simultaneously provide computing and storage resources. Multiple storage servers can... Figure 1 The storage cluster switches shown are interconnected to provide a unified pool of storage resources. The number of storage servers in the distributed storage system can be selected and configured according to actual needs.

[0042] For example, if one of the storage servers has a storage capacity of 200TB, then assuming a distributed storage system includes 5 storage servers, these 5 storage servers can collectively provide a capacity of 100TB. Therefore, distributed storage systems can be applied in scenarios involving massive amounts of data to effectively achieve flexible storage of large amounts of data.

[0043] In order to ensure data consistency, distributed storage systems can employ technologies such as multiple replicas or erasure coding when storing data.

[0044] In this context, multiple replicas involve creating multiple identical copies of the data to be stored and distributing them across multiple different storage servers. This ensures that even if data is lost on some storage servers, the data can still be retrieved from the remaining servers, thus effectively protecting the data.

[0045] Furthermore, erasure coding is a forward error correction technique that, compared to multiple copies, achieves higher data reliability with less data redundancy. The following section provides a more detailed introduction to erasure coding technology.

[0046] Erasure coding (EC) is an encoding technique that can take n original data sets, add m new data sets, and then use any n of the n+m data sets to reconstruct the original data. In other words, even if any m or fewer data sets are corrupted, the original data can still be recovered from the remaining data.

[0047] For example, we can use a system of equations to understand this. Suppose we know four numbers: a, b, c, and d. We can use two different formulas to calculate two check values, x and y, and then save all six values ​​together. If any two of the values ​​a, b, c, or d are lost, we can use the remaining two values ​​and the formulas to deduce the missing two values.

[0048] Based on the above introduction, for example, we can combine... Figure 2 Understanding erasure coding Figure 2 This is a schematic diagram illustrating the implementation of erasure coding provided in an embodiment of this application.

[0049] like Figure 2 As shown, suppose the data that needs to be stored is divided into 4 original data parts, namely... Figure 2 The data shown are A, B, C, and D.

[0050] And assuming that erasure coding technology is used to add 2 checksums to the 4 original data, namely... Figure 2 The data shown is X and Y. These six data sets can then be randomly stored on six different storage servers. That is... Figure 2 Nodes 1 to 6 are shown. Figure 2 The nodes in the table are the storage servers described above.

[0051] In this way, if data is lost due to the failure of any two storage servers, the lost data can be deduced from the unlost data, thus effectively protecting the data.

[0052] Based on the above introduction, the following will combine... Figure 3 This paper provides a more detailed explanation of the execution process for erasure coding currently implemented on storage servers. Figure 3 This is a schematic diagram of the erasure coding process provided in an embodiment of this application.

[0053] like Figure 3As shown, a storage server includes a network card, CPU, system memory, and at least one storage disk.

[0054] Currently, storage servers implement erasure coding according to... Figure 3 The steps shown are complete.

[0055] Step 1: Typically, the network interface card (NIC) in the storage server first receives the user data that needs to be stored. The NIC performs necessary network processing on the user data, such as data verification, packet splitting, and data concatenation. After completing these processes, the NIC can, for example, transfer the processed user data to the storage server's system memory via a PCIe link.

[0056] The full name of the PCIe link is PCI-Express (peripheral component interconnect express), which is a high-speed serial computer expansion bus standard.

[0057] Step 2: Next, the CPU can read user data from system memory and perform erasure coding on the user data using software computation to obtain erasure codewords. Simultaneously, the CPU can also segment the erasure codewords. Afterward, the CPU can write the segmented erasure codewords back to system memory.

[0058] Step 3: After that, the system memory can send the data of each segment to other storage servers via the network card, or transfer the data of each segment to the local storage disk, according to the storage destination of the data segment, in order to complete the erasure coding processing and storage of user data.

[0059] Reference Figure 3 It is certain that existing technologies utilize the CPU within storage servers to perform erasure coding encoding. (And refer to...) Figure 3 It should also be noted that when the CPU performs erasure coding, it does so by querying a table pre-stored in the CPU. This table corresponds to various generator matrices and, for example, stores the vector product results for various situations corresponding to various generator matrices.

[0060] By querying a table, for example, one can obtain multiple vector products corresponding to a certain generator matrix for user data. Then, the results of the vector products are summed to obtain the final product. Figure 3 The verification data shown is used as an example. The user data and verification data together constitute the erasure coding codewords described above.

[0061] against Figure 3 The technical solution shown has the following problems:

[0062] First, after entering the network interface card (NIC), massive amounts of data must pass through the PCIe link to enter system memory, where the CPU performs erasure coding. After the CPU completes the erasure coding, the data needs to pass through the PCIe link back to the NIC, or through the PCIe link to enter the local storage disk. Because the data needs to go through the data transfer process described above within the storage server, it results in more than twice the system bandwidth overhead for simultaneous read and write operations. This drawback may not be very noticeable in low-throughput scenarios, but in network configurations of 200Gb and above, it may trigger a PCIe bandwidth bottleneck.

[0063] Furthermore, during erasure coding processing, system memory bears a very high data throughput pressure. Specifically, for the CPU to execute erasure coding, it needs to read data from system memory, write the erasure code back to system memory, and re-segment it. Additionally, data transfer is required between the network card and system memory. This results in the system memory overhead being amplified by approximately three times at the same time. In other words, the system memory's data throughput is high, potentially leading to a mismatch between system memory bandwidth bottlenecks and system memory capacity.

[0064] Furthermore, the CPU uses software encoding for erasure coding, meaning that the erasure coding process is completed through the execution of an internal erasure coding program within the CPU. However, software encoding suffers from significant latency issues.

[0065] Based on the above description, it can be confirmed that the CPU uses a lookup table to perform erasure coding. The CPU pre-stores a lookup table containing various vector products corresponding to different generator matrices. Therefore, this table has a very large data volume, and the hardware overhead required for storing and querying the table is also significant.

[0066] To address the technical problems of the prior art described above, this application proposes the following technical concept: When data arrives at the storage server, it first reaches the network interface card (NIC) unit within the storage server. Then, in order to perform erasure coding in the CPU, a series of data transmissions need to be performed within the storage server, resulting in significant bandwidth overhead. Therefore, to avoid this problem, this application proposes performing erasure coding directly within the NIC unit, thereby effectively avoiding a series of data transmissions within the storage server.

[0067] However, the reason why erasure coding is performed in the CPU in existing technologies is that it requires table lookup operations, and the amount of data in the pre-stored table is very large, resulting in significant hardware overhead for storing the table and performing the lookup operations. The network interface card (NIC) cannot support such hardware overhead, therefore the erasure coding operations in the CPU cannot be directly ported to the NIC. To address this problem, this application designs an erasure coding circuit in the NIC, which enables effective erasure coding processing within the NIC with minimal hardware overhead.

[0068] The erasure coding-based data storage method provided in this application will be described in detail below with reference to specific embodiments. It should be noted that the erasure coding-based data storage method provided in this application can be applied to a distributed storage system, which may include multiple storage servers. Since the implementation methods of multiple storage servers are similar, the following description uses any one first storage server as an example. More specifically, the first storage server may include a network interface card (NIC) unit, and all the data processing described below is performed by the NIC unit in the first storage server.

[0069] First, combine Figure 4 To explain, Figure 4 A flowchart illustrating the erasure coding-based data storage method provided in this application embodiment.

[0070] like Figure 4 As shown, the method includes:

[0071] S401, The network card unit acquires the first data to be stored and divides the first data into multiple raw data blocks.

[0072] In this embodiment, the network interface card (NIC) unit can acquire the first data to be stored. In one possible implementation, the NIC unit may, for example, receive the first data sent by the device that needs to store the data, or the NIC unit may acquire the first data from some storage device.

[0073] The specific data content of the first data can be selected and set according to actual needs. This embodiment does not limit this. Any data that needs to be stored in the storage server can be understood as the first data in this embodiment.

[0074] In this embodiment, after acquiring the first data, the network interface card (NIC) unit can divide the first data into data blocks, thereby dividing the first data into multiple raw data blocks. In one possible implementation, for example, the first data can be divided into multiple raw data blocks of the same size.

[0075] It is understood that the network card unit in this embodiment needs to perform erasure coding on the first data. Erasure coding requires dividing the data into n parts first, and then generating m sets of check data for the n parts of data to achieve n+m erasure coding.

[0076] For example, in this embodiment, the primary method used is 4+2 erasure coding, so the first data can be divided into 4 parts. For instance, if the size of the first data is 32KB, then it can be divided into 4 original data blocks, each of which is 8KB in size.

[0077] In other words, in this embodiment, when dividing the first data into original data blocks, the first data can be equally divided into n data blocks. In the actual implementation, the specific choices of n and m in the n+m erasure coding to be implemented can be made according to actual needs, as long as n and m are both integers greater than or equal to 1, and n is greater than m.

[0078] S402. The network interface card unit performs erasure coding processing on multiple raw data blocks to obtain at least one check data block corresponding to the multiple raw data blocks.

[0079] After dividing the data into multiple raw data blocks, the network interface card unit in this embodiment can perform erasure coding processing on the multiple raw data blocks to obtain at least one check data block corresponding to the multiple raw data blocks.

[0080] For example, if n+m erasure coding is performed, then the corresponding number of parity data blocks is m. That is, for n original data blocks, m parity data blocks are generated.

[0081] S403, the network interface card unit sends multiple raw data blocks and at least one check data block to the second storage server, so that the second storage server stores the raw data blocks and the check data block.

[0082] After obtaining at least one checksum data block, the erasure coding processing of the original data block is essentially complete. The next step is to store both the original data block and the checksum data block. In this embodiment, the network interface card (NIC) unit can send the original data block and the checksum data block to the second storage server to complete the data block storage.

[0083] The number of secondary storage servers can be multiple. In one possible implementation, for example, multiple original data blocks and multiple check data blocks can be stored on different storage servers. Thus, each original data block and each check data block can have its own secondary storage server.

[0084] For example, the network interface card (NIC) unit can send multiple raw data blocks and at least one first checksum block to the second storage server corresponding to each data block, so that each second storage server can store its corresponding raw data block or checksum block.

[0085] The second storage server corresponding to each original data block or check data block can be randomly selected, or it can be determined according to some rule, as long as it allows multiple original data blocks and check data blocks to be stored on different storage servers.

[0086] In one possible implementation, a single original data block or verification server may also be stored in the current first storage server. That is, the multiple second storage servers described above may also be included in the first storage server performing erasure coding.

[0087] If a data block needs to be stored in the first storage server, the network interface card (NIC) unit can directly send the contents of that data block to the storage disk in the first storage server.

[0088] It is understood that in the embodiments of this application, data segmentation, erasure coding processing, and data transmission are all directly completed by the network interface card unit. Therefore, it can effectively avoid the need for multiple data back-and-forth transmissions within the storage server, thereby effectively reducing the bandwidth overhead of the storage server.

[0089] The erasure coding-based data storage method provided in this application includes: a network interface card (NIC) unit acquiring first data to be stored and dividing the first data into multiple raw data blocks. The NIC unit performs erasure coding processing on the multiple raw data blocks to obtain at least one check data block corresponding to each raw data block. The NIC unit sends the multiple raw data blocks and at least one check data block to a second storage server, enabling the second storage server to store the raw data blocks and the check data block. After receiving the first data to be stored, the NIC unit directly performs data segmentation, erasure coding, and data transmission after erasure coding. This ensures that all erasure coding-related processing is directly completed by the NIC unit, avoiding the large bandwidth overhead caused by multiple data transmissions via the PCIe link within the storage server. Therefore, it effectively reduces the bandwidth overhead within the storage server during erasure coding processing.

[0090] Based on the above description, it is necessary to further explain that the erasure coding processing performed in the network interface card (NIC) unit does not directly reuse the CPU's erasure coding processing logic. This is because the NIC unit lacks the hardware capabilities of a CPU and cannot directly port the CPU's erasure coding processing. Therefore, this application proposes a method for performing erasure coding processing in the NIC unit that has lower hardware overhead while effectively implementing erasure coding processing. The specific implementation of erasure coding processing will be described below with reference to specific embodiments, which is a further explanation of step S402 above.

[0091] The following is combined with Figures 5 to 11 To explain, Figure 5 The flow of the erasure coding-based data storage method provided in the embodiments of this application Figure 2 , Figure 6 This is a schematic diagram illustrating the implementation of determining data slices according to an embodiment of this application. Figure 7 This is a schematic diagram of matrix processing for erasure coding provided in an embodiment of this application. Figure 8 This is a schematic diagram illustrating the implementation of determining column elements in the generator matrix according to an embodiment of this application. Figure 9 This is a schematic diagram illustrating the implementation of determining the intermediate matrix according to an embodiment of this application. Figure 10 This is a schematic diagram illustrating the implementation of determining the verification vector according to an embodiment of this application. Figure 11 This is a schematic diagram illustrating the implementation of determining the verification data block provided in an embodiment of this application.

[0092] like Figure 5 As shown, the method includes:

[0093] S501. Perform multiple data slicing processes on each original data block to obtain multiple data slices, where each data slice includes w bits, where w is an integer greater than or equal to 1.

[0094] In this embodiment, each original data block includes multiple bytes, and the number of bytes included in each original data block is the same.

[0095] For example, it can be combined Figure 6 To understand the original data block, such as Figure 6 As shown, assuming the first data is divided into 4 raw data blocks, which are: Figure 6 Data blocks A, B, C, and D are shown.

[0096] Each raw data block contains k bytes, for example Figure 6 Data block A in the data includes bytes A1, A2, A3, A4, A5, ..., Ak, and the remaining data blocks are similar.

[0097] It is certain that each byte is 8 bits. In this embodiment, when dividing the original data blocks, the first data is divided equally. Therefore, the specific number of bytes included in each original data block depends on the size of the first data and the number of data blocks into which the first data is divided. This embodiment does not limit the number of bytes included in each original data block.

[0098] Because erasure coding is performed in units of multiple bits, after dividing the original data into multiple blocks, further data slicing is required to obtain multiple data slices, which facilitate subsequent erasure coding.

[0099] The following describes the data slicing process. In this embodiment, the original data block will be processed multiple times. After each data slicing process is completed, a data slice will be obtained. Each data slicing process is similar.

[0100] Specifically, data slicing processing includes: sequentially selecting one byte from each of multiple original data blocks, and combining the bytes selected from each original data block into a data slice.

[0101] For example, it can be combined Figure 6 To understand, Figure 6 The dataset contains four raw data blocks. A byte can be selected sequentially from each of these four blocks. For example, byte A1 can be selected from raw data block A, byte B1 from raw data block B, byte C1 from raw data block C, and byte D1 from raw data block D. Then, bytes A1, B1, C1, and D1 can be combined to form a dataset. Figure 6 Data slice 1 is shown.

[0102] In one possible implementation, when selecting bytes from the original data block, in order to ensure the order of the data, the bytes can be selected sequentially according to the byte order of the original data block. For example, for data block A, byte A1 is selected first, then byte A2 is selected, and so on.

[0103] For example, you can continue to refer to Figure 6 To understand this, after obtaining data slice 1, the next data slice processing can be performed on multiple original data blocks. This involves selecting byte A2 from original data block A, byte B2 from original data block B, byte C2 from original data block C, and byte D2 from original data block D. Then, bytes A2, B2, C2, and D2 can be combined to form... Figure 6 Data slice 2 is shown.

[0104] This process continues until the last byte of each original data block is selected, referring to... Figure 6 Select byte Ak from original data block A, byte Bk from original data block B, byte Ck from original data block C, and byte Dk from original data block D. Then, byte Ak, byte Bk, byte Ck, and byte Dk can be combined to form... Figure 6 The data slice k shown is shown.

[0105] exist Figure 6 In the illustrated scenario, after performing multiple data slicing processes on multiple original data blocks, k data slices can be obtained. Figure 6 Each data slice in the array consists of 32 bits.

[0106] Specifically, each data slice in this embodiment includes w bits. Figure 6 In the illustration, because one byte was selected from each of the four original data blocks, a data slice consists of four bytes, and each byte consists of eight bits. Figure 6 In the illustrated case, each data slice is 32 bits.

[0107] In one possible implementation, each data slice can be viewed as a vector, meaning each data slice can be understood as a 32-row, 1-column data vector.

[0108] S502. Obtain the generator matrix used for erasure coding processing.

[0109] In this embodiment, a generator matrix is ​​designed for erasure coding processing. Both the generator matrix and the intermediate matrix are t-row, w-column matrices, where t is an integer greater than or equal to 1 and t is less than w.

[0110] It should be noted that this embodiment uses n+m erasure coding. When dividing the first data into blocks, it is divided into n original data blocks. During data slicing, a byte is selected sequentially from each of the n element data blocks to form a data slice, where the number of bits in the data slice is w.

[0111] One byte is 8 bits, therefore n and w satisfy the relationship w = 8 × n. For example, in Figure 6 In the example, assuming 4+2 erasure coding is used, it means that the first data is divided into 4 original data blocks, and the corresponding number of bits in the generated data slice is 32.

[0112] It is also understood that this embodiment uses n+m erasure coding, which means that m additional verification data blocks need to be generated.

[0113] For example, we can take 4+2 erasure coding as an example, combined with Figure 7 To understand, Figure 7 In this context, G is the generator matrix, S is the vector corresponding to the data slice, and p is the verification vector.

[0114] Reference Figure 7 It can be confirmed that the vector corresponding to the data slice includes 32 bits, that is... Figure 7 The diagram shows a 32×1 vector S.

[0115] Furthermore, based on the vector S corresponding to the data slice and the generating matrix G, it is necessary to generate... Figure 7 The verification vector p is shown. Since this embodiment uses 4+2 erasure coding, it can be understood that for the 4 bytes included in vector S, a corresponding 2-byte verification vector needs to be generated to ensure that two redundant data blocks are subsequently obtained. Therefore... Figure 7 The parity vector p shown in the diagram consists of 16 bits.

[0116] If we use t to represent the number of bits included in the check vector, then similar to the above description, the relationship t = 8 × m needs to be satisfied.

[0117] To obtain a 16-bit parity vector p from a 32-bit vector S using the generator matrix, the generator matrix needs to be 16×32 in size. In other words, the specific size of the generator matrix depends on the choice of n and m in the n+m erasure coding process. It must satisfy the relationships w = 8×n and t = 8×m, as described above, where w is the number of columns and t is the number of rows in the generator matrix.

[0118] Furthermore, after determining the size of the generating matrix, the elements in the generating matrix need to be further determined. In one possible implementation, for example, it can be restricted that H = [G; I], where G is the generating matrix and I is the identity matrix, requiring that the H matrix must be full rank. Based on this, the specific implementation of the generating matrix can be selected according to actual needs.

[0119] S503. Sequentially obtain each bit in the data slice.

[0120] After determining the data slices and generating matrix, the specific implementation method for determining the check vector is described below.

[0121] In this embodiment, the data slice includes w bits, and these w bits need to be acquired sequentially. It is understood that the w bits in the data slice are either 1 or 0.

[0122] S504. For the i-th bit in the data slice, obtain the element of the i-th column in the generator matrix, where the value of i is from 0 to t-1.

[0123] Then, each bit in the data slice is processed sequentially. The processing method for each bit is similar. The following explanation takes the i-th bit as an example, where the value of i can be from 0 to t-1, that is, numbering starts from the 0th bit, and similar processing is performed on each bit in the data slice.

[0124] Specifically, for the i-th bit in the data slice, the corresponding i-th column element needs to be obtained from the generator matrix.

[0125] For example, you can refer to Figure 8 To understand this, let's assume i is 0, meaning the 0th bit was selected from the data slice, and then the corresponding element in the 0th column of the generator matrix is ​​obtained. Figure 8 The case shown, where i=0, yields the following column of elements: G(0,0), G(1,0), G(2,0), G(3,0), ..., G(15,0).

[0126] For example, if i is set to 1, it means that the first bit is selected from the data slice, and then the corresponding element in the first column of the generator matrix is ​​obtained. Figure 8 The case shown for i=1 yielded the following column of elements: G(0,1), G(1,1), G(2,1), G(3,1), ..., G(15,1).

[0127] Similarly, when i is 31, it means the 31st bit has been selected from the data slice, which is the last bit in the data slice. Then, the corresponding element in the 31st column of the generator matrix is ​​obtained. Figure 8 The case shown, i=31, yielded the following column of elements: G(0,31), G(1,31), G(2,31), G(3,31), ..., G(15,31).

[0128] S505. Based on the i-th bit in the data slice, update the i-th column element in the generator matrix to obtain the updated i-th column element.

[0129] After obtaining the corresponding column element of the generator matrix for each bit in the data slice, the i-th column element of the generator matrix can be updated based on the i-th bit in the data slice to obtain the updated i-th column element.

[0130] In one possible implementation, if the i-th bit is 0, then the element in the i-th column of the generator matrix is ​​set to 0, resulting in the updated element in the i-th column; or...

[0131] If the i-th bit is 1, then the element in the i-th column of the generated matrix will be used as the updated element in the i-th column.

[0132] For example, you can refer to Figure 9 To understand, such as Figure 9 As shown, assuming that when i = 0, the 0th bit in the data slice is specifically 1, then refer to... Figure 9 The element in column 0 of the generated matrix can be directly retained, meaning that the element in column 0 of the generated matrix can be directly used as the updated element in column 0 of the intermediate matrix.

[0133] For example, when i=1, the first bit in the data slice is specifically 1, then refer to... Figure 9 The element in the first column of the generated matrix can be directly retained, meaning that the element in the first column of the generated matrix can be directly used as the updated element in the first column of the intermediate matrix.

[0134] For example, when i=2, the second bit in the data slice is specifically 0, then refer to... Figure 9 You can set all the elements in the 0th column of the generated matrix to 0, thus obtaining the updated 2nd column elements (all 0) shown in the intermediate matrix.

[0135] This process continues in the same manner. Finally, for example, when i = 31, the 31st bit in the data slice is specifically 0. Then refer to... Figure 9 You can set all the elements in the 31st column of the generated matrix to 0, thus obtaining the updated 31st column elements (all 0) shown in the intermediate matrix.

[0136] S506. Based on the updated column elements corresponding to the generated matrix, obtain the updated intermediate matrix.

[0137] After obtaining the updated column elements of the generating matrix, the matrix formed by the updated column elements can be determined as the updated intermediate matrix. The intermediate matrix has the same dimensions as the generating matrix, remaining a t-row, w-column matrix.

[0138] S507. Perform XOR operations on the elements of row t in the intermediate matrix to obtain the XOR results for each element of row t.

[0139] After obtaining the intermediate matrix, we can perform XOR operations on each row of the intermediate matrix to obtain the XOR results for each row of elements.

[0140] For example, you can refer to Figure 10 To understand, Figure 10The example above continues to use the intermediate matrix. For example, the elements in the 0th row of the intermediate matrix can be XORed to obtain the XOR result corresponding to the elements in the 0th row. Figure 10 The result of the XOR operation on the 0th bit is shown.

[0141] For example, we can perform an XOR operation on the elements in the first row of the intermediate matrix to obtain the XOR result corresponding to the elements in the first row, that is... Figure 10 The result of the first bit XOR is shown.

[0142] This process continues until the last row of elements in the intermediate matrix is ​​reached. Figure 10 The elements in row 15 shown are XORed to obtain the XOR result corresponding to the elements in row 15, which is... Figure 10 The XOR result of the 15th bit shown.

[0143] S508. The XOR result of each element in row t is used to determine the check vector corresponding to the data slice. The check vector includes t bits.

[0144] After obtaining the XOR result of each row, the XOR result of each of the t rows can be used to determine the check vector of the data slice. The check vector includes t bits, and these t bits are the XOR result of each of the t rows.

[0145] For example, you can refer to Figure 10 As illustrated in the diagram, the 16 elements from the XOR result of bit 0 to bit 15 can be used as the check vector p.

[0146] S509. Determine the verification data block based on the verification vector corresponding to each data slice.

[0147] It is understood that the above processing is performed for each data slice in this embodiment, so that a verification vector p can be determined for each data slice. Therefore, the verification vector corresponding to each data slice can be used as verification data.

[0148] Based on the above description, it can be determined that this embodiment performs n+m erasure coding. The first data is divided into n original data blocks, therefore, m check data blocks need to be generated. For example, this can be combined with... Figure 11 Understand the relationship between the check vector and the check data block.

[0149] Understandably, assuming a 4+2 erasure coding scheme is used, the resulting check vector would consist of two bytes.

[0150] like Figure 11As shown, assuming a data slice consisting of bytes A1, B1, C1, and D1 is determined... Figure 11 The check vector shown is composed of bytes X1 and byte Y1.

[0151] Specifically, byte X1 can be identified as a byte in check data block X, and byte Y1 can be identified as a byte in check data block Y. A similar process is performed on the check vector corresponding to each data slice. That is, the check vector contains m bytes, and these m bytes are respectively identified as bytes in their respective check data blocks, resulting in m check data blocks. This allows for the determination of m check data blocks corresponding to n original data blocks, thus completing n+m erasure coding.

[0152] The erasure coding-based data storage method provided in this application first divides multiple original data blocks into multiple data fragments. Then, each data fragment is treated as a vector and subjected to matrix processing with a generator matrix to obtain the check vector corresponding to the data fragment. The bytes of the data fragments and the check vector satisfy a numerical relationship of n and m, thus enabling simple and effective n+m erasure coding. Furthermore, the erasure coding processing in this embodiment can be easily and effectively implemented using the erasure coding circuitry in the network interface card (NIC) without requiring table lookup operations. Compared to software implementations that rely on table lookup operations, the hardware implementation in this embodiment effectively reduces latency.

[0153] It is understood that the above embodiments describe the processing logic of the network interface card (NIC) unit when performing erasure coding. However, in actual practice, the NIC unit is equipped with an erasure coding circuit, which automatically completes the above process. The erasure coding processing of the NIC unit will be further described in detail below with reference to the specific erasure coding circuit.

[0154] Figure 12 Implementation diagram of the erasure coding processing circuit provided in the embodiments of this application Figure 1 , Figure 13 This is a schematic diagram of the erasure coding processing circuit provided in the embodiments of this application. Figure 2 .

[0155] like Figure 12 As shown, the erasure coding processing circuit includes multiple circuit units, each of which is similar.

[0156] For example, refer to Figure 12 The circuit unit indicated by 1201 includes an AND gate, an XOR gate, and a register. The output of the AND gate is connected to the input of the XOR gate, the output of the XOR gate is connected to the input of the register, and the output of the register is connected to the input of the XOR gate.

[0157] This section first introduces the AND gate, a basic logic gate circuit that performs the "AND" operation. It has multiple inputs and one output. The output is high only when all inputs are simultaneously high (logic 1); otherwise, the output is low (logic 0). In other words, an AND gate with 0 and 0 inputs outputs 0; an AND gate with 0 and 1 inputs outputs 0; an AND gate with 1 and 0 inputs outputs 0; and an AND gate with 1 and 1 inputs outputs 1.

[0158] Next, we will introduce the XOR gate, a logic gate used in digital logic to implement the logical XOR operation. It has multiple inputs and one output. If the two inputs are different in level, the output is a high level (1); if the two inputs are the same in level, the output is a low level (0). In other words, if the two inputs are different, the XOR gate outputs a high level (1). That is, an AND gate with 0 and 0 inputs outputs 0; an AND gate with 0 and 1 inputs outputs 1; an AND gate with 1 and 0 inputs outputs 1; and an AND gate with 1 and 1 inputs outputs 0.

[0159] Based on the above introduction, it can be determined that during the erasure coding process, it is necessary to obtain each bit in the data slice in sequence, for example, let S(i) represent the i-th bit in the data slice.

[0160] Additionally, it is also necessary to obtain the element in the i-th column corresponding to the i-th bit in the generator matrix. Based on the above introduction, it can actually be represented as G(0,i), G(1,i), G(2,i), G(3,i), ..., G(15,i).

[0161] Furthermore, the elements in the i-th column of the generated matrix need to be updated based on the i-th bit in the data slice. In essence, this means that if the i-th bit in the data slice is 0, all elements in the i-th column of the generated matrix are set to 0; if the i-th bit in the data slice is 1, the elements in the i-th column of the generated matrix remain unchanged. This is essentially the processing logic of an AND gate, therefore, an AND gate can be used to perform the aforementioned update of the i-th column element.

[0162] It can be determined that the i-th column actually contains t elements, therefore t circuit units can be determined in the erasure coding circuit of the network interface card (NIC) unit. In actual implementation, the erasure coding circuit in the NIC unit will include multiple circuit units to ensure that it can handle various erasure coding processes. The specific settings of the circuit units can be selected and configured according to actual needs.

[0163] Specifically, the t elements of the i-th column can be input into the t AND gates in the network interface card (NIC) unit, and the i-th bit in the data slice can also be input into the t AND gates in the NIC unit to obtain the t AND results output by the t AND gates; then the t AND results are used as the updated i-th column element.

[0164] Taking t = 16 as an example, for instance, you can refer to... Figure 12 To understand, such as Figure 12 As shown, the 16 elements in the i-th column correspond to... Figure 12 G(0,i), G(1,i), G(2,i), G(3,i), ..., G(15,i) in .

[0165] Referring to 12, the 16 elements of the i-th column can be input into 16 AND gates respectively. Specifically, the first element G(0,i) is input into the AND gate of the first circuit unit, the second element G(0,i) is input into the AND gate of the first circuit unit, and so on.

[0166] as well as, Figure 12 S(i) in the data slice represents the i-th bit. Referring to 12, for each AND gate, S(i) will also be input into each AND gate.

[0167] Then, each AND gate processes the two inputs, thus obtaining the AND result of its respective output. Figure 12 The AND results of the 16 AND gates shown together constitute the updated element of the i-th column.

[0168] After obtaining the updated elements of the i-th column, a row-wise XOR operation needs to be performed on the intermediate matrix composed of the updated columns. This XOR operation can be achieved by... Figure 12 The XOR gate shown is used for processing.

[0169] It is understandable that when performing XOR operations on a row of elements, for example, if there are 32 elements in a row, the XOR operation is performed on two of these 32 elements first, and then the determined XOR result is XORed with the next element, and so on, until the last element is XORed with the previous XOR results.

[0170] In actual implementation, for example, after determining each XOR result, the XOR result is stored in a register so that it can be XORed with the next element.

[0171] Specifically, the implementation of XOR processing for each row element in the intermediate matrix can be as follows:

[0172] The t AND results output by the t AND gates are respectively input into the t XOR gates connected to the t AND gates in the network card unit, and,

[0173] The stored historical XOR results are obtained from the t registers connected to the t XOR gates in the network card unit, and the t historical XOR results are input into their respective XOR gates to obtain the XOR results output by the t XOR gates.

[0174] The t AND results output by the t AND gates are actually the updated i-th column element described above. In this embodiment, the t circuit units can actually implement the XOR processing of the t rows of elements respectively.

[0175] For example, you can refer to Figure 13 This is because the processing of each circuit unit is similar, therefore Figure 13 Lieutenant General Figure 12 The t circuit units shown are abbreviated to 1 circuit unit.

[0176] like Figure 13 As shown, when i equals 0, S(0) and G(j,0) can be input into the AND gate to obtain the AND result of the AND gate output, where the value of j is 0 to 15, which means that 15 circuit units are represented in a simplified way.

[0177] The AND result S(0)*G(j,0) output by the AND gate actually represents the updated element of the 0th column of the generator matrix.

[0178] Next, the AND result S(0)*G(j,0) output by the AND gate can be input into the XOR gate, and the historical XOR result stored in the register can also be input into the XOR gate. Initially, the historical XOR result stored in the register is 0. Then, the XOR result output by the XOR gate is obtained. The current XOR result output by the XOR gate is actually S(0)*G(j,0).

[0179] The XOR result output by the XOR gate can then be stored in a register to update the historical XOR results stored in the register.

[0180] Then, when i equals 1, S(1) and G(j,1) can be input into the AND gate to obtain the AND result of the AND gate output, where the value of j is 0 to 15, which means that 15 circuit units are abbreviated.

[0181] The AND result S(1)*G(j,1) output by the AND gate actually represents the updated first column element of the generator matrix.

[0182] Next, the AND result S(1)*G(j,1) output by the AND gate can be input into the XOR gate, and the historical XOR result stored in the register can also be input into the XOR gate. After the above processing, the historical XOR result stored in the register is S(0)*G(j,0). The XOR result of these two inputs is then obtained, and the XOR result output by the XOR gate is obtained. The current XOR result output by the XOR gate is actually...

[0183] The XOR result output by the XOR gate can then be stored in a register to update the historical XOR results stored in the register.

[0184] The process described above is repeated until i equals 31. S(31) and G(j,31) can be input into the AND gate to obtain the AND result of the AND gate output. The value of j is 0 to 15, which means that 15 circuit units are represented in a simplified way.

[0185] The AND result S(31)*G(j,31) output by the AND gate actually represents the updated 31st column element of the generator matrix.

[0186] Then, the AND result S(31)*G(j,31) output by the AND gate can be input into the XOR gate, and the historical XOR result stored in the register can be input into the XOR gate. After the above processing, the historical XOR result stored in the register is... The two inputs are XORed, and the XOR result is then obtained from the output of the XOR gate. The XOR result output by the current XOR gate is actually the final XOR result corresponding to the element in the j-th row. Then, the final XOR result corresponding to each row element can be used to form the check vector p.

[0187] The XOR result output by the XOR gate can then be stored in a register to update the historical XOR results stored in the register.

[0188] It should also be noted that, in this embodiment, after processing each data slice and obtaining the corresponding check vector, the register stores the final XOR result for each row. To facilitate processing the next data slice, the contents stored in the register are set to 0 after processing each data slice.

[0189] Based on the above description, it can be understood that this embodiment incorporates an erasure coding circuit in the network interface card (NIC). This circuit, through a simple configuration of AND gates, XOR gates, and registers, effectively determines the corresponding parity vector for each data fragment. For example, for 4+2 erasure coding, only 16 AND gates, 16 XOR gates, and 16 registers as described above are needed to effectively implement erasure coding. Compared to lookup tables, which require significant memory storage and powerful computing capabilities during the lookup process, the erasure coding circuit provided in this embodiment has a relatively low circuit area overhead and a simple hardware design. Therefore, it can be effectively deployed within the NIC unit, enabling the NIC to perform erasure coding without relying on the CPU's computing power. This effectively shortens the data transmission path, reduces memory copying frequency, and simplifies the data processing flow.

[0190] Based on the above introduction, the following will combine... Figure 14 This paper provides a brief description of the erasure coding processing flow provided in this application, using a system architecture approach. Figure 14 This is a schematic diagram illustrating the execution process of erasure coding provided in an embodiment of this application.

[0191] Figure 14 The structure of the storage server shown is similar to that described above, and will not be repeated here. Figure 14 The same step in erasure coding is also shown.

[0192] Step 1: The data that needs to be stored first arrives at the network card, where necessary data processing can be performed.

[0193] Step 2: The network card directly performs erasure coding on the data. The specific implementation of erasure coding can be referred to the description in the above embodiment, and will not be repeated here.

[0194] Step 3: The network card stores the verification data block and the original data block obtained after erasure coding in the local storage disk, or sends them to other storage disks for storage, thus effectively completing the erasure coding processing for the data that needs to be stored.

[0195] Based on the above introduction, it can be determined that, compared with the existing implementation schemes, this embodiment implements erasure coding in the network card. At the same time, combined with the characteristics of erasure coding construction, it optimizes hardware overhead, avoids the bandwidth bottleneck of memory and PCIe link, reduces memory copying, and shortens the overall latency.

[0196] Figure 15 This is a schematic diagram of the structure of a data storage device based on erasure coding provided in an embodiment of this application. Figure 15As shown, the device 150 includes: an acquisition module 1501, a processing module 1502, and a transmission module 1503.

[0197] The acquisition module 1501 is used to acquire the first data to be stored and divide the first data into multiple raw data blocks;

[0198] Processing module 1502 is used to perform erasure coding processing on the plurality of original data blocks to obtain at least one check data block corresponding to the plurality of original data blocks;

[0199] The transmission module 1503 is used to send the plurality of original data blocks and the at least one verification data block to the second storage server, so that the second storage server stores the original data blocks and the verification data block.

[0200] In one possible design, each of the original data blocks includes the same number of bytes;

[0201] The processing module 1502 is specifically used for:

[0202] Multiple data slicing processes are performed on each of the original data blocks to obtain multiple data slices, wherein each data slice includes w bits, where w is an integer greater than or equal to 1.

[0203] Obtain the generator matrix used for the erasure coding process;

[0204] Based on the generated matrix, erasure coding is performed on each of the data slices to obtain the check vector corresponding to each of the data slices.

[0205] The verification data block is determined based on the verification vector corresponding to each of the data slices.

[0206] In one possible design, the data slicing process includes:

[0207] One byte is selected from each of the plurality of original data blocks in sequence, and the bytes selected from each of the original data blocks are combined to form a data slice.

[0208] In one possible design, the processing module 1502 is specifically used for:

[0209] Sequentially obtain each bit in the data slice;

[0210] Based on each bit in the data slice, the elements in the generated matrix are updated to obtain an updated intermediate matrix. Both the generated matrix and the intermediate matrix are t-row, w-column matrices, where t is an integer greater than or equal to 1 and t is less than w.

[0211] Perform an XOR operation on each of the elements in row t of the intermediate matrix to obtain the XOR result for each element in row t.

[0212] The XOR result of each of the t rows of elements is used to determine the check vector corresponding to the data slice, and the check vector includes t bits.

[0213] In one possible design, the processing module 1502 is specifically used for:

[0214] For the i-th bit in the data slice, obtain the element of the i-th column in the generator matrix, where the value of i is from 0 to t-1;

[0215] Based on the i-th bit in the data slice, update the i-th column element in the generator matrix to obtain the updated i-th column element;

[0216] The updated intermediate matrix is ​​obtained based on the updated column elements corresponding to the generated matrix.

[0217] In one possible design, the processing module 1502 is specifically used for:

[0218] If the i-th bit is 0, then set the i-th column element in the generated matrix to 0 to obtain the updated i-th column element; or,

[0219] If the i-th bit is 1, then the element in the i-th column of the generated matrix is ​​used as the updated element in the i-th column.

[0220] In one possible design, the i-th column of the generating matrix comprises t elements;

[0221] The processing module 1502 is specifically used for:

[0222] The t elements of the i-th column are respectively input into the t AND gates in the network card unit, and the i-th bit in the data slice is also input into the t AND gates in the network card unit to obtain the t AND results output by the t AND gates;

[0223] The t AND results are determined as the updated i-th column element.

[0224] In one possible design, the processing module 1502 is specifically used for:

[0225] The t AND results output by the t AND gates are respectively input into the t XOR gates connected to the t AND gates in the network card unit, and,

[0226] The stored historical XOR results are obtained from the t registers connected to the t XOR gates in the network card unit, and the t historical XOR results are respectively input into their respective XOR gates to obtain the XOR results output by the t XOR gates.

[0227] In one possible design, the processing module 1502 is further configured to:

[0228] The XOR results output by each of the t XOR gates are stored in their respective registers to update the historical XOR results stored in each register.

[0229] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0230] Figure 16 This is a schematic diagram of the hardware structure of a data storage device based on erasure coding provided in an embodiment of this application, as shown below. Figure 16 As shown, the erasure coding-based data storage device 160 of this embodiment includes: a processor 1601 and a memory 1602; wherein

[0231] Memory 1602 is used to store instructions executed by the computer;

[0232] The processor 1601 is used to execute computer execution instructions stored in the memory to implement the various steps of the erasure coding-based data storage method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0233] Alternatively, the memory 1602 can be either standalone or integrated with the processor 1601.

[0234] When the memory 1602 is set up independently, the erasure coding-based data storage device also includes a bus 1603 for connecting the memory 1602 and the processor 1601.

[0235] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the erasure coding-based data storage method executed by the erasure coding-based data storage device described above.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0237] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0238] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0239] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0240] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0241] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0242] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data storage method based on erasure coding, characterized in that, Applied to a first storage server, the first storage server including a network interface card (NIC) unit, the method includes: The network interface card unit acquires the first data to be stored and divides the first data into multiple raw data blocks; each raw data block contains the same number of bytes. The network interface card unit performs multiple data slicing processes on each of the original data blocks to obtain multiple data slices, wherein each data slice includes w bits, where w is an integer greater than or equal to 1, and each data slice is composed of one byte selected sequentially from each of the multiple original data blocks; The network interface card unit obtains the generator matrix for erasure coding processing, and performs erasure coding processing on each of the data slices according to the generator matrix to obtain the check vector corresponding to each of the data slices. The network interface card unit determines at least one check data block corresponding to the plurality of original data blocks based on the check vector corresponding to each of the data slices. The network interface card unit sends the plurality of raw data blocks and the at least one check data block to the second storage server, so that the second storage server stores the raw data blocks and the check data block.

2. The method according to claim 1, characterized in that, Based on the generated matrix, erasure coding is performed on each of the data slices to obtain a check vector corresponding to each data slice, including: Sequentially obtain each bit in the data slice; Based on each bit in the data slice, the elements in the generated matrix are updated to obtain an updated intermediate matrix. Both the generated matrix and the intermediate matrix are t-row, w-column matrices, where t is an integer greater than or equal to 1 and t is less than w. Perform an XOR operation on each of the elements in row t of the intermediate matrix to obtain the XOR result for each element in row t. The XOR result of each of the t rows of elements is used to determine the check vector corresponding to the data slice, and the check vector includes t bits.

3. The method according to claim 2, characterized in that, Based on each bit in the data slice, the elements in the generated matrix are updated to obtain the updated intermediate matrix, including: For the i-th bit in the data slice, obtain the element of the i-th column in the generator matrix, where the value of i is 0 to t-1; Based on the i-th bit in the data slice, update the i-th column element in the generator matrix to obtain the updated i-th column element; The updated intermediate matrix is ​​obtained based on the updated column elements corresponding to the generated matrix.

4. The method according to claim 3, characterized in that, Based on the i-th bit in the data slice, the i-th column element in the generated matrix is ​​updated to obtain the updated i-th column element, including: If the i-th bit is 0, then set the i-th column element in the generated matrix to 0 to obtain the updated i-th column element; or, If the i-th bit is 1, then the element in the i-th column of the generated matrix is ​​used as the updated element in the i-th column.

5. The method according to claim 3 or 4, characterized in that, The i-th column of the generated matrix includes t elements; Based on the i-th bit in the data slice, the i-th column element in the generated matrix is ​​updated to obtain the updated i-th column element, including: The t elements of the i-th column are respectively input into the t AND gates in the network card unit, and the i-th bit in the data slice is also input into the t AND gates in the network card unit to obtain the t AND results output by the t AND gates; The t AND results are determined as the updated i-th column element.

6. The method according to claim 5, characterized in that, Perform an XOR operation on each of the t rows of the intermediate matrix to obtain the XOR result for each of the t rows, including: The t AND results output by the t AND gates are respectively input into the t XOR gates connected to the t AND gates in the network card unit, and, The stored historical XOR results are obtained from the t registers connected to the t XOR gates in the network card unit, and the t historical XOR results are respectively input into their respective XOR gates to obtain the XOR results output by the t XOR gates.

7. The method according to claim 6, characterized in that, The method further includes: The XOR results output by each of the t XOR gates are stored in their respective registers to update the historical XOR results stored in each register.

8. A data storage device based on erasure coding, characterized in that, Applied to a first storage server, the first storage server including a network interface card (NIC) unit, the device includes: An acquisition module is used to acquire first data to be stored and divide the first data into multiple raw data blocks; each raw data block contains the same number of bytes; The processing module is configured to perform multiple data slicing processes on each of the original data blocks to obtain multiple data slices, wherein each data slice includes w bits, where w is an integer greater than or equal to 1, and each data slice is composed of one byte selected sequentially from each of the multiple original data blocks; obtain the generation matrix of the erasure coding process, and perform erasure coding processing on each of the data slices according to the generation matrix to obtain a check vector corresponding to each of the data slices; and determine at least one check data block corresponding to the multiple original data blocks based on the check vectors corresponding to each of the data slices. The transmission module is used to send the plurality of original data blocks and the at least one verification data block to the second storage server, so that the second storage server stores the original data blocks and the verification data block.

9. A data storage device based on erasure coding, characterized in that, include: Memory, used to store programs; A processor for executing the program stored in the memory, wherein when the program is executed, the processor is configured to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images