Data processing method and device based on erasure codes, equipment and storage medium

Abstract

The invention provides a data processing method and device based on erasure codes, equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that first erasure data corresponding to data to be processed is determined based on an erasure code algorithm, and the first erasure data comprises at least one first source data block and at least one first verification data block; splicing all the first verification data blocks, and segmenting the spliced first verification data blocks to obtain at least two segmented second verification data blocks; performing erasure coding on the at least two second verification data blocks to obtain at least one third verification data block corresponding to the at least two segmented second verification data blocks; and determining second erasure data based on the at least one first source data block, the at least two second verification data blocks and the at least one third verification data block, and performing discrete storage on the second erasure data. According to the invention, the error-tolerant rate of erasure code storage can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium based on erasure code. Background Art

[0002] In video surveillance cloud storage systems, erasure code storage is widely used in the industry to replace multi-copy storage mechanisms due to its flexible erasure redundancy strategy configuration, low storage overhead and network overhead.

[0003] However, the fault tolerance of existing erasure code storage is low, that is, when the erasure redundancy ratio is fixed, it can only tolerate a fixed number of hard disk failures at most. Once the number of hard disk failures exceeds the redundancy limit, the entire erasure data will be lost. Therefore, how to improve the fault tolerance of erasure code storage is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a data processing method, device, equipment and storage medium based on erasure code, so as to solve the defect of low fault tolerance of erasure code storage in the prior art and improve the fault tolerance of erasure code storage.

[0005] The present invention provides a data processing method based on erasure code, comprising:

[0006] Determine, based on an erasure code algorithm, first erasure data corresponding to the data to be processed, where the first erasure data includes at least one first source data block and at least one first check data block;

[0007] All the first verification data blocks are spliced ​​together, and the spliced ​​first verification data blocks are divided to obtain at least two second verification data blocks after division, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks;

[0008] Performing erasure coding on the at least two second verification data blocks to determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0009] Based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block, second erasure correction data is determined, and the second erasure correction data is discretely stored.

[0010] According to the data processing method based on erasure codes provided by the present invention, the first check data block after splicing is segmented to obtain at least two second check data blocks after segmentation, including:

[0011] Determining the number of blocks corresponding to the at least one first source data block;

[0012] Determine a target number of divisions for dividing the spliced ​​first verification data block according to a difference between the number of hard disks in the target storage pool in the storage server and the number of blocks corresponding to the at least one first source data block;

[0013] When the target segmentation number is greater than the number of blocks of the first verification data block, the spliced ​​first verification data block is segmented based on the target segmentation number to obtain the target segmentation number of second verification data blocks after segmentation.

[0014] According to the data processing method based on erasure codes provided by the present invention, determining the second erasure data based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block includes:

[0015] Determine a target source data block from the at least one first source data block, the number of blocks of the target source data block being the same as the number of blocks of the at least one third verification data block;

[0016] Splicing each of the third verification data blocks with the target source data block to obtain at least one spliced ​​source data block;

[0017] Determine at least two second source data blocks based on the spliced ​​source data block and other first source data blocks among all the first source data blocks except the target source data block, wherein the number of blocks of the second source data blocks is equal to the number of blocks of the first source data block;

[0018] Second erasure correction data is determined based on the at least one second source data block and the at least two second check data blocks.

[0019] According to the data processing method based on erasure codes provided by the present invention, after discretely storing the second erasure data, the method further includes:

[0020] In the event that a hard drive failure is detected, determining the number of blocks of lost data;

[0021] Determine a data type of the lost data; the data type is used to characterize that the lost data is a lost source data block and / or a lost check data block, the lost source data block belongs to the second source data block, and the lost check data block belongs to the second check data block;

[0022] The lost data is recovered based on the data type of the lost data.

[0023] According to the data processing method based on erasure codes provided by the present invention, the lost data is recovered based on the data type of the lost data, including:

[0024] In a case where the data type includes the lost source data block, and the lost source data block includes the first source data block, reading all second check data blocks and other first source data blocks except the lost source data block, and recovering the lost source data block;

[0025] In the case that the data type includes the lost source data block, and the lost source data block includes the spliced ​​source data block, all second check data blocks and other first source data blocks except the lost source data block are read, and the first source data block in the spliced ​​source data block is restored, and all second check data blocks and the third check data blocks that are not lost are read, and the third check data block in the spliced ​​source data block is restored.

[0026] According to the data processing method based on erasure codes provided by the present invention, the lost data is recovered based on the data type of the lost data, including:

[0027] In the case that the data type includes the lost check data block, all third check data blocks and the non-lost second check data blocks are read to recover the lost check data block.

[0028] According to the data processing method based on erasure codes provided by the present invention, the lost data is recovered based on the data type of the lost data, including:

[0029] In the case where the data type includes the lost source data block and the lost check data block, all second check data blocks and other first source data blocks except the lost source data block are read, or all second check data blocks, other first source data blocks except the lost source data block and non-lost third check data blocks are read to recover the lost source data block;

[0030] All third check data blocks and the non-lost second check data blocks are read, and the lost check data blocks are recovered.

[0031] The present invention also provides a data processing device based on erasure code, comprising:

[0032] A first erasure correction module, configured to determine first erasure correction data corresponding to the data to be processed based on an erasure correction code algorithm, wherein the first erasure correction data includes at least one first source data block and at least one first check data block;

[0033] A segmentation module, used for splicing all the first verification data blocks, and segmenting the spliced ​​first verification data blocks to obtain at least two segmented second verification data blocks, wherein the number of segments of the second verification data blocks is greater than the number of segments of the first verification data blocks;

[0034] A second erasure correction module, configured to perform erasure correction coding on the at least two second verification data blocks, and determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0035] A storage module is used to determine second erasure correction data based on the at least one first source data block, the at least two second verification data blocks and the at least one third verification data block, and discretely store the second erasure correction data.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method based on erasure codes as described above is implemented.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data processing method based on erasure codes as described above is implemented.

[0038] The data processing method, device, equipment and storage medium based on erasure codes provided by the present invention, after determining the first erasure data corresponding to the data to be processed by the erasure code algorithm, all the first verification data in the first erasure data are spliced ​​and then divided, and the second verification data blocks obtained after the division are erasure encoded to obtain at least one third verification data block, and the second erasure data is determined according to the first source data block, the second verification data block and the third verification data block, and the second erasure data is discretely stored. Since the number of the obtained second verification data blocks is greater than the number of blocks of the first verification data block, the discreteness of the second erasure data is improved. At the same time, the secondary erasure of the second verification data block increases the data allowed to be lost in the second erasure data, thereby improving the fault tolerance rate of erasure code storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flowchart of a data processing method based on erasure codes provided by an embodiment of the present invention;

[0041] Figure 2 is an exemplary schematic diagram of a data processing method based on erasure codes provided by an embodiment of the present invention;

[0042] Figure 3is one of the schematic diagrams of data recovery provided by the embodiments of the present invention;

[0043] Figure 4 is the second of the schematic diagrams of data recovery provided by the embodiments of the present invention;

[0044] Figure 5 is the structural schematic diagram of a data processing device based on erasure code provided by the embodiments of the present invention;

[0045] Figure 6 is the structural schematic diagram of an electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0046] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0047] Aiming at the problem of low fault tolerance rate in erasure code storage in the prior art, the embodiments of the present invention provide a data processing method based on erasure code. Figure 1 is the flowchart of the data processing method based on erasure code provided by the embodiments of the present invention. As Figure 1 shown, the method includes:

[0048] Step 110: Determine first erasure data corresponding to the data to be processed based on an erasure code algorithm, where the first erasure data includes at least one first source data block and at least one first parity data block.

[0049] Specifically, after obtaining the data to be processed, according to the erasure code algorithm, taking the erasure redundancy ratio as N+M as an example, that is, the number of data blocks in the first erasure data is N+M. First, the data to be processed is divided into N first source data blocks, a target matrix is constructed based on the N first source data blocks, and a reversible generator matrix is selected. Performing matrix multiplication operation on the generator matrix and the target matrix can obtain M first parity data blocks, and M<N. In the prior art, at most M data blocks can be tolerated to be lost. The lost data blocks can be any M first source data blocks, or M first parity data blocks, or a first source data blocks and b first parity data blocks, and a+b = M, where a and b are both integers greater than or equal to 1.

[0050] Optionally, the above generator matrix can be a Vandermonde matrix or a Cauchy matrix, and the erasure code operation complexity of the Cauchy matrix is lower than that of the Vandermonde matrix.

[0051] Optionally, in the prior art, after determining the first erasure data, the first erasure data may be stored in different storage media in the storage server, and the storage media may include: a hard disk, a solid state drive, a cloud storage, a memory, and a flash memory, etc., which is not limited in the embodiment of the present invention. For ease of description, the hard disk is used as the storage medium below.

[0052] Optionally, in the field of video surveillance, the above-mentioned data to be processed may include: video data, text data and index data, etc., wherein: video data may be the original video stream captured by the surveillance camera. Text data may be data related to video data, and the text data may include the acquisition date, acquisition time, location, event type, etc. of the video data. Index data is the tag data of video data and text data stored in the storage server, which is convenient for fast query of video data and text data. The index data may include file name, timestamp and data type, etc., which is not limited in the embodiment of the present invention.

[0053] It should be noted that the Erasure Coding (EC) algorithm is a data redundancy and recovery method in a distributed storage system. The data to be processed is divided into several source data blocks according to a fixed erasure redundancy ratio, and then several source data blocks are erasure-coded to generate verification data blocks. Finally, each source data block and verification data block are stored on different storage media. When some data blocks are lost or damaged, the data can be reconstructed through the remaining data blocks, and the number of lost data blocks is less than the number of verification data blocks. When storing the same data to be processed, according to the 2-copy storage mechanism, double the data to be processed needs to be stored when storing the data to be processed, which occupies double the storage overhead. However, taking the erasure redundancy ratio of 4+2 as an example, according to the erasure coding algorithm, 4 source data blocks and 2 verification data blocks can be obtained. When storing data, only 6 data blocks need to be stored based on the erasure coding algorithm, and the storage overhead is less than the storage overhead of the 2-copy storage mechanism.

[0054] For example, Figure 2 is an example schematic diagram of a data processing method based on erasure codes provided by an embodiment of the present invention, such as Figure 2 As shown, taking the correction and erasure redundancy ratio of 4+2 and the number of hard disks in the storage server as 8 as an example, first, the data to be processed is divided into the first source data blocks S0-S3, and the Cauchy matrix is ​​selected as the generating matrix. The Cauchy matrix and the target matrix composed of the first source data blocks S0-S3 are subjected to Galois field operation to obtain the first verification data block P0-P1. The first source data block S0-S3 and the first verification data block P0-P1 constitute the first correction and erasure data. The first source data block S0-S3 and the first verification data block P0-P1 can be collectively referred to as a stripe unit. In the prior art, it is allowed to lose any two stripe units.

[0055] Step 120: splice all first verification data blocks, and divide the spliced ​​first verification data blocks to obtain at least two divided second verification data blocks, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks.

[0056] Specifically, after determining the first verification data, all first verification data blocks can be spliced ​​and then segmented, and it is ensured that the number of blocks of the second verification data block obtained after segmentation is greater than the number of blocks of the first verification data block, thereby improving the discreteness of the first verification data block. It should be noted that the sum of the number of blocks of the second verification data block obtained after segmentation and the number of blocks of the first source data block must be less than or equal to the number of hard disks in the storage server to ensure subsequent successful storage. For example, the number of hard disks in the storage server is 10, the number of blocks of the first source data block is 4, and the number of blocks of the first verification data block is 2. Then the number of blocks of the second verification data block after segmentation can be any one of 3-6.

[0057] Furthermore, the step of segmenting the spliced ​​first verification data block to obtain at least two segmented second verification data blocks includes:

[0058] Determining the number of blocks corresponding to the at least one first source data block;

[0059] Determine a target number of divisions for dividing the spliced ​​first verification data block according to a difference between the number of hard disks in the target storage pool in the storage server and the number of blocks corresponding to the at least one first source data block;

[0060] When the target segmentation number is greater than the number of blocks of the first verification data block, the spliced ​​first verification data block is segmented based on the target segmentation number to obtain the target segmentation number of second verification data blocks after segmentation.

[0061] Specifically, after all the first verification data blocks are spliced ​​and before they are split, the number of hard disks in the storage server corresponding to the target storage pool can be determined first, and the number of hard disks is the total number of all hard disks in the target storage pool, and the number of blocks of all the first source data blocks can be further determined. According to the difference between the number of hard disks and the number of blocks of all the first source data blocks, the target number of splits for splitting the spliced ​​first verification data block can be determined. The difference can be understood as the number of remaining hard disks when all the first source data blocks are stored in the target storage pool. The target number of splits is the number of blocks of the second verification data block obtained after the spliced ​​first verification data block is split. After determining the target number of splits for splitting the spliced ​​first verification data block, it is determined whether the target number of splits is greater than the number of blocks of the first verification data block. If the target number of splits is greater than the number of blocks of the first verification data block, the spliced ​​first source data block is split according to the target number of splits to obtain the target number of splits of the second verification data block. When storing all the second verification data blocks and all the first source verification data blocks, they can be distributed and stored in all the hard disks in the storage server, ensuring that all the hard disks can evenly share the load and reducing the storage pressure of individual hard disks.

[0062] It should be noted that in order to avoid excessive segmentation of the first check data block after splicing when the total number of hard disks in the storage server is large, the storage server can create at least one storage pool according to the total number of hard disks and the error correction ratio. Each storage pool includes multiple hard disks, and one piece of data to be processed can be discretely stored in one storage pool. For example, if the total number of hard disks corresponding to the storage server is 60, and the error correction ratio is 6+2, the 60 hard disks can be divided into 6 storage pools, each storage pool includes 10 hard disks, and one piece of data to be processed can be discretely stored in each hard disk in any storage pool based on the error correction code algorithm. If the total number of hard disks corresponding to the storage server is 10, the 10 hard disks can be divided into 1 storage pool, that is, this storage server is 1 storage pool.

[0063] For example, Figure 2 As shown, after determining the first erasure correction data, the first check data block P0 and the first check data block P1 can be spliced ​​and then split. According to the difference between the number of hard disks in the target storage pool and the number of blocks of the first source data block, it can be determined that the remaining number of hard disks or the target number of splits is 4, that is, the first check data block P0 and the first check data block P1 are spliced ​​and then split to obtain the second check data block P0-part1, the second check data block P0-part2, the second check data block P1-part1 and the second check data block P1-part2.

[0064] Step 130: Perform erasure coding on the at least two second check data blocks to determine at least one third check data block corresponding to the at least two second check data blocks.

[0065] Specifically, taking the case where the erasure redundancy ratio is 4+2 and 4 second verification data blocks are obtained after segmentation as an example, after the second verification data blocks are obtained after segmentation, if the first source data blocks and the second verification data blocks are directly stored discretely and there is a hard disk failure, the number of data blocks actually allowed to be lost is not 4, but still two, that is, if 4 second verification data blocks are lost, two data blocks are actually lost, and all the remaining first source data blocks that are not lost can be read to recover the 4 second verification data blocks. However, if two first source data blocks and two second verification data blocks are lost, three data blocks are actually lost, and even if all the remaining first source data blocks that are not lost and the remaining two second verification data blocks are read, the lost data blocks cannot be recovered. Therefore, after the spliced ​​first verification data is segmented, only the data discreteness is increased, and the number of data blocks allowed to be lost is not changed, that is, the fault tolerance rate of the erasure code storage is not changed. Therefore, in an embodiment of the present invention, after segmentation to obtain at least two second verification data blocks, all second verification data blocks can be error-corrected and erasure encoded to obtain at least one third verification data block, that is, the second verification data blocks can be regarded as source data blocks, a new target matrix can be generated based on all second verification data blocks, and a reversible new generator matrix can be selected to perform matrix multiplication operation on the new target matrix and the new generator matrix to obtain at least one third verification data block, so that after the second verification data block is lost, all the third verification data blocks can be read to recover the data of the lost second verification data block. For example, taking the case where the erasure redundancy ratio is 4+2 and 4 second verification data blocks are obtained after segmentation, after the second verification data blocks are erasure encoded to obtain at least two third verification data blocks, and when there is a hard disk failure, the number of data blocks that are actually allowed to be lost is increased from two to 4, that is, if 4 second verification data blocks are lost, the 4 second verification data blocks can be restored through all third verification data blocks; however, if two first source data blocks and two second verification data blocks are lost, all third verification data blocks and the remaining two second verification data blocks can be read to restore the lost two second verification data blocks, and the lost two first source data blocks can be restored through the restored 4 second verification data blocks, which fundamentally increases the number of data blocks allowed to be lost, thereby improving the fault tolerance of erasure code storage.

[0066] It should be noted that the number of blocks of the at least one third verification data block is smaller than the number of blocks of the at least one first source data block, and the number of blocks of the at least one third verification data block is also smaller than the number of blocks of the at least two second verification data blocks, so that the subsequent third verification data block can be spliced ​​with the first source data block, and after the second verification data block is lost, there is no need to read the non-lost first source data block. The lost second verification data block can be restored by reading all the third verification data blocks and the non-lost second verification data blocks, thereby reducing the amount of data read and the number of reads, and improving the write performance of data storage.

[0067] Step 140: Determine second erasure correction data based on the at least one first source data block, the at least two second check data blocks, and the at least one third check data block, and discretely store the second erasure correction data.

[0068] Specifically, after determining the first source data block, the second verification data block and the third verification data block, the second erasure correction data can be determined based on the first source data block, the second verification data block and the third verification data block, and each data block in the second erasure correction data can be stored on each hard disk in the storage server respectively.

[0069] Further, the determining the second erasure correction data based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block includes:

[0070] Determine a target source data block from the at least one first source data block, the number of blocks of the target source data block being the same as the number of blocks of the at least one third verification data block;

[0071] Splicing each of the third verification data blocks with the target source data block to obtain at least one spliced ​​source data block;

[0072] Determine at least two second source data blocks based on the spliced ​​source data block and other first source data blocks among all the first source data blocks except the target source data block, wherein the number of blocks of the second source data blocks is equal to the number of blocks of the first source data block;

[0073] Second erasure correction data is determined based on the at least one second source data block and the at least two second check data blocks.

[0074] Specifically, after obtaining the third verification data block, the target source data block can be determined from all the first source data blocks according to the number of the third verification data blocks, and the third verification data block can be spliced ​​with the corresponding target source data block in sequence to obtain a spliced ​​source data block. Since the number of blocks of the third verification data block is less than the number of blocks of the first source data block, after determining the target source data block, some of the first source data blocks are still left, and the spliced ​​source data block and the remaining first source data block can be determined as the second source data block, and the second erasure correction data can be determined according to the second source data block and the second verification data block.

[0075] Optionally, after determining the target source data block, the third check data block can be sequentially spliced ​​with the target source data blocks with corresponding numbers according to the numbering sequence of the target source data blocks to obtain a spliced ​​source data block. In addition, the third check data block can also be spliced ​​with any target source data block, but an association table needs to be constructed, and the association table needs to store the correspondence between the numbers of the third check data block and the numbers of the target source data blocks.

[0076] For example, Figure 2 As shown, after the second check data block is divided, the second check data block can be error-corrected and erasure encoded to obtain a third check data block P3 and a third check data block P4, and the third check data block P3 is spliced ​​with the first source data block S0 to obtain a second source data block S0-P3. At the same time, the third check data block P4 is spliced ​​with the first source data block S1 to obtain a second source data block S1-P4, and the remaining first source data block S2 is determined as the second source data block S2, and the first source data block S3 is determined as the second source data block S3. The second source data block S0-P3, the second source data block S1-P4, the second source data block S2, the second source data block S3, the second check data block P0-part1, the second check data block P0-part2, the second check data block P1-part1 and the second check data block P1-part2 together constitute the second erasure-corrected data. When the second erasure data is discretely stored, the second source data block S0-P3 can be stored in hard disk 1 (DESK1), the second source data block S1-P4 can be stored in hard disk 2 (DESK2), the second source data block S2 can be stored in hard disk 3 (DESK3), the second source data block S3 can be stored in hard disk 4 (DESK4), the second verification data block P0-part1 can be stored in hard disk 5 (DESK5), the second verification data block P0-part2 can be stored in hard disk 6 (DESK6), the second verification data block P1-part1 can be stored in hard disk 7 (DESK7), and the second verification data block P1-part2 can be stored in hard disk 8 (DESK8). After the second erasure data is discretely stored, the number of stripe units allowed to be lost is increased from two to four, thereby improving the fault tolerance of erasure code storage.

[0077] Furthermore, after discretely storing the second erasure data, the method further includes:

[0078] In the event that a hard drive failure is detected, determining the number of blocks of lost data;

[0079] Determine a data type of the lost data; the data type is used to characterize that the lost data is a lost source data block and / or a lost check data block, the lost source data block belongs to the second source data block, and the lost check data block belongs to the second check data block;

[0080] The lost data is recovered based on the data type of the lost data.

[0081] Further, the recovering the lost data based on the data type of the lost data includes:

[0082] In a case where the data type includes the lost source data block, and the lost source data block includes the first source data block, reading all second check data blocks and other first source data blocks except the lost source data block, and recovering the lost source data block;

[0083] In the case that the data type includes the lost source data block, and the lost source data block includes the spliced ​​source data block, all second check data blocks and other first source data blocks except the lost source data block are read, and the first source data block in the spliced ​​source data block is restored, and all second check data blocks and the third check data blocks that are not lost are read, and the third check data block in the spliced ​​source data block is restored.

[0084] Specifically, after the second erasure data is discretely stored, if a hard disk failure is detected, the data type of the lost data is first detected. If the data type of the lost data includes a lost source data block, that is, only the second source data block is lost, it is necessary to determine whether the lost source data block is a first source data block or a spliced ​​source data block. If the lost source data block is the first source data block, all second check data blocks and other first source data blocks that have not been lost can be read to perform erasure decoding calculations to recover data from the lost first source data block. If the lost source data block is a spliced ​​source data block, that is, the lost source data block includes both the first source data block and the third check data block, at this time, on the one hand, all second check data blocks and other first source data blocks that have not been lost can be read to perform erasure decoding calculations to recover data from the lost first source data block, and on the other hand, all second check data blocks and third check data blocks that have not been lost can be read to perform erasure decoding calculations to recover data from the lost third check data block.

[0085] Further, the recovering the lost data based on the data type of the lost data includes:

[0086] In the case that the data type includes the lost check data block, all third check data blocks and the non-lost second check data blocks are read to recover the lost check data block.

[0087] Specifically, if the data type of the lost data includes a lost check data block, that is, the lost data only includes a second check data block, all third check data blocks and other non-lost second check data blocks can be read to perform erasure decoding calculations to recover data of the lost second check data block.

[0088] For example, Figure 3 This is a second example schematic diagram of data recovery provided by an embodiment of the present invention. Figure 3 As shown, taking the second erasure data including the second source data block S0-P3, the second source data block S1-P4, the second source data block S2, the second source data block S3, the second source data block S4, the second source data block S5, the second source data block S6, the second source data block S7, the second check data block P0-part1, the second check data block P0-part2, the second check data block P1-part1 and the second check data block P1-part2, and the second check data block P0-part1 and the second check data block P0-part2 are lost for a total of 2 stripe units as an example, the data recovery steps are as follows:

[0089] 1) Read the third check data block P3 from DISK1, read the third check data block P4 from DISK2, read the second check data block P1-part1 from DISK11, and read the second check data block P1-part2 from DISK12;

[0090] 2) Use the third check data block P3, the third check data block P4, the second check data block P1-part1 and the second check data block P1-part2 to perform erasure decoding calculations to recover the lost second check data block P0-part1 and the second check data block P0-part2. There is no need to read all the stripe units in the remaining 10 hard disks. Only the stripe units in 4 hard disks need to be read to complete the data recovery.

[0091] Further, the recovering the lost data based on the data type of the lost data includes:

[0092] In the case where the data type includes the lost source data block and the lost check data block, all second check data blocks and other first source data blocks except the lost source data block are read, or all second check data blocks, other first source data blocks except the lost source data block and non-lost third check data blocks are read to recover the lost source data block;

[0093] All third check data blocks and the non-lost second check data blocks are read, and the lost check data blocks are recovered.

[0094] Specifically, if the data type of the lost data includes both lost source data blocks and lost check data blocks, that is, there is a loss of the first source data block and the second check data block, or a loss of the spliced ​​source data block and the second check data block, for the loss of the first source data block and the second check data block, all the second check data blocks and other first source data blocks that have not been lost can be read to perform erasure correction and decoding calculations, and data of the lost first source data block can be recovered. At the same time, all the third check data blocks and other second check data blocks that have not been lost are read to perform erasure correction and decoding calculations, and data of the lost second check data block can be recovered. For the lost spliced ​​source data block and the second check data block, all the second check data blocks and other first source data blocks except the lost source data block can be read to perform erasure decoding calculation, and the first source data block in the lost spliced ​​source data block can be recovered, and all the second check data blocks and the third check data blocks that have not been lost can be read to perform erasure decoding calculation, and the third check data blocks in the lost spliced ​​source data block can be recovered. After that, all the third check data blocks and the second check data blocks that have not been lost can be read to perform erasure decoding calculation, and the lost second check data blocks can be recovered. Compared with the prior art that requires reading all the non-lost stripe units to recover the lost stripe units, in the embodiment of the present invention, when performing data recovery, the second check data block and the second source data block formed after the secondary erasure are mutually verified, and the lost stripe units can be successfully recovered by reading part of the stripe units in the remaining non-lost stripe units, thereby reducing the amount of data read and the number of reads, and improving the IO performance of the storage server.

[0095] In addition, if the number of blocks of the lost data exceeds the number of blocks of the second verification data blocks, the lost data cannot be recovered; if, in the preset erasure data composed of all second verification data blocks and all third verification data blocks, the number of blocks of the lost data exceeds the number of blocks of the third verification data blocks, that is, the data type of the lost data includes both the second verification data blocks and the third verification data blocks, and the sum of the number of blocks of the lost second verification data blocks and the third verification data blocks exceeds the number of blocks of the third verification data blocks, the lost data cannot be recovered.

[0096] For example, Figure 4 is one of the exemplary schematic diagrams of data recovery provided by an embodiment of the present invention, such as Figure 4 As shown, taking the discretely stored second erasure data including the second source data block S0-P3, the second source data block S1-P4, the second source data block S2, the second source data block S3, the second check data block P0-part1, the second check data block P0-part2, the second check data block P1-part1 and the second check data block P1-part2, and the second source data block S2, the second source data block S3, the second check data block P0-part1, and the second check data block P0-part2 are lost, a total of 4 stripe units are lost, the data recovery steps are as follows:

[0097] 1) Read the third check data block P3 from DISK1, read the third check data block P4 from DISK2, read the second check data block P1-part1 from DISK7, and read the second check data block P1-part2 from DISK8 respectively;

[0098] 2) Perform erasure decoding calculation using the third check data block P3, the third check data block P4, the second check data block P1-part1 and the second check data block P1-part2 to recover the lost second check data block P0-part1 and the second check data block P0-part2;

[0099] 3) Perform erasure correction and decoding calculation using the non-lost second check data block P1-part1 and the second check data block P1-part2, and the recovered second check data block P0-part1 and the second check data block P0-part2 to recover the lost second source data block S2 and the second source data block S3.

[0100] The data processing method based on erasure codes provided in the embodiment of the present invention determines the first erasure data corresponding to the data to be processed through the erasure code algorithm, then splices all the first verification data in the first erasure data and then divides them, and performs erasure coding on the second verification data block obtained after the division to obtain at least one third verification data block, and determines the second erasure data according to the first source data block, the second verification data block and the third verification data block, and stores the second erasure data discretely. Since the number of the obtained second verification data blocks is greater than the number of blocks of the first verification data block, the discreteness of the second erasure data is improved. At the same time, the secondary erasure of the second verification data block increases the data allowed to be lost in the second erasure data, thereby improving the fault tolerance of the erasure code storage. In addition, the second source data block and the second verification data block in the second erasure data verify each other, which can reduce the amount of data read and the number of reads, and improve the IO performance of the storage server.

[0101] The data processing device based on the erasure code provided by the present invention is described below. The data processing device based on the erasure code described below and the data processing method based on the erasure code described above can refer to each other.

[0102] The embodiment of the present invention also provides a data processing device based on erasure code, Figure 5 is a schematic diagram of the structure of a data processing device based on erasure codes provided in an embodiment of the present invention. Figure 5 As shown, the data processing device 500 based on erasure code includes: a first erasure module 510, a segmentation module 520, a second erasure module 530 and a storage module 540, wherein:

[0103] A first erasure correction module 510, configured to determine first erasure correction data corresponding to the data to be processed based on an erasure correction code algorithm, wherein the first erasure correction data includes at least one first source data block and at least one first check data block;

[0104] A segmentation module 520 is used to splice all the first verification data blocks and segment the spliced ​​first verification data blocks to obtain at least two segmented second verification data blocks, wherein the number of segments of the second verification data blocks is greater than the number of segments of the first verification data blocks;

[0105] A second erasure correction module 530, configured to perform erasure correction coding on the at least two second verification data blocks, and determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0106] The storage module 540 is configured to determine second erasure correction data based on the at least one first source data block, the at least two second verification data blocks, and the at least one third verification data block, and to discretely store the second erasure correction data.

[0107] The data processing device based on the erasure code provided by the embodiment of the present invention determines the first erasure data corresponding to the data to be processed through the erasure code algorithm, then splices all the first verification data in the first erasure data and divides them, and performs erasure coding on the second verification data block obtained after the division to obtain at least one third verification data block, and determines the second erasure data according to the first source data block, the second verification data block and the third verification data block, and stores the second erasure data discretely. Since the number of the obtained second verification data blocks is greater than the number of blocks of the first verification data block, the discreteness of the second erasure data is improved. At the same time, the secondary erasure of the second verification data block increases the data allowed to be lost in the second erasure data, thereby improving the fault tolerance of the erasure code storage. In addition, the second source data block and the second verification data block in the second erasure data verify each other, which can reduce the amount of data read and the number of reads, and improve the IO performance of the storage server.

[0108] Optionally, the segmentation module 520 is specifically configured to:

[0109] Determining the number of blocks corresponding to the at least one first source data block;

[0110] Determine a target number of divisions for dividing the spliced ​​first verification data block according to a difference between the number of hard disks in the target storage pool in the storage server and the number of blocks corresponding to the at least one first source data block;

[0111] When the target segmentation number is greater than the number of blocks of the first verification data block, the spliced ​​first verification data block is segmented based on the target segmentation number to obtain the target segmentation number of second verification data blocks after segmentation.

[0112] Optionally, the storage module 540 is specifically used for:

[0113] Determine a target source data block from the at least one first source data block, the number of blocks of the target source data block being the same as the number of blocks of the at least one third verification data block;

[0114] Based on the numbering sequence of each of the target source data blocks, each of the third verification data blocks is spliced ​​with the target source data block to obtain at least one spliced ​​source data block;

[0115] Determine at least two second source data blocks based on the spliced ​​source data block and other first source data blocks among all the first source data blocks except the target source data block, wherein the number of blocks of the second source data blocks is equal to the number of blocks of the first source data block;

[0116] Second erasure correction data is determined based on the at least one second source data block and the at least two second check data blocks.

[0117] Optionally, the erasure code-based data processing device 500 further includes a data recovery module, which is specifically used to:

[0118] In the event that a hard drive failure is detected, determining the number of blocks of lost data;

[0119] Determine a data type of the lost data; the data type is used to characterize that the lost data is a lost source data block and / or a lost check data block, the lost source data block belongs to the second source data block, and the lost check data block belongs to the second check data block;

[0120] The lost data is recovered based on the data type of the lost data.

[0121] Optional data recovery module, specifically used for:

[0122] In a case where the data type includes the lost source data block, and the lost source data block includes the first source data block, reading all second check data blocks and other first source data blocks except the lost source data block, and recovering the lost source data block;

[0123] In the case that the data type includes the lost source data block, and the lost source data block includes the spliced ​​source data block, all second check data blocks and other first source data blocks except the lost source data block are read, and the first source data block in the spliced ​​source data block is restored, and all second check data blocks and the third check data blocks that are not lost are read, and the third check data block in the spliced ​​source data block is restored.

[0124] Optional data recovery module, specifically used for:

[0125] In the case that the data type includes the lost check data block, all third check data blocks and the non-lost second check data blocks are read to recover the lost check data block.

[0126] Optional data recovery module, specifically used for:

[0127] In the case where the data type includes the lost source data block and the lost check data block, all second check data blocks and other first source data blocks except the lost source data block are read, or all second check data blocks, other first source data blocks except the lost source data block and non-lost third check data blocks are read to recover the lost source data block;

[0128] All third check data blocks and the non-lost second check data blocks are read, and the lost check data blocks are recovered.

[0129] Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a data processing method based on erasure codes, the method comprising:

[0130] Determine, based on an erasure code algorithm, first erasure data corresponding to the data to be processed, where the first erasure data includes at least one first source data block and at least one first check data block;

[0131] All the first verification data blocks are spliced ​​together, and the spliced ​​first verification data blocks are divided to obtain at least two second verification data blocks after division, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks;

[0132] Performing erasure coding on the at least two second verification data blocks to determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0133] Based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block, second erasure correction data is determined, and the second erasure correction data is discretely stored.

[0134] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0135] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data processing method based on erasure code provided by the above methods, the method includes:

[0136] Determine, based on an erasure code algorithm, first erasure data corresponding to the data to be processed, where the first erasure data includes at least one first source data block and at least one first check data block;

[0137] All the first verification data blocks are spliced ​​together, and the spliced ​​first verification data blocks are divided to obtain at least two second verification data blocks after division, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks;

[0138] Performing erasure coding on the at least two second verification data blocks to determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0139] Based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block, second erasure correction data is determined, and the second erasure correction data is discretely stored.

[0140] In another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the erasure code-based data processing method provided by the above methods is implemented, and the method includes:

[0141] Determine, based on an erasure code algorithm, first erasure data corresponding to the data to be processed, where the first erasure data includes at least one first source data block and at least one first check data block;

[0142] All the first verification data blocks are spliced ​​together, and the spliced ​​first verification data blocks are divided to obtain at least two second verification data blocks after division, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks;

[0143] Performing erasure coding on the at least two second verification data blocks to determine at least one third verification data block corresponding to the at least two second verification data blocks;

[0144] Based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block, second erasure correction data is determined, and the second erasure correction data is discretely stored.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on erasure code, characterized in that: include: Determine, based on an erasure code algorithm, first erasure data corresponding to the data to be processed, where the first erasure data includes at least one first source data block and at least one first check data block; All the first verification data blocks are spliced ​​together, and the spliced ​​first verification data blocks are divided to obtain at least two second verification data blocks after division, wherein the number of blocks of the second verification data blocks is greater than the number of blocks of the first verification data blocks; Performing erasure coding on the at least two second verification data blocks to determine at least one third verification data block corresponding to the at least two second verification data blocks; Based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block, second erasure correction data is determined, and the second erasure correction data is discretely stored.

2. The data processing method based on erasure code according to claim 1, characterized in that: The step of segmenting the spliced ​​first verification data block to obtain at least two segmented second verification data blocks includes: Determining the number of blocks corresponding to the at least one first source data block; Determine a target number of divisions for dividing the spliced ​​first verification data block according to a difference between the number of hard disks in the target storage pool in the storage server and the number of blocks corresponding to the at least one first source data block; When the target number of divisions is greater than the number of blocks of the first verification data block, the first verification data block is divided based on the target number of divisions to obtain the target number of divisions and the second verification data blocks after division.

3. The data processing method based on erasure code according to claim 1 or 2, characterized in that: The determining the second erasure correction data based on the at least one first source data block, the at least two second check data blocks and the at least one third check data block comprises: Determine a target source data block from the at least one first source data block, the number of blocks of the target source data block being the same as the number of blocks of the at least one third verification data block; Splicing each of the third verification data blocks with the target source data block to obtain at least one spliced ​​source data block; Determine at least two second source data blocks based on the spliced ​​source data block and other first source data blocks among all the first source data blocks except the target source data block, wherein the number of blocks of the second source data blocks is equal to the number of blocks of the first source data block; Second erasure correction data is determined based on the at least one second source data block and the at least two second check data blocks.

4. The data processing method based on erasure code according to claim 3, characterized in that: After discretely storing the second erasure data, the method further includes: In the event that a hard drive failure is detected, determining the number of blocks of lost data; Determine a data type of the lost data; the data type is used to characterize that the lost data is a lost source data block and / or a lost check data block, the lost source data block belongs to the second source data block, and the lost check data block belongs to the second check data block; The lost data is recovered based on the data type of the lost data.

5. The data processing method based on erasure code according to claim 4, characterized in that: The recovering the lost data based on the data type of the lost data includes: In a case where the data type includes the lost source data block, and the lost source data block includes the first source data block, reading all second check data blocks and other first source data blocks except the lost source data block, and recovering the lost source data block; In the case that the data type includes the lost source data block, and the lost source data block includes the spliced ​​source data block, all second check data blocks and other first source data blocks except the lost source data block are read, and the first source data block in the spliced ​​source data block is restored, and all second check data blocks and the third check data blocks that are not lost are read, and the third check data block in the spliced ​​source data block is restored.

6. The data processing method based on erasure code according to claim 4, characterized in that: The recovering the lost data based on the data type of the lost data includes: In the case that the data type includes the lost check data block, all third check data blocks and the non-lost second check data blocks are read to recover the lost check data block.

7. The data processing method based on erasure code according to claim 4, characterized in that: The recovering the lost data based on the data type of the lost data includes: In the case where the data type includes the lost source data block and the lost check data block, all second check data blocks and other first source data blocks except the lost source data block are read, or all second check data blocks, other first source data blocks except the lost source data block and non-lost third check data blocks are read to recover the lost source data block; All third check data blocks and the non-lost second check data blocks are read, and the lost check data blocks are recovered.

8. A data processing device based on erasure code, characterized in that: include: A first erasure correction module, configured to determine first erasure correction data corresponding to the data to be processed based on an erasure correction code algorithm, wherein the first erasure correction data includes at least one first source data block and at least one first check data block; A segmentation module, used for splicing all the first verification data blocks, and segmenting the spliced ​​first verification data blocks to obtain at least two segmented second verification data blocks, wherein the number of segments of the second verification data blocks is greater than the number of segments of the first verification data blocks; A second erasure correction module, configured to perform erasure correction coding on the at least two second verification data blocks, and determine at least one third verification data block corresponding to the at least two second verification data blocks; A storage module is used to determine second erasure correction data based on the at least one first source data block, the at least two second verification data blocks and the at least one third verification data block, and discretely store the second erasure correction data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method based on erasure code as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method based on erasure code as described in any one of claims 1 to 7 is implemented.

Images