Method, device and equipment for verifying fragmentation consistency of erasure code data and medium
By performing byte-by-byte summation and linear matching verification on erasure coding data fragments and parity fragments, the problem of low efficiency in erasure coding data fragment consistency verification is solved, achieving efficient fragment consistency verification with strong verification capabilities and fast calculation speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies struggle to efficiently verify the consistency of erasure coding data fragments, especially in distributed storage systems where consistency verification between erasure coding data fragments consumes significant amounts of network bandwidth and CPU resources.
Data blocks of a preset size are obtained by fragmenting erasure coding data and parity fragments, summing them byte by byte to obtain the parity code, and performing fragment consistency verification based on linear matching verification. Fragment consistency is identified by using the linear matching relationship between the parity codes.
It improves the efficiency of erasure coding data fragment consistency verification, reduces the consumption of network bandwidth and CPU resources, has high data verification capability, can identify byte interaction and random errors, and improves verification speed.
Smart Images
Figure CN121643779A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data verification, in particular to a method and device for verifying the consistency of erasure code data shards, equipment and medium. BACKGROUND
[0002] Distributed storage must ensure the consistency between data shards. According to normal logic, the data between each shard of distributed storage must be consistent. For example, a 3-copy distributed object will store three pieces of data on three hard disks, and the three pieces of data must be the same data. However, the distributed storage system is complex, and data migration is required in scenarios such as failure and expansion. During the process of garbage collection, data migration or organization may also occur. If any unexpected software bug (defect or error) occurs in any link, it may cause the consistency between data shards to be destroyed.
[0003] The distributed storage shard consistency data scanning technology first adds a 2-byte or 4-byte check code to the local data according to the equal granularity. The check code is calculated by reading the local data and compared with the check code recorded in the metadata to verify and detect the persistence of the stored data, and to determine whether a silent error has occurred. If a silent error occurs, the data of other shards is read and written to the local shard, and data repair can be performed. Further, after ensuring that the local check code and data are consistent, the check codes of multiple shards are read and compared to identify whether the check codes are equal in position, which can identify whether the data between the replicas is consistent.
[0004] However, for erasure code data, the data between shards is not the same by nature, so it is not possible to directly compare the check codes to identify whether the data between shards is consistent by reading the check codes between shards. If the data of each shard is read and the encoding and decoding of the erasure code are performed again, a large amount of network bandwidth and CPU (Central Processing Unit) resources will be consumed. Therefore, how to efficiently implement the shard consistency verification of erasure code data has become a problem to be solved. SUMMARY
[0005] The present application provides a method and device for verifying the consistency of erasure code data shards, equipment and medium, which has high data verification capability, strong verification capability for byte interaction, random error and multi-bit jump, and effectively improves the efficiency of erasure code data shard consistency verification.
[0006] In a first aspect, the present application provides a method for verifying the consistency of erasure code data shards, which comprises: The data blocks of preset size are obtained for each erasure code data slice and each check slice, and byte-by-byte summation is performed on the obtained data blocks respectively to obtain the check code of each erasure code data slice and each check slice. The slice consistency check of the erasure code data is performed based on linear matching verification between the check code of the erasure code data slice and the check code of the check slice.
[0007] In combination with the first aspect, in an implementation, the data blocks of preset size are obtained for each erasure code data slice and each check slice, and byte-by-byte summation is performed on the obtained data blocks respectively to obtain the check code of each erasure code data slice and each check slice, specifically including: The data blocks of preset size are obtained for each erasure code data slice, and the data blocks of preset size are obtained for the check slice P and the check slice Q; The byte-by-byte summation is performed on the data blocks of each erasure code data slice, and the byte-by-byte summation result of the data block corresponding to the current erasure code data slice is taken as the check code of the current erasure code data slice; The byte-by-byte summation is performed on the data blocks of the check slice P and the check slice Q, the byte-by-byte summation result of the data block corresponding to the check slice P is taken as the check code of the check slice P, and the byte-by-byte summation result of the data block corresponding to the check slice Q is taken as the check code of the check slice Q.
[0008] In combination with the first aspect, in an implementation, the slice consistency check of the erasure code data is performed based on linear matching verification between the check code of the erasure code data slice and the check code of the check slice, specifically including: It is judged whether the check code of the erasure code data slice and the check code of the check slice P satisfy a linear matching relationship, and whether the check code of the erasure code data slice and the check code of the check slice Q satisfy a linear matching relationship: If both are yes, it indicates that the erasure code data slices are consistent; If both are no, the determination of the damaged slice is performed based on a unified check method; If not both yes and not both no, the check slice corresponding to no is damaged.
[0009] In combination with the first aspect, in an implementation, when the sum of the products of the check code of each erasure code data slice and the corresponding coefficient is equal to the check code of the current check slice, and the corresponding coefficient corresponds to the encoding coefficient of the current check slice, it is determined that the check code of the erasure code data slice and the check code of the current check slice satisfy a linear matching relationship.
[0010] In combination with the first aspect, in an implementation, for if not both yes and not both no, the check slice corresponding to no is damaged, specifically: If the check code of the erasure code data slice and the check code of the check slice P satisfy the linear matching relationship, and the check code of the erasure code data slice and the check code of the check slice Q do not satisfy the linear matching relationship, it is determined that the check slice Q is damaged; If the check code of the erasure code data slice and the check code of the check slice P do not satisfy the linear matching relationship, and the check code of the erasure code data slice and the check code of the check slice Q satisfy the linear matching relationship, it is determined that the check slice P is damaged.
[0011] In combination with the first aspect, in an implementation manner, the determining of the damaged slice based on the unified check method specifically includes: The slices are sequentially eliminated, and it is determined that: If the check codes of the remaining slices satisfy the linear matching relationship after the current slice is eliminated, it is determined that the current slice is damaged; If the check codes of the remaining slices do not satisfy the linear matching relationship after each slice is eliminated, it is indicated that multiple slices are damaged.
[0012] In combination with the first aspect, in an implementation manner, when the damaged slice is determined based on the unified check method: When the eliminated slice is the erasure code data slice, if the check code of the remaining erasure code data slice and the check code of the check slice P satisfy the linear matching relationship, and the check code of the remaining erasure code data slice and the check code of the check slice Q satisfy the linear matching relationship, it is indicated that the current eliminated erasure code data slice is damaged, otherwise, it is indicated that multiple slices are damaged; When the eliminated slice is the check slice, if the check code of the erasure code data slice and the check code of the remaining check slice satisfy the linear matching relationship, it is indicated that the current eliminated check slice is damaged, otherwise, it is indicated that multiple slices are damaged.
[0013] The second aspect provides a slice consistency check device of erasure code data, which includes: A calculation module is configured to obtain data blocks of a preset size for each erasure code data slice and each check slice, and perform byte-by-byte summation on the obtained data blocks to obtain check codes of the erasure code data slices and the check slices; A check module is configured to perform slice consistency check of the erasure code data based on linear matching verification between the check codes of the erasure code data slices and the check codes of the check slices.
[0014] In a third aspect, an embodiment of the present application provides a device for checking consistency of shards of erasure code data, the device for checking consistency of shards of erasure code data comprising a processor, a memory, and a program for checking consistency of shards of erasure code data stored in the memory and executable by the processor, wherein the program for checking consistency of shards of erasure code data is executed by the processor to implement the steps of the method for checking consistency of shards of erasure code data.
[0015] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a program for checking consistency of shards of erasure code data, wherein the program for checking consistency of shards of erasure code data is executed by a processor to implement the steps of the method for checking consistency of shards of erasure code data.
[0016] The technical scheme provided by the embodiments of the present application has the following beneficial effects. By reading the check code information of the check shard, it is identified whether each shard of the check erasure code data is consistent, and compared with directly reading data, the reading speed is effectively improved, and the calculation speed of the present application is fast, which is basically equivalent to the accumulation summation speed; the present application has high data checking capability, and has strong checking capability for byte interaction, random error, and multi-bit jump, thereby effectively improving the erasure code data shard consistency checking efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 FIG. 1 is a flowchart of the method for checking consistency of shards of erasure code data according to an embodiment of the present application; Figure 2 FIG. 2 is an optimization diagram of accumulation calculation; Figure 3 FIG. 3 is a function module diagram of the device for checking consistency of shards of erasure code data according to an embodiment of the present application; Figure 4 FIG. 4 is a hardware structure diagram of the device for checking consistency of shards of erasure code data according to an embodiment of the present application. DETAILED DESCRIPTION
[0018] In order to enable those skilled in the art to better understand the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.
[0019] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.
[0020] In a first aspect, embodiments of this application provide a method for verifying the consistency of erasure code data fragments. For erasure codes calculated by linear matrix, such as Reed-Solomon (RS) codes and Cauchy Reed-Solomon codes, a data check code information is designed, which can identify whether the erasure code data fragments are consistent by reading the check code information of all erasure code fragments.
[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the fragment consistency verification method for erasure coding data in this application. Figure 1 As shown, the fragment consistency verification method for erasure coding data includes: S1: Obtain data blocks of a preset size for each erasure coding data fragment and each check fragment, and sum the data blocks byte by byte to obtain the check codes for each erasure coding data fragment and each check fragment. S2: Verify the consistency of erasure code data fragments by performing linear matching verification between the check codes of erasure code data fragments and the check codes of the check fragments.
[0022] To better explain the fragmentation consistency verification method of this application, the corresponding principles on which this application is based will be explained first.
[0023] For the erasure coding data fragments A, B, C, and D of the linear matrix, and the erasure coding check fragments P and Q, each data fragment is summed byte by byte to obtain the result. , , , , , ,So , , , The calculation of erasure coding should also be for , P and Q represent two different checksums in erasure coding, generated based on different operations.
[0024] Regarding the properties of linear erasure codes, the encoding process is linear, meaning the check data is a linear combination of data fragments. For example, for Reed-Solomon codes or other linear erasure codes, the check fragments can be represented as:
[0025]
[0026] in, , , , This represents the coding coefficient of P. , , , This represents the coding coefficient of Q.
[0027] Given the byte-by-byte summation property, assuming each slice is a byte array (or more generally, a vector), byte-by-byte summation is equivalent to calculating the sum of all bytes in the slice (i.e., the component sums of the vector): , , ,
[0028] ,
[0029] in, This represents the erasure coding data fragment A's first... Each byte has a similar meaning; other similar parameters have similar meanings. This indicates the total number of bytes in the current slice.
[0030] For the calculation of the sum of the check fragments, since Byte-by-byte summation can be assigned to each component: ; Since the summation is linear, we can change the order to get: ; Right now: ; Similarly: .
[0031] In summary, we can conclude that: yes , , , A linear combination of , with coefficients identical to those of P; yes , , , A linear combination of , with coefficients identical to those of Q.
[0032] Furthermore, in one embodiment, a data block of a preset size is obtained for each erasure coding data fragment and each check fragment, and the obtained data blocks are summed byte by byte to obtain the checksum of each erasure coding data fragment and each check fragment, specifically including: S101: Obtain a data block of a preset size for each erasure coding data fragment, and at the same time obtain a data block of a preset size for both parity fragment P and parity fragment Q; S102: Sum the data blocks of each erasure coding data segment byte by byte, and use the summation result of the data block corresponding to the current erasure coding data segment as the check code of the current erasure coding data segment; S102: Sum the data blocks of both parity fragment P and parity fragment Q byte by byte, and use the sum of the data blocks corresponding to parity fragment P as the check code of parity fragment P, and use the sum of the data blocks corresponding to parity fragment Q as the check code of parity fragment Q.
[0033] For example, take an equal-sized data block (e.g., 4KB) from each slice. Sum the data from each slice byte-by-byte. The sum of these bytes gives the checksum of the 4KB data. It's important to note that byte-by-byte summation is necessary because erasure coding typically performs matrix calculations at the byte level. If the erasure coding calculation granularity is not bytes, such as two bytes, then summation by two bytes is required. For instance, for erasure coding data slices A, B, C, and D of a linear matrix, and check slices P and Q, take an equal-sized 4KB data block from each slice, and sum the data from each slice byte-by-byte to obtain the checksum. , , , , , .
[0034] Furthermore, in one embodiment, the fragment consistency verification of erasure-coded data is performed based on the linear matching verification between the checksum of the erasure-coded data fragment and the checksum of the checksum fragment, specifically including: Determine whether the checksums of erasure-coded data fragments satisfy a linear matching relationship with the checksums of checksum fragment P, and whether the checksums of erasure-coded data fragments satisfy a linear matching relationship with the checksums of checksum fragment Q: If all are yes, it means that the erasure coding data fragments are consistent; If all are negative, then the damaged fragment is determined based on the unified verification method; If neither all are yes nor all are no, then the corresponding check segment for no is corrupted.
[0035] Specifically, if the sum of the products of the check codes and corresponding coefficients of each erasure code data segment equals the check code of the current check segment, and the corresponding coefficients correspond to the coding coefficients of the current check segment, then it is determined that the check codes of the erasure code data segments and the check codes of the current check segments satisfy a linear matching relationship.
[0036] Specifically, for cases where neither all are yes nor all are no, the corresponding check segment is corrupted. If the checksum of the erasure code data fragment satisfies a linear matching relationship with the checksum of the checksum of the checksum P, and the checksum of the erasure code data fragment does not satisfy a linear matching relationship with the checksum of the checksum of the checksum Q, then the checksum Q is determined to be corrupted. If the checksum of the erasure code data fragment does not satisfy a linear matching relationship with the checksum of the checksum of the checksum of the checksum of the checksum of the erasure code data fragment, but satisfies a linear matching relationship with the checksum of ...
[0037] Specifically, for example, for erasure coding data fragments A, B, C, and D of a linear matrix, and parity fragments P and Q, each fragment takes an equal-sized 4KB data block, and then the data blocks of each fragment are summed byte by byte to obtain the result. , , , , , Determine whether the following conditions are met:
[0038] ; If both of the above equations are satisfied, it means that the erasure coding data fragments are consistent; if The formula is satisfied. If the formula is not satisfied, it indicates that the verification fragment Q is corrupted. The formula does not satisfy the requirements. If the formula is satisfied, it indicates that the verification fragment P is corrupted; if formula and If none of the formulas are satisfied, then a unified verification method is needed to determine the damaged fragments.
[0039] In this application, the determination of damaged fragments is based on a unified verification method, specifically including: Remove one slice at a time and determine: If, after removing the current fragment, the checksums of the remaining fragments satisfy a linear matching relationship, then the current fragment is determined to be corrupted. If, after each fragment is removed, the checksums of the remaining fragments do not satisfy a linear matching relationship, it indicates that multiple fragments are corrupted, and it is impossible to determine which fragment is corrupted.
[0040] Specifically, when determining corrupted fragments based on the unified verification method: When erasure coding data fragments are removed, if the checksums of the remaining erasure coding data fragments and the checksums of ... When a parity fragment is removed, if the parity code of the erasure code data fragment and the parity code of the remaining parity fragments satisfy a linear matching relationship, it indicates that the currently removed parity fragment is corrupted; otherwise, it indicates that multiple fragments are corrupted.
[0041] For example, when determining corrupted fragments based on the unified verification method, if erasure code data fragment A is removed, if the equation... and If all conditions are met (the value of the coding coefficient at this time needs to correspond to the value of the corresponding coding coefficient after removing erasure code data fragment A when performing check fragment representation), then it means that erasure code data fragment A is corrupted; otherwise, it means that more than one fragment is corrupted (the corrupted fragments may all be erasure code data fragments, all be check fragments, or both erasure code data fragments and check fragments may be removed), and it is impossible to specifically determine which fragment is corrupted.
[0042] For example, when determining damaged fragments based on the unified verification method, if the verification fragment P is removed, the equation... If the condition is met, it means that the verification fragment P is corrupted; otherwise, it means that more than one fragment is corrupted, and it is impossible to determine which fragment is corrupted.
[0043] It should be further explained that, based on the corresponding principles described above, for a 4KB data block, the checksum of the accumulated sum is recorded byte by byte, with a maximum value of 255 per byte, totaling 4KB, a maximum of 1020×1024, requiring 20 bits of storage to avoid overflow. The checksum in this application must be the sum of the values, not a truncated sum. For example, if the sum X requires 20 bits to record, and only 16 bits are recorded, the actual stored value Y deviates from the expected value by XY. Since XY may not be equal across different fragments, using Y for calculation will no longer satisfy the linear relationship, leading to a misjudgment of inconsistent data fragments.
[0044] This application uses the direct summation of 4KB data to verify the consistency of erasure-coded data fragments. However, this simple summation method has significant drawbacks when verifying local data. For example, if two bytes are identical (one becomes 1 and the other becomes 0), verification is impossible. Similarly, swapping any two bytes also fails to verify consistency. To address this, a strong checksum, such as CRC32 or CRC16, can be calculated simultaneously with the cumulative sum. One checksum is used for inter-fragment consistency verification, and the other for local fragment data consistency verification. However, this design involves calculating the checksum twice, at least doubling the computational overhead. To further enhance the fragment data integrity verification capability while reducing performance overhead, this application also provides a lightweight auxiliary checksum calculation method. This involves calculating the cumulative sum byte-by-byte while simultaneously calculating the cumulative sum two bytes at a time, and recording the lower 12 bits of the result as auxiliary local consistency verification. Since only the lower 12 bits of the sum are used, to record some information from the higher bits, the 12-bit result is incremented by 1 each time data overflows from the higher bits. Additionally, to prevent the sum from becoming too large during accumulation, it is first wrapped 16 bits apart, resulting in a 16-bit wrapped sum that does not exceed 16 bits. Then, it is wrapped 12 bits apart again to obtain the final auxiliary local consistency check code. See also... Figure 2 The image shows an optimization of the accumulation calculation based on the Intel CPU's avx512. Since one instruction of avx512 can process 64 bytes, the 4KB data is divided into 64 processing steps, simultaneously calculating the sum of bytes and the sum of 16 bits. A sample code snippet is shown below: / * Sum of 64 uint8_t values * / m512i chunk =_mm512_ loadu_ si512(data + i); sum8 =_ mm512_ _add_ epi32 (sum8,_ _mm512_ sad_ epu8 (chunk,_ mm512_setzero_ si512())); / * Sum of 32 uint16_t values * / _m512i chunk16 =_ _mm512_ loadu_ si512(ptr + (i / 2)); sum16 =_ _mm512_ add_ epi16 (sum16, chunk16) .
[0045] The fragment consistency verification method for erasure coding data in this application identifies whether the fragments of the erasure coding data are consistent by reading the check code information of the verification fragments. Compared with directly reading the data, the reading speed is effectively improved, and the calculation speed of this application is fast, basically comparable to the cumulative summation speed. This application has a high data verification capability and strong verification capability for byte interaction, random errors, and multiple bit transitions. Furthermore, this application can also use a dual verification mechanism, and the storage overhead is still the same as CRC32, with less storage space.
[0046] Secondly, embodiments of this application also provide a fragment consistency verification device for erasure coding data.
[0047] In one embodiment, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the functional modules of the fragment consistency verification device for erasure coding data in this application. Figure 3 As shown, the fragment consistency verification device for erasure coding data includes: a calculation module and a verification module.
[0048] The calculation module is used to obtain data blocks of a preset size for each erasure coding data fragment and each check fragment, and to sum the obtained data blocks byte by byte to obtain the check codes for each erasure coding data fragment and each check fragment; the verification module is used to perform fragment consistency verification of erasure coding data based on the linear matching verification between the check codes of the erasure coding data fragments and the check codes of the check fragments.
[0049] Thirdly, embodiments of this application provide a fragment consistency verification device for erasure coding data. The fragment consistency verification device for erasure coding data can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.
[0050] Reference Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of the fragment consistency verification device for erasure coding data involved in the embodiments of this application. In the embodiments of this application, the fragment consistency verification device for erasure coding data may include a processor, a memory, a communication interface, and a communication bus.
[0051] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0052] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the erasure coding data fragmentation consistency verification device, as well as interfaces used for interconnecting the erasure coding data fragmentation consistency verification device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.
[0053] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0054] The processor can be a general-purpose processor, which can call the fragment consistency verification program for erasure coding data stored in memory and execute the fragment consistency verification method for erasure coding data provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the fragment consistency verification program for erasure coding data is called can refer to the various embodiments of the fragment consistency verification method for erasure coding data in this application, and will not be repeated here.
[0055] Those skilled in the art will understand that Figure 4 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0056] Fourthly, embodiments of this application also provide a computer-readable storage medium.
[0057] This application provides a fragmentation consistency verification program for erasure coding data stored on a computer-readable storage medium. When the fragmentation consistency verification program for erasure coding data is executed by a processor, it implements the steps of the fragmentation consistency verification method for erasure coding data as described above.
[0058] The method implemented when the fragment consistency verification procedure for erasure coding data is executed can be referred to in various embodiments of the fragment consistency verification method for erasure coding data in this application, and will not be repeated here.
[0059] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0060] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0061] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0062] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0063] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.
[0064] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for verifying the consistency of fragmented erasure coding data, characterized in that, The method for verifying the fragment consistency of the erasure coding data includes: For each erasure coding data fragment and each check fragment, a data block of a preset size is obtained, and the obtained data blocks are summed byte by byte to obtain the check code of each erasure coding data fragment and each check fragment. Consistency verification of erasure code data fragments is performed based on linear matching verification between the checksum of erasure code data fragments and the checksum of the checksum fragments.
2. The method of claim 1, wherein, The process of acquiring data blocks of a preset size for each erasure coding data fragment and each check fragment, and then summing the acquired data blocks byte by byte to obtain the checksums for each erasure coding data fragment and each check fragment, specifically includes: For each erasure coding data fragment, a data block of a preset size is obtained. At the same time, for both parity fragment P and parity fragment Q, a data block of a preset size is obtained. Each data block of each erasure coding data segment is summed byte by byte, and the sum of each data block corresponding to the current erasure coding data segment is used as the checksum of the current erasure coding data segment. The data blocks of both the parity fragment P and the parity fragment Q are summed byte by byte. The sum of the data blocks corresponding to the parity fragment P is used as the check code of the parity fragment P, and the sum of the data blocks corresponding to the parity fragment Q is used as the check code of the parity fragment Q.
3. The method of claim 2, wherein, The linear matching verification between the checksum of the erasure coding data fragment and the checksum of the checksum fragment, which is based on erasure coding data fragment consistency verification, specifically includes: Determine whether the checksums of erasure-coded data fragments satisfy a linear matching relationship with the checksums of checksum fragment P, and whether the checksums of erasure-coded data fragments satisfy a linear matching relationship with the checksums of checksum fragment Q: If all are yes, it means that the erasure coding data fragments are consistent; If all are negative, then the damaged fragment is determined based on the unified verification method; If neither all are yes nor all are no, then the corresponding check segment for no is corrupted.
4. The method of claim 3, wherein the parity data is Reed-Solomon parity data. If the sum of the products of the check codes and corresponding coefficients of each erasure code data fragment is equal to the check code of the current check fragment, and the corresponding coefficients correspond to the coding coefficients of the current check fragment, then it is determined that the check codes of the erasure code data fragments and the check codes of the current check fragments satisfy a linear matching relationship.
5. The method of claim 4, wherein, If neither all are yes nor neither are no, then the corresponding check fragment is corrupted. Specifically: If the checksum of the erasure code data fragment satisfies a linear matching relationship with the checksum of the checksum of the checksum P, and the checksum of the erasure code data fragment does not satisfy a linear matching relationship with the checksum of the checksum of the checksum Q, then the checksum Q is determined to be corrupted. If the checksum of the erasure code data fragment does not satisfy a linear matching relationship with the checksum of the checksum of the checksum of the checksum of the checksum of the erasure code data fragment, but satisfies a linear matching relationship with the checksum of ...
6. The method of claim 4, wherein, The determination of corrupt fragments based on the unified verification method specifically includes: Remove one slice at a time and determine: If, after removing the current fragment, the checksums of the remaining fragments satisfy a linear matching relationship, then the current fragment is determined to be corrupted. If, after each fragment is removed, the checksums of the remaining fragments do not satisfy a linear matching relationship, it indicates that multiple fragments are corrupted.
7. The method of claim 6, wherein, In the determination of the damaged fragment based on the uniform check method: When the removed is the erasure code data fragment, if the check code of the remaining erasure code data fragment and the check code of the check fragment P satisfy the linear matching relationship, and the check code of the remaining erasure code data fragment and the check code of the check fragment Q satisfy the linear matching relationship, it indicates that the current removed erasure code data fragment is damaged, otherwise, it indicates that multiple fragments are damaged. When the removed is the check fragment, if the check code of the erasure code data fragment and the check code of the remaining check fragment satisfy the linear matching relationship, it indicates that the current removed check fragment is damaged, otherwise, it indicates that multiple fragments are damaged.
8. A fragment consistency verification device for erasure coding data, characterized in that, The erasure code data fragment consistency checking device comprises: A calculation module is configured to obtain data blocks of a preset size for each erasure code data fragment and each check fragment, and perform byte-by-byte summation on the obtained data blocks to obtain the check code of each erasure code data fragment and each check fragment. A check module is configured to perform erasure code data fragment consistency checking based on the linear matching verification between the check code of the erasure code data fragment and the check code of the check fragment.
9. A fragment consistency verification device for erasure coding data, characterized in that, The erasure code data fragment consistency checking device comprises a processor, a memory, and an erasure code data fragment consistency checking program stored in the memory and executable by the processor, wherein when the erasure code data fragment consistency checking program is executed by the processor, the steps of the erasure code data fragment consistency checking method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an erasure code data fragment consistency checking program, wherein when the erasure code data fragment consistency checking program is executed by the processor, the steps of the erasure code data fragment consistency checking method according to any one of claims 1 to 7 are implemented.