Data processing method and device

Abstract

The embodiments of this specification provide a data processing method and device, wherein the data processing method includes: dividing the data to be processed into multiple data blocks; dividing the multiple data blocks into at least two data block sets according to the calculation rules of a preset verification algorithm; determining a data block group according to a set of adjacent data blocks in the at least two data block sets; processing the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group; and determining a target verification block of the data to be processed according to the initial verification block. Thus, the security of the data to be processed is ensured. And based on the target verification block, the initial verification block can be restored later, ensuring the security of the initial verification block; at the same time, the data redundancy ratio of the verification block data is reduced, further reducing the storage cost of the data.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of data processing technology, and in particular, to a data processing method. Background Art

[0002] With the development of computer technology, in order to ensure data reliability and low storage costs, erasure coding technology, which can minimize the system storage overhead while ensuring data reliability, has been widely used in the field of data storage technology. For example, when data is stored in multiple AZs (availability zones), each AZ is an independently managed physical data center. Erasure coding technology is used to configure corresponding erasure code data for user data, and user data and erasure code data are stored in multiple separate data centers. This ensures that when a single AZ encounters problems such as computer room failure or network equipment failure, resulting in user data loss, it can still be restored through erasure code data, thereby ensuring the security of user data.

[0003] However, in order to ensure the security of erasure coded data in the event of abnormalities in the computer room and the machine, the designed erasure coded data often has a higher data redundancy ratio, which increases the data storage cost to a certain extent. Summary of the invention

[0004] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0006] Divide the data to be processed into multiple data blocks;

[0007] According to a calculation rule of a preset verification algorithm, the plurality of data blocks are divided into at least two data block sets;

[0008] Determine a data block group according to adjacent data block sets in the at least two data block sets;

[0009] Processing the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group;

[0010] A target check block of the data to be processed is determined according to the initial check block.

[0011] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0012] A segmentation module is configured to segment the data to be processed into a plurality of data blocks;

[0013] A division module, configured to divide the plurality of data blocks into at least two data block sets according to a calculation rule of a preset verification algorithm;

[0014] A first determining module is configured to determine a data block group according to adjacent data block sets in the at least two data block sets;

[0015] A processing module, configured to process the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group;

[0016] The second determination module is configured to determine a target check block of the to-be-processed data according to the initial check block.

[0017] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0018] Memory and processor;

[0019] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method are implemented.

[0020] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the data processing method are implemented.

[0021] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the data processing method.

[0022] The data processing method provided in the present specification includes dividing the data to be processed into multiple data blocks; dividing the multiple data blocks into at least two data block sets according to the calculation rules of a preset verification algorithm; determining a data block group according to adjacent data block sets in the at least two data block sets; processing the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group; and determining a target verification block of the data to be processed according to the initial verification block.

[0023] Specifically, the method determines a data block set based on the data blocks after the data to be processed is segmented; then determines a data block group based on adjacent data block sets in at least two data block sets, and then processes the data block group according to a preset verification algorithm to obtain an initial verification block of the data block group, thereby facilitating the subsequent recovery of the data to be processed based on the initial data block, thereby ensuring the security of the data to be processed. In addition, the target verification block of the data to be processed is determined based on the initial verification block, thereby facilitating the subsequent recovery of the initial verification block based on the target verification block, thereby ensuring the security of the initial verification block; at the same time, the data redundancy ratio of the verification block is reduced, further reducing the data storage cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of a design scheme for erasure code data provided by an embodiment of this specification;

[0025] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0026] Figure 3 is a schematic diagram of an erasure code in a data processing method provided by an embodiment of this specification;

[0027] Figure 4 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0028] Figure 5 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0029] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0030] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0031] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0033] First, the terms involved in one or more embodiments of this specification are explained.

[0034] Erasure Code: It is a coding fault-tolerant technology. Its basic principle is to divide the stored data into pieces and generate k+m pieces of data through a certain verification calculation method from k pieces of original data. And it can restore the original data through any k pieces of data among the k+m pieces. In this way, even if part of the data is lost, the system can still restore the original data.

[0035] Availability Zone (AZ): refers to a physical area in the same region with independent power and network. For example, an AZ can be an independently managed physical data center. The network latency between instances in the same availability zone is smaller.

[0036] Multi-AZ (Availability Zones): The multi-AZ solution distributes data to multiple separate data centers and ensures that data is still available when a single AZ encounters a computer room or network equipment failure.

[0037] Data redundancy ratio: The erasure code uses the erasure code algorithm to segment the data into k data blocks and then encode them to generate m redundant check blocks to achieve fault tolerance. The total storage data accounts for the multiple of the original data, that is, (k+m) / k, which is the data redundancy ratio of the erasure code.

[0038] With the continuous development of computer technology, the storage scale of distributed systems is becoming larger and larger; and device errors in distributed systems are also a problem that cannot be ignored. Therefore, the storage cost and reliability of data are factors that need to be considered when designing distributed systems. Erasure code files can minimize the storage overhead of the system while ensuring data reliability. Therefore, erasure code technology is widely used in the field of storage technology.

[0039] As users' requirements for data reliability continue to increase, many data storage systems support data storage in multiple AZs. Each AZ is an independently managed physical data center. When data is stored in multiple AZs, user data is distributed to multiple separate data centers, ensuring that when a single AZ encounters a computer room or network device failure, user data can still be accessed.

[0040] At the same time, in order to reduce the storage space occupied by data in multi-AZ storage, erasure codes are widely used in multi-AZ data storage in data storage systems. When a single AZ has a computer room or network equipment failure, the system can obtain data from other computer rooms and recover the abnormal data to provide it to users. However, in order to ensure the security of erasure coded data in the event of computer room and machine abnormalities, the erasure code design under multi-AZ often has a higher data redundancy ratio, which increases the data storage cost to a certain extent.

[0041] See also Figure 1 , Figure 1 is a schematic diagram of a design scheme for erasure code data provided by an embodiment of the present specification, wherein: AZ is an available area, A1-A10 is a data block obtained after user data is segmented, P(1-1) and P(2-1) are local check blocks generated by A1-A5 using RS erasure code calculation; P(1-2) and P(2-2) are local check blocks generated by A6-A10 using the same RS erasure code calculation; X1-X7 are local check blocks generated by RS erasure code calculation for corresponding replicas in AZ1 and AZ2, for example, X1 is calculated using A1 and A6; however, this design scheme has the problem of high data redundancy, such as Figure 1 As shown in the figure, when the number of local check blocks in a single AZ is set to 1, the system can only tolerate the loss of any 3 local check blocks (assuming that P(2-1), P(2-2), and X7 check blocks do not exist, then when data blocks A1, A2, A6, and A7 are lost, these four data blocks cannot be recovered). Therefore, in order to tolerate any 4 replica errors, two local check blocks need to be stored in each data AZ, which means that 11 check blocks need to be generated for 10 data blocks, resulting in a high data redundancy ratio, which can be as high as 2.1.

[0042] Based on this, in this specification, a data processing method is provided. This specification also involves a data processing device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0043] Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0044] Step 202: Divide the data to be processed into multiple data blocks.

[0045] In practical applications, the data processing method can be applied to a data storage system that can support data storage in multiple AZs (availability zones); for example, the data storage system can be a distributed storage system that supports data storage in multiple AZs.

[0046] The data to be processed can be understood as data that needs to be stored, such as documents or multimedia files sent by users that need to be stored. The data blocks are obtained by segmenting the data to be processed, and the data blocks contain part of the data content in the data to be processed. For example, if the data to be processed sent by the user is 10MB, it can be segmented into 10 data blocks, and the size of each data block is 1MB.

[0047] Specifically, when receiving the data to be processed, the data storage system can segment the data to be processed to obtain multiple data blocks.

[0048] In the specific implementation process, the data storage system can segment the data to be processed based on the calculation rules of the preset verification algorithm, so that the multiple data blocks obtained after segmentation can be dispersedly stored in the data storage center. The specific implementation method is as follows.

[0049] The step of dividing the data to be processed into a plurality of data blocks includes:

[0050] Receive data to be processed sent by users;

[0051] The data to be processed is divided into a plurality of data blocks based on the calculation rules of a preset verification algorithm.

[0052] Among them, the preset check algorithm can be understood as an algorithm that can calculate the check block, for example, an erasure code algorithm. In practical applications, the preset check algorithm can be an RS erasure code algorithm (Reed-Solomon erasure code), an array erasure code, a low-density parity check erasure code, etc. The following takes the preset check algorithm as an RS erasure code algorithm as an example to illustrate the data processing method provided in this specification. In the case where the preset check algorithm is an array erasure code, or a low-density parity check erasure code, you can refer to the corresponding or corresponding description content of this specification, and this specification will not go into too much detail on this.

[0053] Correspondingly, the calculation rule can be understood as the erasure code encoding configuration of the preset verification algorithm in the process of calculating the verification block each time. For example, the erasure code encoding configuration can be (5,1) and (2,1), where (5,1) means that the preset verification algorithm can calculate 5 data blocks each time to obtain 1 verification block. The (2,1) means that the preset verification algorithm can calculate 2 data blocks each time to obtain 1 verification block.

[0054] The data storage center may be able to store the data to be processed and the initial check blocks and target check blocks corresponding to the data to be processed; for example, when the data storage system to which the data processing method is applied can support data storage in a multi-AZ manner, the data storage center can be understood as AZ, physical data center, computer room or database, etc.

[0055] Specifically, when the data processing system receives the data to be processed sent by the user, it can determine the calculation rules of the preset verification algorithm pre-configured in the data processing system, and divide the data to be processed based on the technical rules to obtain multiple data blocks.

[0056] The following takes the application of the data processing method in a data storage scenario as an example to further illustrate the segmentation of the data to be processed based on the number of data storage centers, wherein the data storage system may be a distributed storage system, the data to be processed is a document, the preset verification algorithm may be an RS erasure code algorithm, and the calculation rule is an erasure code encoding configuration (5,1).

[0057] Based on this, when the distributed storage system receives a 10MB document sent by a user, it determines the erasure code configuration of the pre-configured RS erasure code algorithm, which is (5,1). Based on the erasure code configuration, it is determined that the RS erasure code algorithm generates 1 check block for 5 data blocks. Based on this, the distributed storage system divides the 10MB document into 10 data blocks, so that the subsequent RS erasure code algorithm configures the corresponding check block for the 10MB document sent by the user.

[0058] Furthermore, in a specific implementation process, the data storage system can segment the data to be processed based on the number of data storage centers, so that the multiple data blocks obtained after segmentation can be dispersedly stored in the data storage centers. The specific implementation method is as follows.

[0059] The step of dividing the data to be processed into a plurality of data blocks includes:

[0060] Receive data to be processed sent by users and determine the number of data storage centers;

[0061] The data to be processed is divided into a plurality of data blocks based on the number of the data storage centers.

[0062] Among them, the data storage center can store the data to be processed and the initial check blocks and target check blocks corresponding to the data to be processed; for example, when the data storage system to which the data processing method is applied can support data storage in a multi-AZ manner, the data storage center can be understood as AZ, physical data center, computer room or database, etc.

[0063] Specifically, when the data storage system receives the data to be processed sent by the user, the system determines the number of data storage centers, and divides the data to be processed based on the number of data storage centers, thereby obtaining multiple data blocks.

[0064] Using the above example, the data storage system may be a distributed storage system, the data storage center is AZ, and the data to be processed is a document.

[0065] The user sends a 10MB document to the distributed storage system. After receiving the document sent by the user, the distributed storage system determines the number of AZs, which may be 3. The distributed storage system then divides the 10MB document into 10 data blocks according to the number of AZs, where the size of each data block can be set according to the actual application scenario, for example, the data block size may be 1MB.

[0066] In actual applications, the 10MB document sent by the user can also be segmented according to the data block size pre-defined by the distributed system. For example, the data block size in the distributed storage system is 2MB; based on the data block size, the 10MB document can be segmented into 5 data blocks.

[0067] Step 204: Divide the multiple data blocks into at least two data block sets according to the calculation rules of the preset verification algorithm.

[0068] The data block set can be understood as a set consisting of a specific number of data blocks; in practical applications, a preset verification algorithm can process a data block set to obtain a verification block of the data block set.

[0069] Specifically, after dividing the data to be processed into a plurality of data blocks, the data storage system can determine the calculation rule of the preset verification algorithm, and divide the plurality of data blocks into at least two data block sets based on the calculation rule.

[0070] Using the above example, the preset verification algorithm can be the RS erasure code algorithm. Based on this, after the distributed storage system divides the 10MB document into 10 data blocks of 1MB, it can determine the erasure code configuration of the erasure code algorithm. The erasure code configuration can be (5,1); based on the erasure code configuration, the 10 data blocks are divided into two data block sets, each of which contains 5 data blocks. See Figure 3 , Figure 3 It is a schematic diagram of an erasure code in a data processing method provided by an embodiment of the present specification, wherein A1-A5 and A6-A10 are respectively divided data block sets, the AZ is an available area, A1-A10 are data blocks obtained after the user data is segmented, P(1-1) and P(1-2) are local check blocks generated by calculating the data block set using RS erasure code; X1-X5 are global check blocks generated by calculating the corresponding replicas in AZ1 and AZ2 using RS erasure code; X6 is a check block generated by calculating X1-X5 using RS erasure code, and P is a check block generated by calculating P(1-1) and P(1-2) using RS erasure code.

[0071] It should be noted that this manual only Figure 3 Taking the data processing method as an example to explain the data processing method, the parameters such as the number and size of the AZ (available area), data block, data block set and check block can be set according to the actual application scenario, and this manual does not make any specific restrictions on this.

[0072] Step 206: Determine a data block group according to adjacent data block sets in the at least two data block sets.

[0073] The data block group may be understood as a data block group consisting of a specific number of data block sets.

[0074] In one scenario provided in this specification, determining the data block group according to adjacent data block sets in the at least two data block sets includes:

[0075] At least two adjacent data block sets among the at least two data block sets are determined as a data block group.

[0076] Specifically, the data storage system determines at least two of the at least two data block sets, and two adjacent data block sets are determined as a data block group. That is, the data block group contains at least two data block sets, and each of the at least two data block sets is an adjacent data block set. At the same time, each data block set can only construct one data block group, that is, the data block set only exists in one data block group.

[0077] Using the above example, Figure 3As shown, the data block set A1-A5 and the data block set A6-A10 are adjacent data block sets. Based on this, the data storage system forms the data block set A1-A5 and the data block set A6-A10 into a data block group.

[0078] In the embodiment of the present specification, by determining at least two adjacent data block sets in at least two data block sets as a data block group, it is convenient for a subsequent preset verification algorithm to quickly process the data block group, thereby improving the efficiency of determining the initial verification block and the target verification block.

[0079] In another scenario provided in this specification, the determining the data block group according to adjacent data block sets in the at least two data block sets includes:

[0080] Determine at least two adjacent data block sets in the at least two data block sets, and position information of data blocks in the at least two adjacent data block sets;

[0081] The data block group is determined according to the position information of the data blocks in the at least two adjacent data block sets.

[0082] The location information of the data block may be understood as information indicating the location of the data block in the data block set.

[0083] Specifically, after determining the data block set, the data storage system can determine at least two adjacent data block sets in the at least two data block sets, and location information of the data blocks contained in the at least two adjacent data block sets, and then determine the data block group based on the location information of the data blocks in the at least two adjacent data block sets.

[0084] Further, determining the data block group according to the position information of the data blocks in the at least two adjacent data block sets includes:

[0085] Determine, according to the position information of the data blocks in the at least two adjacent data block sets, a corresponding relationship between the data blocks in each data block set in the at least two adjacent data block sets and the data blocks in other data block sets;

[0086] A data block group is determined according to the corresponding relationship.

[0087] Using the above example, Figure 3As shown, data block set A1-A5 and data block set A6-A10 are adjacent data block sets. Based on this, the data storage system can determine the corresponding relationship between the data blocks included in the data block set A1-A5 and the data blocks included in the data block set A6-A10 based on the location information of the data blocks in each data block set. For example, data block A1 has a corresponding relationship with data block A6. Then, based on the corresponding relationship, the data storage system constructs a data block group with the corresponding two data blocks, and the data block group includes data block A1 and data block A6.

[0088] The data processing method provided in the present specification determines a data block group based on the correspondence between the data blocks of each data block set in at least two adjacent data block sets and the data blocks of other data block sets, which is determined by the position information of the data blocks in at least two adjacent data block sets, so that a subsequent preset verification algorithm can quickly process the data block group, thereby improving the efficiency of determining the initial verification block and the target verification block.

[0089] Step 208: Process the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group.

[0090] Among them, the initial check block can be understood as a check block that can recover data from the data block group. In actual applications, when a specific number of data blocks in the data block group are lost, they can be recovered based on the initial check block to obtain a complete data block group, further ensuring the security of the data to be processed.

[0091] Specifically, after determining the data block group, the data storage system can calculate the data block based on a preset verification algorithm, thereby generating an initial verification block corresponding to the data block group.

[0092] In the case where at least two adjacent data block sets in at least two data block sets are determined as one data block group, in the data processing method provided in this specification, the preset verification algorithm is a first verification algorithm;

[0093] Accordingly, the processing of the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group includes:

[0094] Determine a set of data blocks in each data block group;

[0095] Each data block set in each data block group is processed according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group.

[0096] The first verification algorithm can be understood as an algorithm that can process a data block set to obtain an initial verification block; for example, when the data block set contains 5 data blocks, the first verification algorithm can be an RS erasure code algorithm that can calculate the 5 data blocks and obtain a verification block, that is, an RS erasure code algorithm with the erasure code encoding configuration of (5,1). Correspondingly, the initial verification block can be understood as an initial verification block obtained by calculating the first verification algorithm. For example, the initial verification block can be a verification block obtained after processing the data block set by the RS erasure code algorithm. In practical applications, the initial verification block can be generated by the RS erasure code algorithm with the erasure code encoding configuration of (5,1), which can be understood as a local verification block. At the same time, the number of initial verification blocks obtained by the first verification algorithm can be set according to the actual application scenario, and this specification does not impose specific restrictions on this. For example, the number of the initial verification blocks can be 1.

[0097] Specifically, after determining at least two adjacent data block sets in at least two data block sets as a data block group, the data storage system determines the data block sets in each data block group, and calculates the data block sets in each data block group according to the first verification algorithm, thereby obtaining an initial verification block for each data block set in each data block group, thereby facilitating the subsequent recovery of the data to be processed based on the initial data block, thereby ensuring the security of the data to be processed.

[0098] In practical applications, calculating the data block set in each data block group according to the first verification algorithm can be understood as calculating the data blocks included in the data block set in each data block group according to the first verification algorithm, and using the verification blocks obtained by calculating the data blocks as the initial verification blocks of each data block set in each data block group.

[0099] Using the above example, see Figure 3 After determining a data block group based on the two sets of data block set A1-A5 and data block set A6-A10, the distributed storage system can obtain the data block set included in the data block group, that is, the data block set A1-A5 and the data block set A6-A10, and calculate and process the data blocks included in the data block set through the RS erasure code, so as to obtain the local check code P(1-1) of the data block set A1-A5 and the local check code P(1-2) of the data block set A6-A10.

[0100] Further, the processing each data block set in each data block group according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group includes:

[0101] Determining matrix parameters of the first verification algorithm, and constructing a first verification matrix according to the matrix parameters;

[0102] Each data block set in each data block group is calculated according to the first check matrix to obtain an initial check block of each data block set in each data block group.

[0103] Among them, the matrix parameters can be understood as the parameters required to form the first check matrix, wherein the matrix parameters can be linearly non-correlated parameters. In practical applications, the matrix parameters can be completely independent and linearly non-correlated RS erasure code parameters. Correspondingly, the first check matrix can be understood as a matrix that can calculate the initial check block of each data block set in each data block group.

[0104] Using the above example, after determining the data block set in each data block group, the distributed storage system can determine the RS erasure code parameters of the RS erasure code algorithm. The RS erasure code parameters are completely independent and linearly non-correlated parameters. Based on the RS erasure code parameters, an RS erasure code matrix for calculating the data block set is constructed. The matrix can be<a1,a2,a3,a4,a5> or<b1,b2,b3,b4,b5> , wherein a1 to a5 represent each row in a matrix, and b1 to b5 represent each column in a matrix.

[0105] The distributed storage system then combines the data block set A1-A5 with<a1,a2,a3,a4,a5> Multiply them together to obtain the local check block P(1-1), that is, P(1-1) is A1–A5 with parameters<a1,a2,a3,a4,a5> The RS erasure code is calculated and generated. The data block set A6-A10 and<b1,b2,b3,b4,b5> Multiply them together to obtain the local checksum P(1-2), that is, P(1-2) is the parameter adopted by A6–A10<b1,b2,b3,b4,b5> The RS erasure code is calculated and generated.

[0106] It should be noted that the local check blocks in each AZ are generated by calculating completely independent and linearly non-correlated RS erasure code parameters. In addition, the rank of any submatrix of the above two sets of coding parameters is non-zero.

[0107] In the embodiments provided in the present specification, a first check matrix is ​​constructed according to the matrix parameters of the determined first check algorithm, and each data block set in each data block group is calculated according to the first check matrix to obtain an initial check block for each data block set in each data block group, thereby facilitating subsequent recovery of the data to be processed based on the initial data block and ensuring the security of the data to be processed.

[0108] In the case where a data block group is determined according to position information of data blocks in at least two adjacent data block sets, in the data processing method provided in this specification, the preset verification algorithm is a second verification algorithm;

[0109] Accordingly, the processing of the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group includes:

[0110] Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters;

[0111] The data block group is calculated according to the second check matrix to obtain an initial check block of the data block group.

[0112] Among them, the data block group includes two corresponding data blocks. Correspondingly, the second verification algorithm can be understood as an algorithm that can process the data blocks in the data block group to obtain the initial verification block; for example, when the data block group contains 2 data blocks, the second verification algorithm can be an RS erasure code algorithm that can calculate the 2 data blocks and obtain the verification block, that is, the RS erasure code algorithm with the erasure code encoding configuration of (2,1). Correspondingly, the initial verification block can be understood as the initial verification block obtained by calculating the second verification algorithm, for example, the initial verification block can be the verification block obtained after the data block set is processed by the RS erasure code algorithm. In practical applications, the initial verification block can be generated by the RS erasure code algorithm with the erasure code encoding configuration of (2,1), which can be understood as a global verification block. In addition, the number of initial verification blocks obtained by the second verification algorithm can be set according to the actual application scenario, and this specification does not make specific restrictions on this. For example, the number of the initial verification blocks can be 1. The matrix parameters can be understood as the parameters required to form the second verification matrix, wherein the matrix parameters can be linearly non-correlated parameters. The second check matrix can be understood as a matrix constructed based on matrix parameters and capable of calculating an initial check block of a data block group.

[0113] Using the above example, see Figure 3 After determining the data block group, the distributed storage system can determine the RS erasure code parameters of the RS erasure code algorithm. Based on the RS erasure code parameters, an RS erasure code matrix for calculating the data block group is constructed. The matrix can be<a1,a2> , where a1 to a2 represent each row in a matrix respectively.

[0114] The distributed storage system then stores the data blocks and<a1,a2> Multiply by , thereby obtaining the global check block of the data block group, which can be specifically obtained by multiplying the data blocks A1 and A6 included in the data block group by<a1,a2> Multiply them together to obtain the global check block X1 of the data block group. That is to say, X1-X5 are calculated and generated by using RS erasure code for the corresponding copies in AZ1 and AZ2, that is, X1 is calculated and generated by A1 and A6.

[0115] In the embodiments provided in this specification, a second check matrix is ​​constructed by determining the matrix parameters of a second check algorithm; and the data block group is calculated according to the second check matrix to obtain an initial check block of the data block group, thereby facilitating subsequent recovery of the data to be processed based on the initial data block, thereby ensuring the security of the data to be processed.

[0116] Step 210: Determine a target check block of the data to be processed according to the initial check block.

[0117] Among them, the target check block can be understood as a check block that can recover data from the initial check block. In actual applications, during data recovery of the processed data, when the initial check block is lost, the initial check block can be recovered based on the target check block, and the processed data can be recovered based on the recovered initial check block.

[0118] In the data processing method provided by this specification, when each data block set in each data block group is processed according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group, in the data processing method provided by this specification, determining the target verification block of the data to be processed according to the initial verification block includes:

[0119] The initial check block of each data block set in each data block group is processed according to the second check algorithm to obtain the target check block of each data block group.

[0120] The target check block may be understood as a check block obtained by calculating the initial check block of each data block set by the second check algorithm.

[0121] Specifically, after determining the initial check block of each data block set, the data storage system can process the initial check block of each data block set in each data block group based on the target algorithm to obtain the target check block of each data block group.

[0122] Further, the processing of the initial check block of each data block set in each data block group according to the second check algorithm to obtain the target check block of each data block group includes:

[0123] Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters;

[0124] An initial check block of each data block set in each data block group is calculated according to the second check matrix to obtain a target check block of the data block group.

[0125] Using the above example, see Figure 3 After the distributed storage system determines the initial check block of each data block set in the data block group, that is, determines Figure 3 After P(1-1) and P(1-2) in , the distributed storage system can determine the RS erasure code parameters of the RS erasure code algorithm. Based on the RS erasure code parameters, an RS erasure code matrix for calculating the data block group is constructed. The matrix can be<a1,a2> , where a1 to a2 represent each row in a matrix respectively.

[0126] The distributed storage system then combines P(1-1) and P(1-2) with<a1,a2> Multiply by to obtain the target check block of the data block group ( Figure 3 That is, the check block P is generated by using P(1-1) and P(1-2) using the RS erasure code consistent with the above.

[0127] In the embodiment of the specification, the second check matrix is ​​constructed by determining the matrix parameters of the second check algorithm; and the initial check block of each data block set in each data block group is calculated according to the second check matrix to obtain the target check block of each data block group. This facilitates the subsequent recovery of the initial check block based on the target check block, ensuring the security of the initial check block; at the same time, the data redundancy ratio of the check block data is reduced, further reducing the storage cost of the data.

[0128] In the data processing method provided in this specification, when a second check matrix is ​​constructed according to matrix parameters of a second check algorithm to calculate a data block group to obtain an initial check block of the data block group, the data processing method provided in this specification, determining a target check block of the data to be processed according to the initial check block, further includes:

[0129] Determining matrix parameters of a first verification algorithm, and constructing a first verification matrix according to the matrix parameters;

[0130] The initial check blocks of the data block groups are calculated according to the first check matrix to obtain the target check blocks of each data block group.

[0131] The target check block may be understood as a check block obtained by calculating the initial check block of each data block group through the first check algorithm.

[0132] Using the above example, see Figure 3 After determining the initial check blocks of each data block group, that is, X1-X5, the distributed storage system can determine the RS erasure code parameters of the RS erasure code algorithm. The RS erasure code parameters are completely independent and linearly non-correlated parameters. Based on the RS erasure code parameters, an RS erasure code matrix for calculating the data block set is constructed. The matrix can be<c1,c2,c3,c4,c5> , where c1 to c5 represent each row in a matrix respectively.

[0133] The distributed storage system then compares the initial check blocks X1-X5 of the data block group with<c1,c2,c3,c4,c5> Multiply them together to obtain the check block X6, that is, X6 is X1-X5 using parameters<c1,c2,c3,c4,c5> The X6 is generated by calculating the erasure code of RS. Moreover, the X6 is generated by calculating the RS erasure code parameters which are completely independent and linearly unrelated.

[0134] In the specific implementation process, the<a1,a2,a3,a4,a5> ,<b1,b2,b3,b4,b5> as well as<c1,c2,c3,c4,c5> There are three different matrices, wherein the rank of any sub-matrix of the three groups of coding parameters is non-zero; and the parameters in each matrix can be set according to the actual application scenario, and this specification does not impose any specific restrictions on this.

[0135] In the embodiments of the present specification, a first check matrix is ​​constructed according to the matrix parameters of the determined first check algorithm; and an initial check block of each data block group is calculated according to the first check matrix to obtain a target check block of each data block group, thereby facilitating subsequent recovery of the initial check block based on the target check block, thereby ensuring the security of the initial check block; at the same time, the data redundancy ratio of the check block data is reduced, further reducing the data storage cost.

[0136] Furthermore, in the embodiments provided in this specification, when the data to be processed is lost, the data to be processed can be restored based on the target check block and the initial check block, thereby ensuring the security of the data to be processed. The specific implementation method is as follows.

[0137] After determining the target check block of the data to be processed according to the initial check block, the method further includes:

[0138] Receiving a data acquisition request sent by a user for the data to be processed;

[0139] When the data block of the to-be-processed data meets the data recovery condition, performing data recovery on the to-be-processed data according to the initial check block or the target check block;

[0140] The data to be processed after data recovery is sent to the user.

[0141] The data acquisition request may be understood as a request for acquiring the data to be processed, and the data recovery condition may be set according to the actual application scenario. For example, the data recovery condition may be that a data block or a check block corresponding to the data to be processed is lost.

[0142] Specifically, when the data storage system receives a data acquisition request sent by a user for data to be processed, it determines whether the data to be processed meets the data recovery conditions. If so, it recovers the data to be processed according to the initial check block or the target check block, and sends the recovered data to the user; if not, it does not perform data recovery on the data to be processed and directly sends the data to the user.

[0143] In practical applications, data recovery of the processed data through the initial check block can be understood as recovering the lost data blocks of the processed data through the initial check block; data recovery of the processed data through the target check block can be understood as recovering the lost initial data blocks through the target check block, and then recovering the lost data blocks of the processed data based on the recovered initial check block.

[0144] Using the above example, see Figure 3 When the distributed storage system receives a data acquisition request from a user for a document, the distributed storage system determines whether the data block and / or the initial check block of the document is lost, that is, Figure 3 Whether any data block in A1-10 and / or any check block in (X1-X6, P(1-1), P(1-2)) is lost.

[0145] When a data block is lost, try to use the local check block (P(1-1) and P(1-2)) or the global check block (X1-X5) to recover the data block. If the data block still cannot be repaired, use the local check block (P(1-1) and P(1-2)) and the global check block (X1-X5) to jointly repair the lost data copy (data block).

[0146] In actual applications, in the process of data recovery through local check blocks and / or global check blocks, it is also necessary to refer to the non-lost data blocks, that is, the local check blocks and / or global check blocks need to recover the lost data blocks based on the non-lost data blocks.

[0147] For example, when data block A1 is lost, the system can recover the lost data block A1 based on the non-lost data block A6 and the global check block X1, or use the non-lost data blocks A2-A5 and the local check block P(1-1) to recover the lost data block A1.

[0148] When data blocks A1, A2, A6, and A7 are lost, the system can recover these four lost data blocks based on the non-lost data blocks A3-A5, A8-A10 by jointly decoding the local check blocks P(1-1), P(1-2) and the global check blocks X1 and X2.

[0149] When a check block is lost, it can be restored based on the data blocks and / or the check blocks that have not been lost. For example, when the local check block P(1-1) is lost, the system can restore the lost local check block P(1-1) based on the data blocks A1-A5 that have not been lost. Alternatively, the lost local check block P(1-1) can be restored based on the local check block P(1-2) that has not been lost and the check block P.

[0150] When the check blocks and data blocks are lost at the same time, recovery can be performed based on the intact data blocks and the intact check blocks.

[0151] For example, when data blocks A5, A10, and local check blocks P(1-1), P(1-2) are lost, the system can recover the lost data blocks and check blocks based on the non-lost data blocks A1-A4, A6-A9 by using the global check block X5 and the check block P for joint decoding.

[0152] When data blocks A6, A7, and global check blocks X1, X2 are lost, the system can recover the lost data blocks and check blocks based on the non-lost data blocks A1-A2, A8-A10, local check blocks P(1-2) and global check blocks X3-X6.

[0153] When local check blocks P(1-1) and / or P(1-2) are lost, P(1-1) and / or P(1-2) can be restored through check block P based on the local check blocks that have not been lost; if any global check block among global check blocks X1-X5 is lost, the lost global check block can be restored through check block X6 based on the global check blocks that have not been lost.

[0154] In an embodiment of the present specification, when a data acquisition request sent by a user for data to be processed is received; when a data block of the data to be processed meets a data recovery condition, data recovery is performed on the data to be processed according to an initial check block or a target check block; the data to be processed after data recovery is sent to the user, thereby ensuring the security of the data to be processed and improving the user experience.

[0155] The data processing method provided in this specification determines a data block set based on the data blocks after the data to be processed is segmented; then determines a data block group based on the adjacent data block sets in at least two data block sets, and then processes the data block group according to a preset verification algorithm to obtain the initial verification block of the data block group, so as to facilitate the subsequent recovery of the data to be processed based on the initial data block, thereby ensuring the security of the data to be processed. In addition, the target verification block of the data to be processed is determined based on the initial verification block, so as to facilitate the subsequent recovery of the initial verification block based on the target verification block, thereby ensuring the security of the initial verification block; at the same time, the data redundancy ratio of the verification block data is reduced, further reducing the storage cost of the data.

[0156] The following combination Figure 4 , taking the application of the data processing method provided in this specification in the scenario of configuring erasure codes for user data as an example, the data processing method is further described. Figure 4 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0157] Step 402: The distributed storage system can receive user data written by a user.

[0158] The user data may be data such as documents, multimedia files, etc., for example, a document with a size of 10MB.

[0159] Step 404: The system divides the user data into multiple data blocks.

[0160] Specifically, when the distributed storage system receives a 10MB document sent by a user, it determines the erasure code configuration of the pre-configured RS erasure code algorithm, which is (5,1). Based on the erasure code configuration, it is determined that the RS erasure code algorithm generates 1 check block for 5 data blocks. Based on this, the distributed storage system divides the 10MB document into 10 data blocks, so that the subsequent RS erasure code algorithm configures the corresponding check block for the 10MB document sent by the user.

[0161] Step 406: The system divides the multiple data blocks into at least two data block sets.

[0162] Specifically, the distributed storage system determines the erasure code configuration of the RS erasure code algorithm, which may be (5,1). Based on the erasure code configuration, every 5 adjacent data blocks in the above 10 data blocks are divided into a set, thereby obtaining data block sets A1-A5 and data block sets A6-A10.

[0163] Step 408: The system divides at least two data block sets into data block groups.

[0164] Specifically, after obtaining the data block set, the distributed storage system divides at least two adjacent data block sets in the data block set into a data block group.

[0165] For example, the data block set A1-A5 and the data block set A6-A10 are adjacent data block sets, so the data block set A1-A5 and the data block set A6-A10 can be divided into one data block group.

[0166] Step 410: Generate global check blocks X1-X5 using data blocks A1-A10 by using RS erasure coding with erasure coding configuration (2, 1).

[0167] Specifically, after obtaining the data block group, first determine the correspondence between the data blocks included in each data block set in the data block group and the data blocks included in other data block sets. For example, data block A1 in data block set A1-A5 has a correspondence with data block A6 in data block set A6-A10.

[0168] Secondly, based on the RS erasure code with the erasure code configuration of (2, 1), multiple data blocks with an associated relationship are calculated to obtain global check blocks corresponding to the multiple data blocks. For example, data block A1 and data block A6 are calculated to obtain the global check block X1 corresponding to the data block A1 and data block A6. Based on this, A2-A5 and A7-A10 are calculated to obtain the global check blocks X1-X5 corresponding to the data blocks A1-A10.

[0169] Step 412: Construct coding matrix coding parameters in different AZs.

[0170] Specifically, an RS erasure code parameter of the RS erasure code algorithm is determined, and the RS erasure code parameter is a completely independent and linearly non-correlated parameter.

[0171] Based on the RS erasure code parameters, an RS erasure code matrix for calculating the data block is constructed. The RS erasure code matrix can be<a1,a2,a3,a4,a5> ,<b1,b2,b3,b4,b5> or<c1,c2,c3,c4,c5> , wherein a1 to a5 represent each row in a matrix, b1 to b5 represent each row in a matrix, and c1 to c5 represent each row in a matrix.

[0172] Among them, the rank of any submatrix of the local check block parameters of all AZs is non-0 (specifically, it can be generated using the Cauchy matrix).

[0173] Step 414: Generate local check blocks P(1-1), P(1-2), and X6 by using the local coding matrix coding parameters of AZ and the data blocks in each AZ.

[0174] Specifically, the distributed storage system combines the data block set A1-A5 with<a1,a2,a3,a4,a5> Multiply them together to obtain the local check block P(1-1), that is, P(1-1) is A1–A5 with parameters<a1,a2,a3,a4,a5> The RS erasure code is calculated and generated.

[0175] Combine the data block set A6-A10 with<b1,b2,b3,b4,b5> Multiply them together to obtain the local checksum P(1-2), that is, P(1-2) is the parameter adopted by A6–A10<b1,b2,b3,b4,b5> The RS erasure code is calculated and generated.

[0176] The initial check blocks X1-X5 of the data block group are<c1,c2,c3,c4,c5> Multiply them together to obtain the check block X6, that is, X6 is X1-X5 using parameters<c1,c2,c3,c4,c5> The check block generated by the erasure code calculation.

[0177] Should<a1,a2,a3,a4,a5> ,<b1,b2,b3,b4,b5> as well as<c1,c2,c3,c4,c5> are three different matrices, where the rank of any submatrix of the three sets of coding parameters is non-zero.

[0178] Step 416: Generate a global check block P using P(1-1) and P(1-2) through (2, 1) RS coding.

[0179] Specifically, after determining P(1-1) and P(1-2), P(1-1) and P(1-2) can be calculated based on the RS erasure code with the erasure code configuration of (2, 1) to obtain the global check block P corresponding to P(1-1) and P(1-2).

[0180] Step 418: When data is lost, try to recover the data using local check blocks and / or global check blocks to recover the lost data copy.

[0181] When a data block is lost, try to recover the lost data block based on the non-lost data block using local check blocks P(1-1) and P(1-2) or global check blocks X1-X5. If the data block still cannot be repaired, use the local check blocks P(1-1) and P(1-2) and the global check blocks X1-X5 to repair the lost data copy.

[0182] For example, when data blocks A1, A2, A6, and A7 are lost, the system can recover these four lost data blocks based on data blocks A3-A5, A8-A10, using local check blocks P(1-1), P(1-2) and global check blocks X1 and X2 for joint decoding. When local check blocks P(1-1) or P(1-2) are lost, P(1-1) or P(1-2) can be recovered through check blocks P and / or data blocks; if any check block in global check blocks X1-X5 is lost, the lost check block can be recovered through check block X6 and / or data blocks.

[0183] In practical applications, the above steps take the erasure code configuration of (19, 10, 3) as an example (19 represents the total number of copies, 10 represents the number of data blocks, and 3 represents the number of AZs). When data blocks A1, A2, A6, and A7 are lost, the system can use local check blocks P(1-1) and P(1-2) and global check blocks X1 and X2 to jointly decode and recover the four lost data blocks.

[0184] Compared with AZ encoding with two local copies, this scheme has the same copy fault tolerance and can tolerate the loss of any 5 copies of data, but has a lower data redundancy ratio of 1.9, which is less than the data redundancy ratio of 2.1 of AZ encoding with two local copies. At the same time, compared with AZ encoding with a single local copy, this scheme has higher data fault tolerance. The AZ encoding scheme with a single local copy can only tolerate the loss of any 3 copies, while this scheme can tolerate the loss of any 5 copies, further ensuring the security of the data.

[0185] In a data processing method provided in this specification, an erasure code data encoding scheme for reducing data redundancy ratio in a multi-availability zone environment is proposed. The data repair capability of the erasure code in a multi-AZ environment is improved and the data redundancy ratio of the multi-AZ erasure code is further reduced by adopting a joint decoding design of a local check block within an AZ and a global check block between AZs.

[0186] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0187] A segmentation module 502 is configured to segment the data to be processed into a plurality of data blocks;

[0188] A division module 504 is configured to divide the plurality of data blocks into at least two data block sets according to a calculation rule of a preset verification algorithm;

[0189] A first determining module 506 is configured to determine a data block group according to adjacent data block sets in the at least two data block sets;

[0190] The processing module 508 is configured to process the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group;

[0191] The second determination module 510 is configured to determine a target check block of the to-be-processed data according to the initial check block.

[0192] Optionally, the first determining module 506 is further configured to:

[0193] At least two adjacent data block sets among the at least two data block sets are determined as a data block group.

[0194] Optionally, the preset verification algorithm is a first verification algorithm;

[0195] Accordingly, the processing module 508 is further configured to:

[0196] Determine a set of data blocks in each data block group;

[0197] Each data block set in each data block group is processed according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group.

[0198] Optionally, the processing module 508 is further configured to:

[0199] Determining matrix parameters of the first verification algorithm, and constructing a first verification matrix according to the matrix parameters;

[0200] Each data block set in each data block group is calculated according to the first check matrix to obtain an initial check block of each data block set in each data block group.

[0201] Optionally, the second determining module 510 is further configured to:

[0202] The initial check block of each data block set in each data block group is processed according to the second check algorithm to obtain the target check block of each data block group.

[0203] Optionally, the second determining module 510 is further configured to:

[0204] Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters;

[0205] An initial check block of each data block set in each data block group is calculated according to the second check matrix to obtain a target check block of each data block group.

[0206] Optionally, the first determining module 506 is further configured to:

[0207] Determine at least two adjacent data block sets in the at least two data block sets, and position information of data blocks in the at least two adjacent data block sets;

[0208] The data block group is determined according to the position information of the data blocks in the at least two adjacent data block sets.

[0209] Optionally, the first determining module 506 is further configured to:

[0210] Determine, according to the position information of the data blocks in the at least two adjacent data block sets, a corresponding relationship between the data blocks in each data block set in the at least two adjacent data block sets and the data blocks in other data block sets;

[0211] A data block group is determined according to the corresponding relationship.

[0212] Optionally, the preset verification algorithm is a second verification algorithm;

[0213] Accordingly, the processing module 508 is further configured to:

[0214] Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters;

[0215] The data block group is calculated according to the second check matrix to obtain an initial check block of the data block group.

[0216] Optionally, the second determining module 510 is further configured to:

[0217] Determining matrix parameters of a first verification algorithm, and constructing a first verification matrix according to the matrix parameters;

[0218] An initial check block of the data block group is calculated according to the first check matrix to obtain a target check block of the data block group.

[0219] Optionally, the segmentation module 502 is further configured to:

[0220] Receive data to be processed sent by users and determine the number of data storage centers;

[0221] The data to be processed is divided into a plurality of data blocks based on the number of the data storage centers.

[0222] Optionally, the data processing device further includes a receiving module, and the receiving module is configured to:

[0223] Receiving a data acquisition request sent by a user for the data to be processed;

[0224] When the data block of the to-be-processed data meets the data recovery condition, performing data recovery on the to-be-processed data according to the initial check block or the target check block;

[0225] The data to be processed after data recovery is sent to the user.

[0226] The data processing device provided in this specification determines a data block set based on the data blocks after the data to be processed is segmented; then determines a data block group based on the adjacent data block sets in at least two data block sets, and then processes the data block group according to a preset verification algorithm to obtain the initial verification block of the data block group, so as to facilitate the subsequent recovery of the data to be processed based on the initial data block, thereby ensuring the security of the data to be processed. In addition, the target verification block of the data to be processed is determined based on the initial verification block, so as to facilitate the subsequent recovery of the initial verification block based on the target verification block, thereby ensuring the security of the initial verification block; at the same time, the data redundancy ratio of the verification block data is reduced, further reducing the storage cost of the data.

[0227] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0228] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0229] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0230] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0231] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0232] The processor 620 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor 620.

[0233] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0234] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0235] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0236] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0237] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0238] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0239] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0240] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0241] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0242] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Divide the data to be processed into multiple data blocks; According to a calculation rule of a preset verification algorithm, the plurality of data blocks are divided into at least two data block sets; Determine a data block group according to adjacent data block sets in the at least two data block sets; Processing the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group; A target check block of the to-be-processed data is determined according to the initial check block, wherein the target check block is a target check block of each data block group, and is obtained by calculating an initial check block of each data block set in each data block group.

2. The data processing method according to claim 1, wherein determining the data block group according to adjacent data block sets in the at least two data block sets comprises: At least two adjacent data block sets among the at least two data block sets are determined as a data block group.

3. The data processing method according to claim 2, wherein the preset verification algorithm is a first verification algorithm; Accordingly, the processing of the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group includes: Determine a set of data blocks in each data block group; Each data block set in each data block group is processed according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group.

4. The data processing method according to claim 3, wherein the processing of each data block set in each data block group according to the first verification algorithm to obtain an initial verification block of each data block set in each data block group comprises: Determining matrix parameters of the first verification algorithm, and constructing a first verification matrix according to the matrix parameters; Each data block set in each data block group is calculated according to the first check matrix to obtain an initial check block of each data block set in each data block group.

5. The data processing method according to claim 3, wherein determining the target check block of the data to be processed according to the initial check block comprises: The initial check block of each data block set in each data block group is processed according to the second check algorithm to obtain the target check block of each data block group.

6. The data processing method according to claim 5, wherein the initial check block of each data block set in each data block group is processed according to the second check algorithm to obtain the target check block of each data block group, comprising: Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters; An initial check block of each data block set in each data block group is calculated according to the second check matrix to obtain a target check block of each data block group.

7. The data processing method according to claim 1, wherein determining the data block group according to adjacent data block sets in the at least two data block sets comprises: Determine at least two adjacent data block sets in the at least two data block sets, and position information of data blocks in the at least two adjacent data block sets; The data block group is determined according to the position information of the data blocks in the at least two adjacent data block sets.

8. The data processing method according to claim 7, wherein determining the data block group according to the position information of the data blocks in the at least two adjacent data block sets comprises: Determine, according to the position information of the data blocks in the at least two adjacent data block sets, a corresponding relationship between the data blocks in each data block set in the at least two adjacent data block sets and the data blocks in other data block sets; A data block group is determined according to the corresponding relationship.

9. The data processing method according to claim 7, wherein the preset verification algorithm is a second verification algorithm; Accordingly, the processing of the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group includes: Determining matrix parameters of a second verification algorithm, and constructing a second verification matrix according to the matrix parameters; The data block group is calculated according to the second check matrix to obtain an initial check block of the data block group.

10. The data processing method according to claim 9, wherein the step of determining the target check block of the data to be processed according to the initial check block further comprises: Determining matrix parameters of a first verification algorithm, and constructing a first verification matrix according to the matrix parameters; An initial check block of the data block group is calculated according to the first check matrix to obtain a target check block of the data block group.

11. The data processing method according to claim 1, after determining the target check block of the data to be processed according to the initial check block, further comprising: Receiving a data acquisition request sent by a user for the data to be processed; When the data block of the to-be-processed data meets the data recovery condition, performing data recovery on the to-be-processed data according to the initial check block or the target check block; The data to be processed after data recovery is sent to the user.

12. A data processing device, comprising: A segmentation module is configured to segment the data to be processed into a plurality of data blocks; A division module, configured to divide the plurality of data blocks into at least two data block sets according to a calculation rule of a preset verification algorithm; A first determining module is configured to determine a data block group according to adjacent data block sets in the at least two data block sets; A processing module, configured to process the data block group according to the preset verification algorithm to obtain an initial verification block of the data block group; The second determination module is configured to determine a target check block of the to-be-processed data according to the initial check block, wherein the target check block is a target check block of each data block group, and is obtained by calculating an initial check block of each data block set in each data block group.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method described in any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images