Data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; obtaining an initial check block corresponding to the data block group; processing the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group; and replacing the initial check block based on the target check block. In the process of converting the data state of the initial data, only the initial check block corresponding to the initial data needs to be obtained, and the target check block generated based on the initial check block is used to replace the initial check block, thereby reducing the amount of data read and written, avoiding the need to read the entire initial data in the process of data state conversion, reducing the amount of data read and written, and reducing resource consumption.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a data processing method. BACKGROUND

[0002] With the development of computer technology, in order to ensure the reliability of data and lower storage cost, therefore, the erasure code technology which can minimize the storage overhead of the system under the premise of ensuring the reliability of data is widely used in the field of data storage technology. And in order to balance the storage overhead and access performance of data in different scenarios, when storing, the hot data which has a higher possibility of access is configured to write into the storage system using erasure code with a lower redundancy ratio, and the cold data which has no access for a long time is configured to write into the storage system using erasure code with a higher redundancy ratio, thereby reducing the storage space consumption of user data.

[0003] However, in actual application, when the stored hot data needs to be converted into cold data, the amount of data read and written is large, causing excessive consumption of system resources and leading to a decline in system performance. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a data processing method is provided, comprising:

[0006] determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets;

[0007] obtaining an initial check block corresponding to the data block group;

[0008] processing the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group;

[0009] replacing the initial check block with the target check block.

[0010] According to a second aspect of the embodiments of the present specification, a data processing apparatus is provided, comprising:

[0011] a determining module configured to determine a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets;

[0012] an obtaining module configured to obtain an initial check block corresponding to the data block group;

[0013] a processing module configured to process the initial check block corresponding to the data block group based on a target check algorithm, to determine a target check block corresponding to the data block group;

[0014] a replacing module configured to replace the initial check block based on the target check block.

[0015] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0016] a memory and a processor;

[0017] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the data processing method.

[0018] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the computer executable instructions, when executed by a processor, implement the steps of the data processing method.

[0019] According to a fifth aspect of the embodiments of the present specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the data processing method.

[0020] The data processing method provided by the present specification comprises determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; obtaining an initial check block corresponding to the data block group; processing the initial check block corresponding to the data block group based on a target check algorithm, to determine a target check block corresponding to the data block group; and replacing the initial check block based on the target check block.

[0021] Specifically, in the process of converting the data state of the initial data, only the initial check block corresponding to the initial data needs to be obtained, and the target check block generated based on the initial check block is used to replace the initial check block, thereby reducing the amount of data read and written, avoiding the need to read the entire initial data in the process of data state conversion, reducing the amount of data read and written, reducing resource consumption, and improving system performance. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present specification;

[0023] Figure 2 is a processing schematic diagram of a data processing method provided by an embodiment of the present specification;

[0024] Figure 3 is a schematic diagram of a target configuration encoding matrix in a data processing method provided by an embodiment of the present specification;

[0025] Figure 4 is a process flow diagram of a data processing method provided by an embodiment of the present specification;

[0026] Figure 5 is a structural schematic diagram of a data processing provided by an embodiment of the present specification;

[0027] Figure 6 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0028] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, set forth in this description. Those skilled in the art, in light of the description, can implement the present specification without limiting to the specific details disclosed in this description.

[0029] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. These terms are used only to distinguish one from another. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0031] First, the noun terms related to one or more embodiments of the present specification are explained.

[0032] Erasure Code: an error correction coding technology, which divides the stored data into fragments, and generates k+m data from k original data through certain check calculation, and can restore the original data from any k data. So even if some data is lost, the system can still recover the original data.

[0033] Data redundancy ratio: Erasure Code uses erasure code algorithm to segment the data and generate k data blocks, and then generates m redundant check blocks for error correction. The total data storage is k times the original data, i.e. (k+m) / k, which is the data redundancy ratio of the erasure code.

[0034] With the continuous development of computer technology, the storage scale of distributed systems (such as distributed storage systems) is becoming larger and larger; and the device error in the distributed system is also a problem that cannot be ignored. Therefore, the storage cost and reliability of data are factors that need to be considered in the design of distributed systems. Erasure code files can minimize the storage overhead of the system while ensuring data reliability; therefore, erasure code technology is widely used in the field of storage technology. Erasure code technology calculates M check data blocks from K original data blocks, and when any K data blocks exist, the system can use them to recover the missing data blocks. When M is fixed, the larger the value of K, the less the redundant storage space of the data occupies, but when data is lost, the amount of data to be read is larger, and the impact of data loss on user requests is greater; and the smaller the value of K, the greater the redundant storage space of the data occupies. But when the node is abnormal, the data recovery traffic is smaller. Erasure code is widely used in distributed storage systems, and in order to cope with more and more complex service scenarios and system environments, many institutions have designed different configurations of erasure code storage strategies for distributed storage systems to balance the storage overhead and access performance of data in different scenarios; for hot data that is more likely to be accessed by users after writing, a smaller K value erasure code method can be used for storage, and for cold data that has not been accessed for a long time, a larger K value erasure code configuration can be used to archive the data to reduce the storage space consumption of user data.

[0035] However, in the process of converting hot data to cold data, a scheme provided in the specification is to read out the original configuration of the user data and recalculate the check blocks, and write the user data and the newly generated check blocks into the system. The disadvantage of this scheme is that in the process of changing the erasure code configuration, there is data reading traffic and data writing traffic, which occupies a large amount of network bandwidth and causes large data traffic overhead.

[0036] Based on this, in the present specification, a data processing method is provided, and the present specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0037] Figure 1 A flowchart of a data processing method according to one embodiment of the present specification is shown, which specifically includes the following steps.

[0038] Step 102: determining a data block group corresponding to the initial data, wherein the data block group includes at least two data block sets.

[0039] In actual applications, the data processing method can be applied to a storage node or a data storage system; in the scenario where the data processing method is applied to a storage node, the storage node includes a CPU and a disk, and the data processing method provided by the present specification can be applied in the CPU, and the disk can store the data block group corresponding to the initial data and the initial check block. In the scenario where the data processing method is applied to a data storage system, the data storage system can be a distributed storage system. The distributed storage system can include a client and a data storage center. The data storage center can be understood as a center capable of storing data, for example, the data storage center can be an independently managed data center, a computer room, a database, etc.

[0040] Based on this, the initial data can be understood as data sent by a user to the data storage system for storage, for example, data such as documents or multimedia files sent by the user and needing to be stored. The data block is obtained by cutting the to-be-processed data, and the data block contains part of the data content in the to-be-processed data, for example, the to-be-processed data sent by the user is 8MB, which can be cut into 8 data blocks, each data block being 1MB in size. The data block set can be understood as a set composed of a specific number of data blocks, wherein the specific number of data blocks can be set according to actual scenarios. The data block group can be understood as a data block group composed of at least two data block sets.

[0041] Specifically, the data processing method provided in the specification can determine a data block group corresponding to initial data, the data block group including at least two data block sets, and the at least two data block sets included in the data block group can be adjacent. In actual application, the data processing method provided in the specification can determine a data block group corresponding to initial data through a CPU of a storage node from a disk of the storage node, or can determine a data block group corresponding to initial data through a client of a distributed system from a data storage center of a distributed storage system. In order to avoid excessive repetition, the following takes the scenario in which the data processing method provided in the specification is applied to a data storage system as an example to further explain the data processing method. For a specific description of the scenario in which the data processing method is applied to a storage node, reference can be made to the corresponding or corresponding description of the scenario in which the data processing method is applied to a data storage system.

[0042] In an embodiment of the specification, the data block group corresponding to the initial data can be found when the data state information of the initial data meets the preset processing condition, specifically as follows.

[0043] The determining of the data block group corresponding to the initial data includes:

[0044] obtaining data state information of the initial data;

[0045] finding the data block group corresponding to the initial data when it is determined that the data state information meets the preset processing condition.

[0046] The data state information can be understood as information representing the state of the initial data. For example, when the initial data is hot data, the data state information can be understood as information representing whether the hot data needs to be converted into cold data. In actual application, the conversion of hot data into cold data by the distributed storage system can be understood as the conversion of the erasure code configuration of the hot data into the erasure code configuration of the cold data. For example, the distributed storage system configures 2 parity blocks for a hot data (4 data blocks) and configures 2 parity blocks for a cold data (8 data blocks). Based on this, the process of converting the hot data into the cold data is to convert the configuration of 2 parity blocks for a hot data into the configuration of 2 parity blocks for two hot data (a total of 8 data blocks).

[0047] The preset processing condition can be set according to an actual application scenario, and the specification does not make specific limitations thereon. For example, in the case of data state information being data storage time, satisfying the preset processing condition can be that the data storage time satisfies a preset time threshold. The preset time threshold can be set according to an actual application scenario, and the specification does not make specific settings thereon. For example, the preset time threshold can be 1 day or 1 month. In the case of data state information being the number of data acquisition requests, satisfying the preset processing condition can be that the number of data acquisition requests is less than a preset number threshold. The preset number threshold can be set according to an actual application scenario, and the specification does not make specific settings thereon. For example, the preset number threshold can be 100 times.

[0048] Specifically, the data state information of the initial data is obtained from a state information storage unit. The state information storage unit can be understood as a unit for storing data state information, for example, the unit can be a memory, a file, or a database, etc.

[0049] After obtaining the data state information of the initial data, it is necessary to determine whether the data state information satisfies a preset processing condition. If yes, the data block group corresponding to the initial data is found, and the data block group includes at least two data block sets. If not, the data state information of the initial data is continuously obtained until the data state information satisfies the preset processing condition.

[0050] In the embodiments of the specification, the data block group corresponding to the initial data can be found from the data storage center, and the specific implementation is as follows.

[0051] The finding of the data block group corresponding to the initial data includes:

[0052] The data block group corresponding to the initial data is found from the data storage center.

[0053] Specifically, in the case where it is determined that the data state information of the initial data satisfies the preset processing condition, the data block group corresponding to the initial data can be found from the data storage center, and the data block group includes at least two data block sets.

[0054] In actual applications, the data processing method provided by the specification can find the data block group corresponding to the initial data from the data storage center based on the storage address of the initial data, and the specific implementation is as follows.

[0055] The finding of the data block group corresponding to the initial data from the data storage center includes:

[0056] The storage address corresponding to the initial data is obtained.

[0057] determine the data block group corresponding to the initial data from the data storage center based on the storage address.

[0058] The storage address can be understood as the address of the initial data in the data storage center. In actual application, the data storage system obtains the storage address of the initial data in the data storage center in the process of storing the initial data into the data storage center, so as to facilitate subsequent quick search for the initial data based on the storage address. Since the initial data is stored in the data storage center in the form of a data block group, finding the initial data can be understood as finding the data block group corresponding to the initial data from the data storage center.

[0059] Specifically, the data storage system determines the storage address corresponding to the initial data from the storage address recorded by itself, and determines the data block group corresponding to the initial data from the data storage center based on the storage address corresponding to the initial data.

[0060] In an embodiment provided in the specification, the data state information corresponding to the initial data needs to be determined before the client of the data storage system obtains the data state data. One way is to determine through the data acquisition request for the initial data, and the specific manner is as follows.

[0061] Before obtaining the data state information of the initial data, the method further comprises:

[0062] In the case of receiving the data acquisition request for the initial data, the number of data acquisition requests is recorded;

[0063] The number of data acquisition requests is determined as the data state information of the initial data.

[0064] The data acquisition request can be understood as a request to acquire the initial data, and the number of data acquisition requests can be understood as the number of acquisitions of the initial data.

[0065] Specifically, the client of the data storage system can record the number of data acquisition requests when receiving the data acquisition request for the initial data, and determine the data state information of the initial data based on the number of data acquisition requests.

[0066] In actual application, determining the data state information of the initial data based on the number of data acquisition requests can be understood as determining the number of acquisitions of the initial data within a specific time period as the data state information of the initial data. For example, a document is acquired 100 times in a day, and the record of the document being acquired 100 times in a day is taken as the data state information of the initial data. The specific time period can be set according to actual application scenarios.

[0067] The following takes the scenario of applying the data processing method to converting the parity configuration of data as an example to further illustrate taking the number of data acquisition requests as the data state information of initial data. The initial data is a document uploaded by a user, and the document is hot data. The data state information is the number of acquisitions of the initial data.

[0068] The distributed storage system records the number of data acquisition requests in the case of receiving a data acquisition request for a document uploaded by a user. When 1 data acquisition request for the document is received within a day, the number of data acquisition requests is 1, and when 100 data acquisition requests for the document are received within a day, the number of data acquisition requests is 100. The number of data acquisition requests is taken as the data state information of the document.

[0069] In the embodiments provided in the specification, the number of data acquisition requests recorded in the case of receiving a data acquisition request for initial data is taken as the data state information of the initial data, which facilitates subsequent accurate determination of converting the parity configuration of the initial data based on the data state information.

[0070] In another embodiment provided in the specification, another way is to determine the data state information of initial data through the data storage time of the initial data, and the specific manner is as follows.

[0071] Before obtaining the data state information of the initial data, the method further includes:

[0072] Determining the data storage time of the initial data, and taking the data storage time as the data state information of the initial data.

[0073] The data storage time can be understood as the time at which the client of the data storage system stores the initial data.

[0074] Following the above example, the distributed storage system can receive a document sent by a user and store the document. In the process of storing the document, the storage time of the document is recorded, and the storage time is taken as the data state information of the document.

[0075] In the embodiments provided in the specification, the data storage time of the initial data is determined, and the data storage time is taken as the data state information of the initial data, thereby facilitating subsequent accurate determination of converting the parity configuration of the initial data based on the data state information.

[0076] In an embodiment provided in the specification, before obtaining the data state information of the initial data, the client needs to receive the initial data and configure corresponding check codes for the initial data based on an initial erasure code configuration manner, which can be understood as an erasure code configuration manner for hot data. For example, when a document is received, the document is converted into a data block set containing multiple data blocks by using the erasure code configuration manner for hot data; two corresponding check blocks are configured for the data block set, and the data block set and the two corresponding check blocks are stored, so as to realize storage of the document in the erasure code configuration manner for hot data. Then, the erasure code configuration of the initial data can be converted only when the data state information meets the preset processing condition. The specific implementation manner of configuring corresponding check codes for the initial data based on the initial erasure code configuration manner is as follows.

[0077] Before obtaining the data state information of the initial data, steps 1 to 3 are further included.

[0078] Step 1: The initial data is divided into multiple data blocks, and at least two data block sets are determined based on the multiple data blocks.

[0079] Further, the initial data is divided into multiple data blocks, and at least two data block sets are determined based on the multiple data blocks, including:

[0080] The initial data is divided into multiple data blocks based on the calculation rule of the initial check algorithm, and the multiple data blocks are divided into at least two data block sets.

[0081] The initial check algorithm can be understood as an algorithm that can calculate a data block set and obtain a check block of the data block set. In actual application, the initial check algorithm can be an RS erasure code algorithm (Reed-Solomon type erasure code), an array erasure code, a low-density parity check erasure code, etc. The data processing method provided in the specification is described below by taking the initial check algorithm as the RS erasure code algorithm. In the case where the initial check algorithm is an array erasure code or a low-density parity check erasure code, refer to the corresponding or corresponding description content in the specification, and the specification will not be described in detail.

[0082] For example, the initial check algorithm can be an RS erasure code algorithm with an erasure code configuration of (4, 2). Correspondingly, the calculation rule can be understood as the erasure code configuration of the initial check algorithm in the process of calculating the check block each time. For example, the erasure code configuration can be (4, 2); the (4, 2) indicates that the initial check algorithm can calculate 4 data blocks each time to obtain 2 check blocks. In actual application, the initial check algorithm can process a specific number of data blocks to obtain the check block corresponding to the data block, where the specific number of data blocks can be set according to the actual scene. For example, a data block set containing 4 data blocks is processed to obtain the check block corresponding to the data block set.

[0083] Specifically, the client can divide the to-be-processed data into a plurality of data blocks, and then determine the calculation rule of the initial check algorithm, and divide the plurality of data blocks into at least two data block sets based on the calculation rule.

[0084] Following the above example, the initial check algorithm can be an RS erasure code algorithm with an erasure code configuration of (4, 2), based on which the distributed storage system can determine the erasure code configuration of the erasure code algorithm, which can be (4, 2); based on the erasure code configuration, the 8MB size document can be cut into 8 data blocks with a size of 1MB, and the 8 data blocks can be divided into two data block sets, each of which contains 4 data blocks, so as to facilitate the RS erasure code algorithm with an erasure code configuration of (4, 2) to calculate the data blocks matching the calculation rule thereof. Referring to Figure 2 , Figure 2 is a processing diagram of a data processing method provided by an embodiment of the present application, wherein A1-A8 are data blocks obtained after user data is cut, A1-A4 and A5-A8 are divided data block sets, and the data block set A1-A4 and the data block set A5-A8 constitute an original encoding group, the erasure code configuration of the data block set in the encoding group being (4, 2); P(1-1), P(2-1) and P(2-1), P(2-2) are check blocks generated by calculating the data block set by using the RS erasure code with an erasure code configuration of (4, 2); and P1, P2 are check blocks obtained by calculating A1-A8 by using the RS erasure code with an erasure code configuration of (8, 2).

[0085] In an implementation provided in the specification, the RS erasure code is configured as an (4, 2) RS erasure code, and the check blocks P(1-1), P(2-1) and P(2-1), P(2-2) are obtained by calculating the data blocks through a check matrix. The matrix parameters for constructing the check matrix are obtained from the matrix parameters of the RS erasure code configured as an (8, 2) RS erasure code. Based on this, when the erasure code configuration of the hot data is converted to the erasure code configuration of the cold data, the RS erasure code configured as an (8, 2) RS erasure code can directly calculate the check blocks P(1-1), P(2-1) and P(2-1), P(2-2) corresponding to the hot data, thereby obtaining the check blocks P1 and P2. That is, the P1 and P2 are the check blocks generated by calculating P(1-1), P(2-1) and P(2-1), P(2-2) through the RS erasure code configured as an (8, 2) RS erasure code.

[0086] It should be noted that only the Figure 2 is used as an example to further explain the data processing method in the specification. The number and size of the data blocks, the data block set, and the check block can be set according to the actual application scenario, and the specification does not make specific limitations.

[0087] In the embodiments of the specification, the initial data is divided into a plurality of data blocks through the calculation rule of the initial check algorithm, and the plurality of data blocks are divided into at least two data block sets, so that the initial check block of the initial data can be determined based on the data block set.

[0088] Step two: determining at least two data block sets in the at least two data block sets as a data block group.

[0089] The data block group can be understood as a data block group composed of at least two data block sets. The data block set in the data block group can be adjacent data block sets.

[0090] Specifically, the client determines at least two data block sets in the at least two data block sets as a data block group. That is, the data block group includes at least two data block sets, and each data block set in the at least two data block sets can be adjacent data block sets. Each data block set can only construct one data block group, that is, one data block set only exists in one data block group.

[0091] As shown in the above example, Figure 2 The data block set A1-A4 and the data block set A5-A8 are adjacent data block sets. Based on this, the client constructs a data block group with the data block set A1-A4 and the data block set A5-A8, that is, Figure 2 the original encoding group in the specification.

[0092] In the embodiments of the present specification, at least two adjacent data block sets in at least two data block sets are determined as a data block group, so that the subsequent target check algorithm and the initial check algorithm can quickly process the data block group, thereby improving the efficiency of determining the initial check block and the target check block.

[0093] Step three: processing the data block group based on the initial check algorithm and the target check algorithm to obtain the initial check block corresponding to the data block group.

[0094] The target check algorithm can be understood as an algorithm that can process the data block group and obtain the target check block of the data block. For example, the RS erasure code algorithm with (8, 2) erasure code configuration. The (8, 2) indicates that the target check algorithm can calculate 8 data blocks at a time, thereby obtaining 2 check blocks. In actual application, the target check algorithm can process a data block group, thereby obtaining the check block of the data block group.

[0095] Further, the processing of the data block group based on the initial check algorithm and the target check algorithm to obtain the initial check block corresponding to the data block group comprises:

[0096] determining the initial check algorithm and the target check algorithm corresponding to the initial data;

[0097] determining the matrix parameters of the target check algorithm, and constructing a target check matrix based on the matrix parameters;

[0098] obtaining at least two data block sets in the data block group;

[0099] processing each data block set based on the target check matrix and the initial check algorithm to obtain the initial check block corresponding to the data block group.

[0100] The matrix parameters can be understood as the parameters required to construct the target check matrix. Correspondingly, the target check matrix can be understood as a matrix that can calculate the target check block of each data block group.

[0101] The check block can be understood as a check block capable of recovering data blocks, and the initial check block can be understood as a check block determined based on the initial check algorithm. For example, in the case of RS erasure code algorithm with initial check data and erasure code configuration (4, 2), the initial check block can be understood as a check block determined based on the RS erasure code algorithm with erasure code configuration (4, 2). In actual application, when a certain number of data blocks in the data block group are lost, the initial check block can be used to recover them, so as to obtain a complete data block group and further ensure the security of the data to be processed.

[0102] Determining the initial check algorithm corresponding to the initial data and the target check algorithm can be understood as determining the erasure code configuration for processing the data written by the user. The erasure code configuration is pre-configured in the distributed storage system before the user writes data. For example, in the case of receiving the initial data sent by user A, the initial erasure code encoding configuration and the target erasure code encoding configuration for processing the initial data sent by user A can be determined, wherein the initial erasure code encoding configuration and the target erasure code encoding configuration are pre-configured in the distributed storage system.

[0103] Specifically, the client can determine the initial check algorithm corresponding to the initial data and the target check algorithm, and determine the matrix parameters of the target check algorithm, and construct the target check matrix based on the matrix parameters; then obtain at least two data block sets included in the data block group, and process each data block set based on the target check matrix and the initial check algorithm to obtain the initial check block corresponding to the data block group.

[0104] In the above example, the matrix parameter can be understood as the target erasure code encoding configuration corresponding to the target check algorithm. Based on this, referring to Figure 2 , the distributed storage system can determine the initial check algorithm corresponding to the initial data and the target check algorithm, and determine the target erasure code encoding configuration corresponding to the target check algorithm, and generate the target encoding matrix of the target erasure code encoding configuration (8, 2) based on the target erasure code encoding configuration.

[0105] Then, at least two data block sets in the data block group are determined, that is Figure 3 data block sets A1-A4 and data block sets A5-A8, and data block sets A1-A4 and data block sets A5-A8 are processed based on the target encoding matrix of the target erasure code encoding configuration (8, 2) and the initial check algorithm, to obtain the initial check block of the original encoding group, that is, check blocks P(1_1), P(1_2), P(2_1), P(2_2).

[0106] Further, the processing each data block set based on the target check matrix and the initial check algorithm to obtain the initial check block corresponding to the data block group comprises:

[0107] Determining the matrix parameter corresponding to each data block set from the target check matrix;

[0108] Generating the initial check matrix corresponding to each data block set through the initial check algorithm based on the matrix parameter corresponding to each data block set;

[0109] Calculating each data block set through the initial check matrix corresponding to each data block set to obtain the check block of each data block set;

[0110] Determining the initial check block corresponding to the data block group based on the check block of each data block set.

[0111] In the above example, after determining the target coding matrix of the target erasure code coding configuration (8, 2), it is also necessary to determine the target coding matrix, see Figure 3 , Figure 3 is a schematic diagram of a target coding matrix in a data processing method provided by an embodiment of the present specification. In a manner provided by the present specification, based on the target coding matrix, the data blocks A1-A8 are calculated, and the check blocks P1, P2 corresponding to the data blocks A1-A8 can be determined.

[0112] P1 = Fx(A1, A2, A3, A4, A5, A6, A7, A8)

[0113] = x1 * A1 + x2 * A2 + x3 * A3 + x4 * A4+ x5 * A5 + x6 * A6 + x7 * A7+ x8 * A8

[0114] P2 = Fy(A1, A2, A3, A4, A5, A6, A7, A8)

[0115] = y1 * A1 + y2 * A2 + y3 * A3 + y4 * A4 + y5 * A5 + y6 * A6 + y7 * A7+ y8 * A8

[0116] The above formula is a target configuration encoding matrix of a target erasure code encoding algorithm with a target configuration of (8, 2). In the process of processing the data block through the target configuration encoding matrix, the target configuration encoding matrix can calculate the check blocks P1, P2 corresponding to the data blocks A1-A8. The target configuration encoding matrix is <x1, x2, x3, x4, x5, x6, x7, x8>, <y1, y2, y3, y4, y5, y6, y7, y8>, wherein x1-x8 are the matrix parameters of each row of the target encoding matrix, and y1-y8 are the matrix parameters of each column of the target encoding matrix.

[0117] The data processing method provided by the present specification can generate a target data block by obtaining an initial check block, avoid occupying a large amount of network bandwidth by obtaining initial data, obtain part of the matrix parameters from the target configuration encoding matrix as the encoding parameters of the original matrix of the initial encoding algorithm, and generate the initial check block based on the original matrix, so that the target data block can be generated directly based on the initial check block in the subsequent process.

[0118] Specifically, the distributed storage system determines the matrix parameters corresponding to the data block set A1-A4 and the data block set A5-A8 from the target configuration encoding matrix. The original erasure code configuration of the conversion performed by the data block set A1-A4 and the data block set A5-A8 is (4, 2). In this case, when data writing is performed using the (4, 2) configuration, a specific encoding matrix needs to be used to generate the check blocks P(1_1), P(1_2), P(2_1), P(2_2) of the original configuration. Taking the first four data blocks as an example, the data blocks A1, A2, A3, and A4 are used to generate P(1_1), P(2_1). The encoding parameters of the corresponding first four data blocks in the target encoding matrix are used as the encoding parameters of the original matrix.

[0119] That is, the matrix parameters corresponding to the data block set A1-A4 are <x1, x2, x3, x4>, <y1, y2, y3, y4>, and the matrix parameters corresponding to the data block set A5-A8 are <x5, x6, x7, x8>, <y5, y6, y7, y8>. <x1, x2, x3, x4>, <y1, y2, y3, y4>, <x5, x6, x7, x8>, and <y5, y6, y7, y8> are used as the encoding matrix of the initial check algorithm, and the encoding matrix of the initial check algorithm is multiplied by the data block set A1-A4 and the data block set A5-A8 to obtain the check block of each data block set.

[0120] That is, (A1, A2, A3, A4) * (x1, x2, x3, x4) = P(1_1), (A1, A2, A3, A4) * (y1, y2, y3, y4) = P(2_1) are obtained, so as to obtain the check blocks P(1_1), P(2_1) of the data block set A1-A4.

[0121] (A5, A6, A7, A8) * (x5, x6, x7, x8) = P(2_1), (A5, A6, A7, A8) * (y5, y6, y7, y8) = P(2_2) are obtained, so as to obtain the check blocks P(2_1), P(2_2) of the data block set A5-A8.

[0122] After obtaining the check blocks P(1_1), P(1_2), P(2_1), P(2_2), the check blocks P(1_1), P(1_2), P(2_1), P(2_2) are taken as initial check blocks of the original encoding group.

[0123] Therefore, it can be ensured that the check blocks P(1_1), P(1_2) under the original configuration can be directly used to calculate and generate the check blocks P1 of the target encoding parameter, and the check blocks P(2_1), P(2_2) under the original configuration can be used to calculate and generate the check blocks P2 of the target encoding parameter.

[0124] Further, after the data block group is processed based on the initial check algorithm and the target check algorithm to obtain the initial check block corresponding to the data block group, the method further includes:

[0125] The data block group and the initial check block corresponding to the data block group are sent to a data storage center.

[0126] Specifically, after the initial check block corresponding to the data block group is determined, the client can further send the data block group and the initial check block corresponding to the data block group to the data storage center for storage.

[0127] In the above example, after the initial check blocks P(1_1), P(1_2), P(2_1), P(2_2) corresponding to the original encoding group are determined, the original encoding group and the initial check blocks P(1_1), P(1_2), P(2_1), P(2_2) corresponding to the original encoding group can be sent to the data center, so as to complete the storage of the document sent by the user, and facilitate the conversion of the erasure code configuration of the document based on the initial check block.

[0128] Step 104: obtaining the initial check block corresponding to the data block group.

[0129] Specifically, after determining the data block group corresponding to the initial data, the client can determine and obtain the initial check block corresponding to the data block group.

[0130] In actual applications, when the initial data and the corresponding data block group are stored in the data storage center, after finding the data block group corresponding to the initial data from the data storage center, the initial check block corresponding to the data block group can be obtained from the data storage center.

[0131] Taking the above example, the initial data is a document stored by a user, the data block group corresponding to the document is the original encoding group in Figure 2 , which contains two adjacent data block sets, i.e., data block set A1-A4 and data block set A5-A8, and the initial check block corresponding to the data block group is P(1_1), P(1_2), P(2_1), P(2_2). Based on this, after the distributed storage system determines the original encoding group corresponding to the document from the data center, the initial check block P(1_1), P(1_2), P(2_1), P(2_2) corresponding to the original encoding group can be obtained.

[0132] Step 106: processing the initial check block corresponding to the data block group based on the target check algorithm to determine the target check block corresponding to the data block group.

[0133] The target check block can be understood as a check block determined based on the initial check algorithm, for example, in the case of initial check data being RS erasure code algorithm with erasure code configuration (4, 2), the target check block can be understood as a check block determined based on RS erasure code algorithm with erasure code configuration (8, 2). In actual applications, when a certain number of data blocks in the data block group are lost, the target check block can be used to recover them, thereby obtaining a complete data block group and further ensuring the security of the data to be processed.

[0134] Further, the processing of the initial check block corresponding to the data block group based on the target check algorithm to determine the target check block corresponding to the data block group comprises:

[0135] XOR processing the initial check block based on the target check algorithm to obtain the target data block corresponding to the data block group.

[0136] With the above example, after reading the check blocks in the original coding configuration directly, the corresponding check blocks P(1_1), P(1_2) in the two continuous original coding strips can be operated by XOR based on the RS erasure code algorithm with a configuration of (8, 2) to generate the check blocks P1 of the target coding matrix, and P(2_1), P(2_2) can be used to generate P2 in the same way. Thus, the target check blocks P1 and P2 are obtained.

[0137] In the embodiments of the present specification, the target data block corresponding to the data block group is obtained by performing XOR processing on the initial check block based on the target check algorithm, which avoids the problem of a large amount of network bandwidth being occupied due to the need to read the entire initial data in the data state conversion process.

[0138] Step 108: replacing the initial check block based on the target check block.

[0139] Specifically, after determining the target data block corresponding to the data block group, the original data block corresponding to the data block group is replaced by the target data block corresponding to the data block group, thereby completing the data state conversion of the initial data.

[0140] The data processing method provided in the present specification only needs to obtain the initial check block corresponding to the initial data and generate the target check block based on the initial check block in the process of converting the data state of the initial data, and the initial check block is replaced by the target check block, thereby reducing the amount of data to be read and written, avoiding the need to read the entire initial data in the data state conversion process, reducing the amount of data to be read and written, reducing resource consumption, and improving system performance.

[0141] In one embodiment of the present specification, the operation of replacing the initial check block by the target check block can be storing the target check block to the data storage center, and after receiving the target check block, the data storage center needs to delete the initial check block corresponding to the initial data, thereby realizing the operation of replacing the initial check block by the target check block, and avoiding the initial check block occupying the storage space of the data storage center, reducing the storage cost of data, and the specific implementation manner is as follows.

[0142] The replacing the initial check block based on the target check block comprises:

[0143] sending the target check block to the data storage center;

[0144] receiving the data deletion notification for the initial check block sent by the data storage center.

[0145] The data deletion notification is generated by the data storage center after the initial check block is deleted, and in actual application, the data deletion notification can inform the client that the data storage center has completed the deletion of the initial check block.

[0146] In the above example, after obtaining the target check blocks P1 and P2, the P1 and P2 are sent to the data center for storage, facilitating subsequent recovery of the data blocks corresponding to the initial data based on the target check blocks. After the distributed storage system sends the target check blocks P1 and P2 to the data center, the data center can delete the initial check blocks P(1_1), P(1_2) and P(2_1), P(2_2) corresponding to the data block group, thereby avoiding the initial check blocks occupying the storage space of the data center and reducing the storage cost of data. A data deletion notification is generated and sent to the distributed storage system, thereby informing the distributed storage system that the deletion of the initial check blocks has been completed.

[0147] In actual application, the data processing method provided in the specification can be applied in a single node storage scenario, and in the single node storage scenario, the data processing method can apply the CPU in a single machine (single node).

[0148] Based on this, the single machine determines the corresponding initial check block for the user data sent by the user based on the initial erasure code encoding configuration and the target erasure code encoding configuration through the CPU, and stores the initial check block and the user data in the disk of the single machine. Also, based on the CPU, the initial check block is obtained from the disk of the single machine and the target check block is generated, and the target check block is stored in the disk of the single machine, thereby reducing the occupancy rate of the data reading bandwidth between the CPU and the disk, avoiding the problem of a large amount of occupancy of the data reading bandwidth between the CPU and the disk, and further improving the processing efficiency of the CPU.

[0149] The following describes the data processing method provided in the specification in the application of the data processing method in the scenario of erasure code configuration for user data. Figure 4 The data processing method provided in the specification is further described in the application of the data processing method in the scenario of erasure code configuration for user data. Figure 4 A processing process flow diagram of a data processing method provided in an embodiment of the specification is shown, specifically including the following steps.

[0150] Step 402: The distributed storage system receives user data sent by a user.

[0151] The user data can be documents, multimedia files, etc., for example, a document with a size of 10MB.

[0152] Step 404: The system determines an initial erasure code coding configuration and a target erasure code coding configuration corresponding to the user data.

[0153] The initial erasure code coding configuration can be (4, 2), which means that the initial erasure code can calculate 4 data blocks at a time to obtain 2 check blocks.

[0154] The target erasure code coding configuration can be (8, 2), which means that the target erasure code can calculate 8 data blocks at a time to obtain 2 check blocks.

[0155] Specifically, in the case of receiving the initial data sent by user A, the initial erasure code coding configuration and the target erasure code coding configuration for processing the initial data sent by user A can be determined, wherein the initial erasure code coding configuration and the target erasure code coding configuration are pre-configured in the distributed storage system.

[0156] Step 406: The system determines to split the user data based on the initial erasure code coding configuration to obtain a plurality of data blocks.

[0157] Specifically, the initial erasure code coding configuration is (4, 2), which means that the preset check algorithm can calculate 4 data blocks at a time to obtain 2 check blocks.

[0158] Based on this, the distributed storage system can split the 8MB size document into 8 data blocks of 1MB size based on the erasure code configuration (4, 2).

[0159] Step 408: The system divides the plurality of data blocks into at least two data block sets.

[0160] Specifically, based on the initial erasure code coding configuration (4, 2), every 4 adjacent data blocks in the above-mentioned 8 data blocks are divided into a set, thereby obtaining data block sets A1-A4 and data block sets A5-A8.

[0161] Step 410: The system divides the at least two data block sets into a data block group.

[0162] Specifically, the distributed storage system divides at least two adjacent data block sets in the obtained data block sets into a data block group.

[0163] For example, the above-mentioned data block sets A1-A4 and data block sets A5-A8 are adjacent data block sets, so the data block sets A1-A4 and data block sets A5-A8 can be divided into a data block group.

[0164] Step 412: The system generates a target encoding matrix of the target erasure code encoding configuration as (8, 2) based on the target erasure code encoding configuration.

[0165] In practical applications, the target configuration is (8, 2), and the corresponding target encoding matrix can be <x1, x2, x3, x4, x5, x6, x7, x8>, <y1, y2, y3, y4, y5, y6, y7, y8>, where x1-x8 are the matrix parameters of each row of the target encoding matrix, and y1-y8 are the matrix parameters of each column of the target encoding matrix.

[0166] Step 414: The system selects the encoding parameters corresponding to the data blocks in the original configuration according to the target encoding configuration as the encoding parameters of the initial erasure code encoding matrix, and generates the check blocks corresponding to the data block set based on the initial erasure code encoding matrix.

[0167] In practical applications, the correspondence between the matrix parameters of the target configuration encoding matrix and the data blocks is as follows:

[0168] P1 = Fx(A1, A2, A3, A4, A5, A6, A7, A8)

[0169] = x1 * A1 + x2 * A2 + x3 * A3 + x4 * A4 + x5 * A5 + x6 * A6 + x7 * A7 + x8 * A8

[0170] P2 = Fy(A1, A2, A3, A4, A5, A6, A7, A8)

[0171] = y1 * A1 + y2 * A2 + y3 * A3 + y4 * A4 + y5 * A5 + y6 * A6 + y7 * A7 + y8 * A8

[0172] That is, the matrix parameters corresponding to the data block set A1-A4 are <x1, x2, x3, x4>, <y1, y2, y3, y4>, and the matrix parameters corresponding to the data block set A5-A8 are <x5, x6, x7, x8>, <y5, y6, y7, y8>.

[0173] <x1, x2, x3, x4>, <y1, y2, y3, y4>, and <x5, x6, x7, x8>, <y5, y6, y7, y8> are taken as the matrix parameters of the initial erasure code encoding matrix, so as to construct the initial erasure code encoding matrix, and the initial erasure code encoding matrix is multiplied by the data block set A1-A4 and the data block set A5-A8 to obtain the check blocks of each data block set.

[0174] That is, (A1, A2, A3, A4)*(x1, x2, x3, x4) = P(1_1), (A1, A2, A3, A4)*(y1, y2, y3, y4) = P(2_1), so as to obtain the check blocks P(1_1), P(2_1) of the data block set A1-A4.

[0175] (A5, A6, A7, A8)*(x5, x6, x7, x8) = P(2_1), (A5, A6, A7, A8)*(y5, y6, y7, y8) = P(2_2), so as to obtain the check blocks P(2_1), P(2_2) of the data block set A5-A8.

[0176] Step 416: The system stores the data block and the original check block in the data center.

[0177] Specifically, the data block corresponding to the user data and the initial check blocks P(1_1), P(2_1) and P(2_1), P(2_2) corresponding to the data block are sent to the data center (storage end in the distributed system) for storage.

[0178] Step 418: When performing encoding conversion, the system directly reads the check block under the original encoding configuration, and performs XOR operation to generate the target check block.

[0179] Specifically, when encoding conversion needs to be performed, the check blocks P(1_1), P(2_1) and P(2_1), P(2_2) under the original encoding configuration are directly read, so that the corresponding check blocks P(1_1), P(1_2) in the two consecutive original encoding strips are processed by XOR based on the target erasure code, and the check block P1 of the target encoding matrix is generated, and P2 can be generated by P(2_1), P(2_2) by analogy; so as to complete the encoding conversion.

[0180] In actual application, in the process of storing the data block and the initial check block to the data center, the storage address of the data block and the initial check block is recorded; based on this, when the check block under the original encoding configuration is read, the check block can be directly obtained from the data center based on the storage address.

[0181] Step 420: The system stores the target check block in the data center.

[0182] Specifically, after generating the target check blocks P1 and P2, the P1 and P2 are sent to the data center for storage. The original data block is used as the data block of the target encoding configuration, and there is no need to re-read and re-write the data.

[0183] In actual application, when performing the user-configured encoding conversion, the system can directly read the check blocks of the adjacent block group (adjacent data block set) and perform XOR operation to generate the check block of the target encoding configuration, and use the data block of the original encoding configuration as the data block in the target encoding configuration, thereby reducing the network overhead of original data reading when performing encoding configuration change; in the original scheme, all data blocks A1-A8 need to be read to calculate and generate the target check blocks P1, P2, a total of 8 data are read, and in the current scheme, the check blocks P(1_1), P(2_1), P(1_2), P(2_2) can be read to calculate and generate the target check blocks P1, P2, a total of 4 data are read.

[0184] Also, the data processing method provided in the present specification can be applied to a distributed system scenario, and in the distributed system scenario, the data processing method can be applied to a client in the distributed system. Correspondingly, the data storage center can be understood as a storage end in the distributed system. Based on this, the distributed system determines the corresponding initial check block for the data sent by the user according to the initial erasure code encoding configuration and the target erasure code encoding configuration through the client, and stores the initial check block and the user data in the storage end of the distributed system. Also, the initial check block is obtained from the storage end by the client to generate the target check block, and the target check block is sent to the storage end, thereby reducing the network bandwidth occupancy rate between the client and the storage end and avoiding the problem of large network bandwidth occupation.

[0185] The data processing method provided in the present specification proposes an erasure code file encoding strip configuration conversion technology. The check block of the erasure code data is generated through a specific generation scheme, so that when the erasure code encoding configuration is changed, only the check block under the original erasure code configuration needs to be accessed, and the user's original data does not need to be read, thereby reducing the traffic overhead when the user data encoding configuration is converted. The encoding matrix parameters when the original write data are constructed using the matrix parameters of the target encoding configuration are realized. When the encoding configuration conversion is performed, the check blocks of the original encoding group are directly read to perform XOR operation to generate the check blocks of the target encoding block group.

[0186] Corresponding to the method embodiments described above, the present specification also provides data processing device embodiments, Figure 5 A structural schematic diagram of a data processing device provided by one embodiment of the present specification is shown. As shown in the figure, Figure 5 The device includes:

[0187] The determination module 502 is configured to determine a data block group corresponding to the initial data, wherein the data block group includes at least two data block sets;

[0188] The acquisition module 504 is configured to acquire an initial check block corresponding to the data block group;

[0189] The processing module 506 is configured to process the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group.

[0190] The replacing module 508 is configured to replace the initial check block based on the target check block.

[0191] Optionally, the data processing apparatus further comprises an initial check block generation module configured to:

[0192] divide the initial data into a plurality of data blocks, and determine at least two data block sets based on the plurality of data blocks;

[0193] determine at least two data block sets in the at least two data block sets as one data block group;

[0194] process the data block group based on an initial check algorithm and a target check algorithm to obtain an initial check block corresponding to the data block group.

[0195] Optionally, the initial check block generation module is further configured to:

[0196] divide the initial data into a plurality of data blocks based on a calculation rule of the initial check algorithm, and divide the plurality of data blocks into at least two data block sets.

[0197] Optionally, the initial check block generation module is further configured to:

[0198] determine an initial check algorithm and a target check algorithm corresponding to the initial data;

[0199] determine a matrix parameter of the target check algorithm, and construct a target check matrix based on the matrix parameter;

[0200] obtain at least two data block sets in the data block group;

[0201] process each data block set based on the target check matrix and the initial check algorithm to obtain an initial check block corresponding to the data block group.

[0202] Optionally, the initial check block generation module is further configured to:

[0203] determine a matrix parameter corresponding to each data block set from the target check matrix;

[0204] generate an initial check matrix corresponding to each data block set by the initial check algorithm based on the matrix parameter corresponding to each data block set;

[0205] The processing module 506 is configured to perform calculation on each data block set by using the initial check matrix corresponding to the data block set, to obtain a check block of the data block set.

[0206] The processing module 506 is configured to determine the initial check block of the data block set based on the check block of each data block set.

[0207] Optionally, the initial check block generation module is further configured to:

[0208] The processing module 506 is configured to send the data block set and the initial check block corresponding to the data block set to a data storage center.

[0209] Optionally, the processing module 506 is further configured to:

[0210] The processing module 506 is configured to perform XOR processing on the initial check block based on the target check algorithm, to obtain a target data block corresponding to the data block set.

[0211] Optionally, the determination module 502 is configured to

[0212] Obtain data state information of the initial data;

[0213] In a case where it is determined that the data state information meets a preset processing condition, search for a data block set corresponding to the initial data.

[0214] Optionally, the data processing apparatus further includes a first information determination module configured to:

[0215] In a case where a data acquisition request for initial data is received, record a number of times of the data acquisition request;

[0216] Determine the number of times of the data acquisition request as data state information of the initial data.

[0217] Optionally, the data processing apparatus further includes a second information determination module configured to:

[0218] Determine a data storage time of initial data, and use the data storage time as data state information of the initial data.

[0219] Optionally, the determination module 502 is further configured to:

[0220] Search for a data block set corresponding to the initial data from a data storage center;

[0221] Correspondingly, the obtaining module 504 is further configured to

[0222] Obtain an initial check block corresponding to the data block set from the data storage center.

[0223] Optionally, the replacing module 508 is further configured to:

[0224] sending the target check block to the data storage center;

[0225] receiving the data deletion notification sent by the data storage center for the initial check block.

[0226] The data processing apparatus provided in the specification only needs to obtain the initial check block corresponding to the initial data in the process of converting the data state of the initial data, and replace the initial check block by the target check block generated based on the initial check block, thereby reducing the amount of data read and written, avoiding the need to read the entire initial data in the process of converting the data state, reducing the amount of data read and written, reducing resource consumption, and improving system performance.

[0227] The above is a schematic scheme of the data processing apparatus of the embodiment. It should be noted that the technical scheme of the data processing apparatus belongs to the same concept as the technical scheme of the data processing method described above, and the details of the technical scheme of the data processing apparatus that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0228] Figure 6 A structural block diagram of a computing device 600 according to an embodiment of the specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 through a bus 630, and a database 650 is used to save data.

[0229] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 can include one or more of any type of network interface (e.g., network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near-field communication (NFC) interface, and the like.

[0230] In an embodiment of the specification, the above-described components of the computing device 600 and other components not shown in the Figure 6 may be connected to each other, for example, through a bus. It should be understood that Figure 6The illustrated computing device structural block diagram is merely for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or substituted as needed by those skilled in the art.

[0231] The computing device 600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. The computing device 600 can also be a mobile or stationary server.

[0232] The processor 620 is configured to execute computer-executable instructions, which, when executed by the processor 620, implement the steps of the data processing method.

[0233] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0234] An embodiment of the present specification further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the data processing method.

[0235] The above is a schematic scheme of the computer-readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0236] An embodiment of the present specification further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the data processing method.

[0237] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the data processing method belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0238] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0239] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0240] It should be noted that for the foregoing method embodiments, the acts described therein can be performed in a different order from the order described, and that some acts can be performed in parallel or concurrently. In addition, some of the acts described above can not be performed in all embodiments. Furthermore, the acts described above can be performed by different parties in some embodiments. Furthermore, each of the acts described above can be performed by specialized hardware components or modules or can be embodied in a software-specific or a generalized computing system or module. Similarly, general-purpose computing systems and modules can be configured to constitute one or more specialized components or modules described above.

[0241] In the above embodiments, the description of each embodiment is focused on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0242] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and do not limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; obtaining an initial check block corresponding to the data block group; processing the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group, wherein the initial check block is obtained by calculating each data block set in the data block group by an initial check matrix, and a matrix parameter of the initial check matrix is obtained from a matrix parameter of the target check algorithm, and the data block set is divided into a plurality of data blocks based on a calculation rule of the initial check algorithm and is divided to obtain the plurality of data blocks; replacing the initial check block with the target check block. 2.The data processing method of claim 1, wherein the initial check block is obtained by calculating each data block set in the data block group by an initial check matrix, comprising: determining an initial check algorithm and a target check algorithm corresponding to the initial data; determining a matrix parameter of the target check algorithm and constructing a target check matrix based on the matrix parameter; obtaining at least two data block sets in the data block group; obtaining a matrix parameter of an initial check matrix based on the matrix parameter of the target check matrix, and processing each data block set based on the initial check matrix to obtain the initial check block corresponding to the data block group. 3.The data processing method of claim 2, wherein the initial check block corresponding to the data block group is obtained by obtaining a matrix parameter of an initial check matrix based on a matrix parameter of the target check matrix and processing each data block set based on the initial check matrix, comprising: determining a matrix parameter corresponding to each data block set from the target check matrix; generating an initial check matrix corresponding to each data block set by the initial check algorithm based on the matrix parameter corresponding to each data block set; calculating each data block set by the initial check matrix corresponding to each data block set to obtain a check block of each data block set; determining the initial check block corresponding to the data block group based on the check block of each data block set. 4.The data processing method of claim 2, wherein after obtaining the initial check block corresponding to the data block group by obtaining a matrix parameter of an initial check matrix based on a matrix parameter of the target check matrix and processing each data block set based on the initial check matrix, the method further comprises: sending the data block group and the initial check block corresponding to the data block group to a data storage center. 5.The data processing method of claim 1, wherein the target check block corresponding to the data block group is obtained by processing the initial check block based on the target check algorithm, comprising: performing exclusive or processing on the initial check block based on the target check algorithm to obtain the target data block corresponding to the data block group. 6.The data processing method of claim 1, wherein the data block group corresponding to the initial data is determined, comprising: ​ ​ ​ ​ ​ acquire data state information of the initial data; in a case where it is determined that the data state information meets preset processing conditions, find a data block group corresponding to the initial data.

7. The data processing method of claim 6, before the acquiring of the data state information of the initial data, further comprising: in a case where a data acquisition request for initial data is received, record a number of times of the data acquisition request; determine the data state information of the initial data based on the number of times of the data acquisition request.

8. The data processing method of claim 1, the determining of the data block group corresponding to the initial data, comprising: finding the data block group corresponding to the initial data from a data storage center; correspondingly, the acquiring of the initial check block corresponding to the data block group, comprising: acquiring the initial check block corresponding to the data block group from the data storage center.

9. The data processing method of claim 8, the replacing of the initial check block based on the target check block, comprising: sending the target check block to the data storage center; receiving a data deletion notification for the initial check block sent by the data storage center.

10. A data processing apparatus, comprising: a determining module configured to determine a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; an acquiring module configured to acquire an initial check block corresponding to the data block group; a processing module configured to process the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group, wherein the initial check block is obtained by calculating each data block set in the data block group by an initial check matrix, a matrix parameter of the initial check matrix is obtained from a matrix parameter of the target check algorithm, and the data block set is divided into a plurality of data blocks based on a calculation rule of the initial check algorithm and is divided to obtain the plurality of data blocks; a replacing module configured to replace the initial check block based on the target check block.

11. A computing device, comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement steps of the data processing method of any one of claims 1 to 9.

12. A computer readable storage medium, storing computer executable instructions, and the computer executable instructions, when executed by a processor, implement steps of the data processing method of any one of claims 1 to 9.

Images