Data local recovery method, device and storage medium

Abstract

Embodiments of the present application provide a data local recovery method, device and storage medium. Global check blocks and local check blocks in any data group of a data set are obtained by encoding data blocks, and when any data block in the data group is lost, the data block can be recovered according to the remaining data blocks, global and local check blocks. When the local check block is lost, it can be recovered according to the data blocks in the data group. The local check block contains the encoding information of the global check block, and when the global check block is lost, the global check block can be recovered according to the data blocks and the local check block, realizing local recovery for any object block in the data group. The sum of the local check blocks of the data set is linearly independent of the global check block of the data set, and when the number of lost object blocks in the data set is greater than the number of global check blocks, the lost object blocks can be recovered, further improving the fault tolerance of the data local recovery method.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and storage medium for partial data recovery. Background Technology

[0002] Erasure coding is a fault-tolerance coding technique used to recover original data when some data is lost. Erasure coding can maintain reliability while occupying less storage space, and is therefore widely used in distributed storage systems. Compared to traditional copy-based data recovery methods, erasure coding suffers from read amplification during data reconstruction.

[0003] Currently, there is a method for partial data recovery based on Local Repair Code (LRC). However, existing partial repair schemes still cannot perform partial recovery of the global parity block with high fault tolerance. Therefore, a new solution is needed. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for partial data recovery, which are used to partially recover global check blocks and improve the fault tolerance of the partial data recovery method.

[0005] This application embodiment also provides a method for partial data recovery, comprising: determining multiple data groups contained in a dataset, wherein an object block in any data group includes: a data block, a global verification block, and a local verification block; responding to a recovery request for any target object block in the data group, recovering the target object block according to the remaining readable object blocks in the data group; wherein, in the data group, the global verification block and the local verification block are obtained by encoding the data block, and the local verification block contains the encoding information of the global verification block; the sum of the local verification blocks in the dataset is linearly independent of the global verification blocks in the dataset.

[0006] Optionally, the method further includes: responding to a recovery request for a first number of object blocks in the dataset, and recovering the first number of object blocks based on the remaining readable object blocks in the dataset; wherein the first number is greater than the number of global check blocks in the dataset.

[0007] Optionally, in response to a recovery request for any target object block in the data group, before determining the remaining readable object blocks in the data group, the method further includes: determining a first matrix of r+1 dimensions, where r is a positive integer; generating r derived rows corresponding to any r data rows in the first matrix; the r derived rows are used to encode r local check blocks, each derived row containing elements from the data rows used to generate the derived row; replacing the data rows in the first matrix except for the any r data rows with the r derived rows to obtain a second matrix; appending an identity matrix to the second matrix to obtain an encoding matrix; and encoding the data blocks in the dataset using the encoding matrix to obtain global check blocks and local check blocks corresponding to each of the plurality of data groups.

[0008] Optionally, in the encoding matrix, any r+1 columns are full-rank matrices.

[0009] Optionally, the first matrix is ​​a Cauchy matrix.

[0010] Optionally, generating r derived rows based on any r data rows in the first matrix includes: adding additional terms to the elements of different columns in the r data rows according to the correspondence between the elements of different columns in the r data rows and the data groups during matrix calculation, thereby obtaining the r derived rows; wherein, in the same derived row, the additional terms are added to the elements of the columns corresponding to any data group; in different derived rows, the data groups corresponding to the elements to which the additional terms are added are different.

[0011] Optionally, based on the correspondence between the elements of different columns in the r data rows and the data group during matrix calculation, additional terms are added to the elements of different columns in the r data rows. This includes: for the element of the j-th column in the i-th data row, determining the additional terms for the element of the j-th column in the i-th data row based on the element of the j-th column in the (r+1)-th row of the first matrix; adding the additional terms to the element of the j-th column in the i-th data row to obtain the derived row corresponding to the i-th data row, where i and j are positive integers.

[0012] Optionally, the summation of the elements in the r derived rows is equal to the summation of the elements in the (r+1)th row of the first matrix; wherein the summation includes an XOR operation on the elements.

[0013] In the partial data recovery method provided in this application embodiment, when there is a recovery request for any target object block in any data group of the dataset, the target object block can be recovered based on the remaining readable object blocks in the data group. This target object block can be a data block, a global check block, or a local check block in the data group. The global check block and local check block are obtained by encoding the data block, and can be recovered based on the remaining data block, global check block, and local check block when any data block in the data group is lost. The local check block contains the encoding information of the global check block, and can be recovered based on the data block and local check block when the global check block is lost. When a local check block is lost, it can be recovered based on the data blocks in the data group. Thus, partial recovery of any object block in the data group is achieved. The sum of local check blocks in the dataset is linearly independent of the global check blocks in the dataset; therefore, lost object blocks can be recovered even when the number of lost object blocks in the dataset is greater than the number of global check blocks. Furthermore, this allows for the partial recovery of any object block within a group while simultaneously enhancing the fault tolerance of the local data recovery method. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 A schematic flowchart of a data partial recovery method provided for an exemplary embodiment of this application;

[0016] Figure 2 This is a diagram illustrating the global check block and local check block corresponding to r data groups;

[0017] Figure 3 This is a schematic diagram of the global check block and the local check block corresponding to two data groups;

[0018] Figure 4 A schematic diagram showing the global check block and local check block corresponding to the three data groups;

[0019] Figure 5 This is a schematic diagram of the structure of a server provided for an exemplary embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0024] Currently, there are some methods for partial data recovery based on Local Repair Code (LRC). In this method, data is divided into multiple groups, and local check blocks can be generated within each group. When the number of lost data blocks within a group does not exceed the number of local check blocks within the group, the lost data blocks can be recovered from other data blocks within the group, thereby reducing the throughput during data block recovery.

[0025] In a typical approach, the global check block P is obtained by operating on all data blocks D; the local check block L is obtained by operating on the data blocks within the group. The local check block L can be recovered locally. However, the global check block does not have the property of local recovery; that is, it requires reading all data blocks D to recover the global check block P.

[0026] In another typical approach, the global parity block P is obtained by operating on all data blocks D; the local parity block L is obtained by operating on the data blocks within the group. The difference is that when constructing the encoding matrix, the sum of all local parity blocks must be consistent with the sum of all global parity blocks. Therefore, when a global parity block is lost, it can be recovered using local parity blocks without needing to read all data blocks D. However, this approach sacrifices the fault tolerance of LRC, meaning it can only tolerate the loss of any r blocks. Specifically, when the number of lost blocks is less than or equal to r, the lost blocks can be recovered. Here, r is the number of global parity blocks, and r is a positive integer.

[0027] To address the problem that existing local repair methods cannot simultaneously achieve fault tolerance and local recovery of any lost block, this application provides a solution in some embodiments. The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a partial data recovery method provided in an exemplary embodiment of this application. The method may include, for example: Figure 1 The steps shown are as follows:

[0029] Step 101: Determine the multiple data groups contained in the dataset. The object block in any data group includes: a data block, a global check block, and a local check block; wherein, in the data group, the global check block and the local check block are obtained by encoding the data block, and the local check block contains the encoding information of the global check block.

[0030] Step 102: In response to a recovery request for any target object block in the data group, recover the target object block based on the remaining readable object blocks in the data group; the sum of local check blocks in the dataset is linearly independent of the global check blocks in the dataset.

[0031] The execution entity in this embodiment can be a server storing data. This server can be a conventional server, a cloud server, an elastic computing instance in the cloud, or a virtualized data center, etc., and this embodiment is not limited to any particular type. The server runs a data storage application used to store and manage the received data.

[0032] In this embodiment, the data storage application on the server can perform partial repair of lost data based on erasure coding, thereby reducing the cost required for data repair.

[0033] The dataset contains multiple data groups for local computation during data failure recovery. The number of data blocks in each data group can be determined according to requirements; this embodiment does not impose a limit. Each data group may contain multiple object blocks, which may include: data blocks, global verification blocks, and local verification blocks (local repairparity).

[0034] The target object block to be recovered can be any one of a data block, a global check block, or a local check block. When any object block in the data group is lost, it can be recovered from the remaining data blocks.

[0035] The global check block, obtained by encoding all data blocks in the dataset (i.e., the global data block), can be used to recover the global data block when any global data block is lost. That is, for any data group, when any data block in that data group is lost, the lost data block can be partially recovered using the global data block within that data group. The local check block in any data group is obtained by encoding the data blocks within that data group and can be designed to contain the encoding information of the global check block through the encoding process of the local check block. Therefore, when the global check block corresponding to that data group is lost, it can be recovered using the data blocks in the data group containing the global check block and the local check block.

[0036] When a dataset contains multiple data groups, each data group can correspond to a different global validation block. The sum of these local validation blocks is linearly independent of the global validation blocks in the dataset. That is, the sum of multiple local validation blocks can be considered a new global validation block, which is linearly independent of the original global validation blocks in the dataset. Therefore, when the dataset originally has r global validation blocks, the sum of the local validation blocks can be used as the (r+1)th global validation block. This allows for data recovery of any (r+1)th object block when it is lost, using the remaining object blocks.

[0037] Optionally, based on this, the server may respond to a recovery request for a first number of object blocks in the dataset, and recover the first number of object blocks based on the remaining readable object blocks in the dataset. The first number is greater than the number of global check blocks in the dataset. Typically, this first number is the number of global check blocks plus one. That is, when the dataset has r global check blocks, it can tolerate the loss of any r+1 object blocks; in this case, the remaining object blocks are used to recover the data from those r+1 object blocks.

[0038] In this embodiment, when a recovery request for any target object block in any data group of the dataset exists, the target object block can be recovered based on the remaining readable object blocks in the data group. This target object block can be a data block, a global check block, or a local check block within the data group. The global check block and local check block are obtained by encoding the data block, and can be recovered based on the remaining data blocks, global check block, and local check block when any data block within the data group is lost. The local check block contains the encoded information of the global check block, and can be recovered based on the data block and local check block when the global check block is lost. When a local check block is lost, it can be recovered based on the data blocks within the data group. Thus, local recovery of any object block within the data group is achieved. The sum of local check blocks in the dataset is linearly independent of the number of global check blocks in the dataset; therefore, lost object blocks can be recovered even when the number of lost object blocks in the dataset is greater than the number of global check blocks. Furthermore, this allows for the partial recovery of any object block within a group while simultaneously enhancing the fault tolerance of the local data recovery method.

[0039] Before performing data recovery, the server can generate global and local checksum blocks based on erasure coding principles. These global and local checksum blocks are obtained by encoding data blocks in the dataset using an encoding matrix. The optional methods for generating the encoding matrix will be described below.

[0040] Suppose the dataset contains k data blocks, and the dataset is divided into g data groups. Assume the number of global check blocks and local check blocks is r, where k = g * r. When each data group contains one global check block, g = r.

[0041] In some optional embodiments, the server may determine a first matrix of dimension r+1; wherein the first matrix has r+1 rows and k columns. Based on the first matrix, an encoding matrix can be constructed. This encoding matrix is ​​used to describe the process of encoding a check block in matrix form.

[0042] During the construction of the encoding matrix, r derived rows can be generated based on any r data rows in the first matrix. These r derived rows are used to encode r local parity blocks. Each derived row contains elements from the data rows used to generate it, so that the encoded local parity blocks contain the encoding information of the global parity blocks.

[0043] After determining r derived rows, these r derived rows can be used to replace all data rows in the first matrix except for the r data rows mentioned above, resulting in the second matrix. The terms "first" and "second" are used to define the matrices described, solely for ease of description and distinction, and do not impose any other restrictions on the matrices. After determining the second matrix, an identity matrix can be appended to it to obtain the encoding matrix. In the identity matrix, all elements on the diagonal from the top left to the bottom right (called the main diagonal) are 1, and all other elements are 0, and the identity matrix is ​​a full-rank matrix. In this embodiment, the elements that are 1 in the identity matrix represent the check blocks to be encoded. In the encoding matrix, the elements that are 1 in the same row as the original data rows in the first matrix are used to calculate the global check block, and the elements that are 1 in the same row as the derived data rows are used to calculate the local check block.

[0044] After constructing the encoding matrix, the data blocks in the dataset can be encoded using the encoding matrix to obtain the global check block and the local check blocks corresponding to each of the multiple data groups. This will be explained in detail below.

[0045] Optionally, in this embodiment, the encoding objective can be set to generate r+1 linearly independent global check blocks. Among the r+1 global check blocks, one global check block is obtained by summing r local check blocks. Before generating the derived rows, the first matrix can be designed such that after adding the identity matrices corresponding to the r+1 global check blocks to the end of the first matrix to obtain a new matrix, any r+1 columns taken from this new matrix form a full-rank column matrix, thus making any r+1 columns in the new matrix linearly independent. Therefore, when any r+1 blocks out of the k data blocks and r+1 global check blocks are lost, they can be repaired using the remaining blocks.

[0046] In some embodiments, the first matrix can be implemented as a Cauchy matrix. When an r*g-dimensional Cauchy matrix is ​​followed by an r*r-dimensional identity matrix, the matrix formed by any r+1 columns can be a full-rank matrix.

[0047] In the first matrix with r+1 rows mentioned above, any r data rows can be used to calculate r global check blocks, and the remaining data row can be used to calculate the local check blocks of each of the g data groups. Optionally, r derived rows can be generated based on the remaining data row, and the r derived rows correspond to the g data groups. Based on the r data rows and the r derived rows, the adjusted second matrix can be obtained.

[0048] To ensure that the second encoding matrix followed by the identity matrix has the same linear independence as the first matrix followed by the identity matrix, the sum of the r derived rows in the second matrix can be made the same as the sum of the remaining 1 data row in the first matrix.

[0049] Optionally, based on the correspondence between the elements of different columns in the r data rows of the first matrix and the data groups during matrix calculation, additional terms can be added to the elements of different columns in the r data rows to obtain the r derived rows. Within the same derived row, the additional term is added to the element of the column corresponding to any data group; in different derived rows, the data groups corresponding to the elements to which the additional term is added are different. The additional term must ensure that the sum of the elements in the r derived rows is the same as the sum of the remaining data row.

[0050] For example, the elements corresponding to data group g1 in a certain data row can be locally adjusted to obtain a derived data row corresponding to data group g1. Similarly, the elements corresponding to data group g2 in another data row can be locally adjusted to obtain a derived data row corresponding to data group g2. These r derived data rows are used to calculate the local check blocks corresponding to each of the g data groups, resulting in r local check blocks.

[0051] In the following embodiments, for ease of description and calculation, r global check blocks are calculated using rows 1 to r of the first matrix, and r local check blocks are calculated using row (r+1). When generating r derived rows, additional items can be added to each row of the first matrix to obtain r derived rows.

[0052] The additional items must satisfy the following conditions:

[0053] 1. In the same derived row, the additional item is added to the element corresponding to any data group; in different derived rows, the data groups corresponding to the elements to which the additional item is added are different.

[0054] 2. The summation of the elements in the r derived rows is equal to the summation of the elements in the (r+1)th row of the first matrix; where the summation refers to the XOR operation.

[0055] For example, for the first data group consisting of data from the 1st to the gth data points, additional items can be added to the elements corresponding to the first data group in the first row of the first matrix, while other elements remain unchanged, resulting in the second row of the second matrix. This second row can be used to encode the local parity block corresponding to the first data group. Similarly, for the second data group consisting of data from the (g+1)th to the 2gth data points, additional items can be added to the elements corresponding to the second data group in the second row of the first matrix, while other elements remain unchanged, resulting in the fourth row of the second matrix. This fourth row can be used to encode the local parity block corresponding to the second data group. And so on, for the gth data group consisting of data from the (r-1)g+1th to the rgth data points, additional items can be added to the elements corresponding to the gth data group in the 2r-1th row of the first matrix, while other elements remain unchanged, resulting in the 2rth row of the second matrix. This 2rth row can be used to encode the local parity block corresponding to the gth data group.

[0056] The following will provide further explanation with reference to the specific calculation process.

[0057] Determine the first matrix of dimension (r+1)*k:

[0058] [a ij :1≤i≤r+1,1≤j≤k] (r+1)*k

[0059] Expanded to:

[0060]

[0061] Rows 1 through r of the first matrix can be used to encode r global parity blocks, and row r+1 is used to encode r local parity blocks. Based on the first matrix, a 2r*k dimensional second matrix can be generated. The second matrix contains rows 1 through r of the first matrix and r derived rows, where the derived rows are distributed in rows 2, 4, ..., 2r of the second matrix. During encoding, a 2r*k identity matrix can be added after the second matrix to obtain the encoding matrix H, where any r+1 columns of the encoding matrix H are full-rank matrices.

[0062]

[0063] Where, d j This is an additional term used to locally adjust rows 1 to r in the first matrix to obtain r derived rows.

[0064] Among them, d1~d g Add it to the element in the first row of the first matrix that corresponds to the first data group. Where d g+1 ~d 2g Add it to the element in the second row of the first matrix that corresponds to the second data group. (r-1)g+1 ~d rg The additional item is added to the element corresponding to the g-th data group in the r-th row of the first matrix. That is, it satisfies the following conditions: within the same derived row, the additional item is added to the element corresponding to any data group; in different derived rows, the data group corresponding to the element to which the additional item is added is different.

[0065] Optionally, to ensure that the summation of the elements in the r derived rows is equal to the summation of the elements in the (r+1)th row of the first matrix, the summation operation can be designed as an XOR operation, and the additional terms corresponding to the columns can be calculated based on the elements of the same columns in the r data rows.

[0066] Optionally, taking the element in the j-th column of the i-th data row as an example, the additional term for the element in the j-th column of the i-th data row can be determined based on the element in the j-th column of the (r+1)-th row of the first matrix. Adding this additional term to the element in the j-th column of the i-th data row yields the corresponding derived row. The additional term d in the j-th column of the i-th row... j The calculation process is as follows:

[0067]

[0068] Add corresponding additional items to the elements corresponding to different data groups in the first matrix to generate r derived rows corresponding to the local check block. In the second matrix above, summing (XORing) the second row, fourth row, ..., 2rth row yields:

[0069] h2+h4+...+h 2r =a r+1,1 +a r+1,2 ,...,a r+1,rg

[0070] That is, the summation result is equal to the summation result of the elements in the (r+1)th row of the first matrix. Therefore, the sum of the local check blocks calculated from the r derived rows has a linearly independent relationship with the r global check blocks. The sum of the local check blocks encoded by the r derived rows can be regarded as a new global check block, which is linearly independent of the r global check blocks, thus allowing for the loss of any r+1 data blocks.

[0071] By performing matrix calculations on the encoding matrix H obtained using the above method and the k-dimensional matrix D formed by k data blocks, and ensuring that H*D = 0, the global parity block and the local parity block can be obtained. That is:

[0072]

[0073] Based on the above formula, P1, P2, P3...P can be determined. r This serves as a global check block. Among them, L1, L2, L3…L… r Corresponding to the derived line, it can be used as a local check block.

[0074] The above formula is transformed to obtain the global verification block P. i It can be represented as:

[0075]

[0076] Among them, the local verification block L i Represented as:

[0077]

[0078] Among them, k data blocks, r global check blocks, and r local check blocks can be divided into r data groups, namely:

[0079] G1 = {D1,D2,...,D} g ,L1,P1},

[0080] G2={D g+1 D g+2 ,...,D 2g ,L2,P2}

[0081] …

[0082] G r ={D (r-1)g+1 D (r-1)g+2 ,...,D rg ,L r ,P r}

[0083] In this system, if a single check block is lost in any data group, data recovery can be achieved using the other g+1 data blocks within the group. The local check blocks L1, L2, ..., L... r The sum can be considered as a new global check block P. r+1 Because L1, L2, ..., L r With P1, P2, ... P r There is a linearly independent relationship between P1, P2, P3, ..., P... r P r+1 They are linearly independent. Therefore, when D1, D2, D3, ..., D... k P1, P2, ..., P r L1, L2, ..., L r In this case, if any r+1 data blocks are lost, the lost data blocks can be recovered from the remaining data blocks.

[0084] Based on the above embodiments, a highly reliable, high-performance, and parameter-flexible encoding method is achieved. High reliability means that the encoding result can be recovered even if any r+1 blocks are lost. High performance means that even if any single block in a group is lost, it can be recovered using local data blocks, reducing read amplification during data recovery. Parameter flexibility means that the number of data blocks can be divided by the number of global check blocks, and an accurate encoding matrix structure can be provided even when the number of data blocks changes, meeting encoding requirements in various situations.

[0085] To further describe the embodiments of this application in detail, the following will use an example where r is implemented as 2 and k is implemented as 4.

[0086] Suppose the dataset contains data blocks D1, D2, D3, and D4. Divide the dataset into groups G1 containing D1 and D2, and G2 containing D3 and D4.

[0087] Determine the first matrix:

[0088]

[0089] Summing each row of the first matrix yields d. j :

[0090]

[0091] Using the first matrix and the result calculated by Formula 1, construct the second matrix, and then append the identity matrix to the second matrix to obtain the encoding matrix:

[0092]

[0093] Based on Formula 2-1 above, the sum of the second and fourth rows in the second matrix is ​​calculated to obtain:

[0094] h2+h4=a 1,1 +d1+a 1,2 +d2+a 1,3 +a 1,4 +a 2,1 +a 2,2 +a 2,3 +d3+a 2,4 +d4

[0095] =a 1,1 +(a 1,1 +a 2,1 +a 3,1 )+a 1,2 +(a 1,2 +a 2,2 +a 3,2 )+a 1,3 +a 1,4 +a 2,1 +a 2,2 +

[0096] +a 2,3 +(a 1,3 +a 2,3 +a 3,3 )+a 2,4 +(a 1,4 +a 2,4 +a 3,4 )

[0097] =a 3,1 +a 3,2 +a 3,3 +a3,4

[0098] That is, the sum of the second and third rows in the second matrix is ​​equal to the sum of the third row in the first matrix, such that any three columns in the encoding matrix satisfy the linear independence property.

[0099] Next, matrix operations are performed using data blocks D1, D2, D3, and D4 with the aforementioned encoding matrix to obtain the global parity block and the local parity block:

[0100] Among them, global verification block P i Represented as:

[0101]

[0102] Among them, the local verification block L i Represented as:

[0103]

[0104] Based on the above formulas 2-2 and 2-3, global check blocks P1 and P2 and local check blocks L1 and L2 can be obtained.

[0105] The aforementioned data blocks D1, D2, D3, and D4, global check blocks P1 and P2, and local check blocks L1 and L2 can be divided into two data groups: G1 = {D1, D2, L1, P1} and G2 = {D3, D4, L2, P2}.

[0106] In this scenario, if a single check block is lost in any data group, the data can be recovered using the other three data blocks in the group. For example, if D1 is lost, it can be recovered using D2, L1, and P1.

[0107] The sum of local check blocks L1 and L2 can be considered as a new global check block P3. Since the sum of L1 and L2 is linearly independent of P1 and P2, P1, P2, and P3 are linearly independent. Therefore, if any three data blocks among D1, D2, D3, D4, P1, P2, L1, and L2 are lost, the lost data blocks can be recovered from the remaining data blocks.

[0108] The following explanation will be based on the example of r being 3 and k being 9.

[0109] Suppose the dataset contains data blocks D1, D2, D3, D4, D5, D6, D7, D8, and D9. Grouping the dataset yields data group G1 containing D1, D2, and D3; data group G2 containing D4, D5, and D6; and data group G3 containing D7, D8, and D9.

[0110] Determine the first matrix:

[0111]

[0112] Summing each row of the first matrix yields d. j :

[0113]

[0114] Using the first matrix and the result calculated by Formula 1, construct the second matrix, and then append the identity matrix to the second matrix to obtain the encoding matrix:

[0115]

[0116] Based on Formula 3-1 above, the sum of the second, fourth, and sixth rows of the second matrix is ​​calculated to obtain:

[0117] h2 + h4 + h6 = a 4,1 +a 4,2 +a 4,3 +a 4,4 +a 4,4

[0118] That is, the sum of the 2nd, 4th and 6th rows in the second matrix is ​​equal to the sum of the 3rd row in the first matrix, so that any 3 columns in the encoding matrix satisfy the linear independence property.

[0119] Next, matrix operations are performed using data blocks D1, D2, D3, D4, D5, D6, D7, D8, and D9 with the aforementioned encoding matrix to obtain the global check block and the local check block:

[0120] Among them, global verification block P i Represented as:

[0121]

[0122] Among them, the local verification block L i Represented as:

[0123]

[0124] Based on the above formulas 3-2 and 3-3, global check blocks P1, P2, and P3, and local check blocks L1, L2, and L3 can be obtained.

[0125] The aforementioned data blocks D1, D2, D3, D4, D5, D6, D7, D8, D9, global check blocks P1, P2, P3, and local check blocks L1, L2, L3 can be divided into three data groups: G1 = {D1, D2, D3, L1, P1}, G2 = {D4, D5, D6, L2, P2}, and G3 = {D7, D8, D9, L3, P3}.

[0126] In this scenario, if a single check block is lost in any data group, the data can be recovered using the other four data blocks in the group. For example, if D1 is lost, it can be recovered using D2, D3, L1, and P1.

[0127] The sum of local check blocks L1, L2, and L3 can be considered as a new global check block P4. Since the sum of L1, L2, and L3 is linearly independent of P1, P2, and P3, P1, P2, P3, and P4 are also linearly independent. Therefore, if any four data blocks from D1, D2, D3, D4, D5, D6, D7, D8, D9, P1, P2, P3, L1, L2, and L3 are lost, the lost data blocks can be recovered from the remaining data blocks.

[0128] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0129] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0130] Figure 5 This illustration shows a structural diagram of a server provided in an exemplary embodiment of this application, which is suitable for the information processing system provided in the foregoing embodiments. Figure 5 As shown, the server includes: a memory 501, a processor 502, and a communication component 503.

[0131] Memory 501 is used to store computer programs and can be configured to store various other data to support operations on the server. Examples of this data include instructions for any application or method used to operate on the server.

[0132] Processor 502, coupled to memory 501, is configured to execute a computer program in memory 501 for: determining a dataset containing multiple data groups, wherein an object block in any data group includes: a data block, a global check block, and a local check block; responding to a recovery request for any target object block in the data group, recovering the target object block based on the remaining readable object blocks in the data group; wherein, in the data group, the global check block and the local check block are obtained by encoding the data block, and the local check block contains the encoding information of the global check block; the sum of the local check blocks in the dataset is linearly independent of the global check blocks in the dataset.

[0133] Optionally, the processor 502 is further configured to: respond to a recovery request for a first number of object blocks in the dataset, and recover the first number of object blocks based on the remaining readable object blocks in the dataset; wherein the first number is greater than the number of global check blocks in the dataset.

[0134] Optionally, before responding to a recovery request for any target object block in the data group, the processor 502 further comprises: determining a first matrix of r+1 dimensions, where r is a positive integer; generating r derived rows corresponding to any r data rows in the first matrix; the r derived rows are used to encode r local check blocks, each derived row containing elements from the data rows used to generate the derived row; replacing the data rows in the first matrix except for the any r data rows with the r derived rows to obtain a second matrix; appending an identity matrix to the second matrix to obtain an encoding matrix; and encoding the data blocks in the dataset using the encoding matrix to obtain global check blocks and local check blocks corresponding to each of the plurality of data groups.

[0135] Optionally, in the encoding matrix, any r+1 columns are full-rank matrices.

[0136] Optionally, the first matrix is ​​a Cauchy matrix.

[0137] Optionally, when the processor 502 generates r derived rows based on any r data rows in the first matrix, it specifically performs the following: based on the correspondence between the elements of different columns in the r data rows and the data groups during matrix calculation, it adds additional terms to the elements of different columns in the r data rows to obtain the r derived rows; wherein, in the same derived row, the additional terms are added to the elements of the columns corresponding to any data group; in different derived rows, the data groups corresponding to the elements to which the additional terms are added are different.

[0138] Optionally, when the processor 502 adds additional terms to the elements in different columns of the r data rows based on the correspondence between the elements in different columns of the r data rows and the data group during matrix calculation, it specifically performs the following: for the element in the j-th column of the i-th data row, based on the element in the j-th column of the r+1 row of the first matrix, determines the additional term for the element in the j-th column of the i-th data row; adds the additional term to the element in the j-th column of the i-th data row to obtain the derived row corresponding to the i-th data row, where i and j are positive integers.

[0139] Optionally, the summation of the elements in the r derived rows is equal to the summation of the elements in the (r+1)th row of the first matrix; wherein the summation includes an XOR operation on the elements.

[0140] Furthermore, such as Figure 5 As shown, the server also includes other components such as the power supply component 504. Figure 5 The diagram only shows some components and does not mean that the server only includes... Figure 5 The components shown.

[0141] The memory 501 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0142] The communication component 503 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), and other technologies.

[0143] The power supply component 504 is used to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0144] In this embodiment, when a recovery request for any target object block in any data group of the dataset exists, the target object block can be recovered based on the remaining readable object blocks in the data group. This target object block can be a data block, a global check block, or a local check block within the data group. The global check block and local check block are obtained by encoding the data block, and can be recovered based on the remaining data blocks, global check block, and local check block when any data block within the data group is lost. The local check block contains the encoded information of the global check block, and can be recovered based on the data block and local check block when the global check block is lost. When a local check block is lost, it can be recovered based on the data blocks within the data group. Thus, local recovery of any object block within the data group is achieved. The sum of local check blocks in the dataset is linearly independent of the number of global check blocks in the dataset; therefore, lost object blocks can be recovered even when the number of lost object blocks in the dataset is greater than the number of global check blocks. Furthermore, this allows for the partial recovery of any object block within a group while simultaneously enhancing the fault tolerance of the local data recovery method.

[0145] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiments.

[0146] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for partial data recovery, characterized in that, include: The dataset contains multiple data groups, and the object blocks in any data group include: data blocks, global validation blocks, and local validation blocks; In response to a recovery request for any target object block in the data group, the target object block is recovered based on the remaining readable object blocks in the data group; In the data group, the global check block and the local check block are obtained by encoding the data block. The local check block is obtained by adding additional items to the encoding elements of the global check block so that the local check block contains the encoding information of the global check block. The sum of the local check blocks in the dataset is used as a new global check block. The new global check block is linearly independent of the original global check blocks in the dataset. The method further includes: responding to a recovery request for a first number of object blocks in the dataset, and recovering the first number of object blocks based on the remaining readable object blocks in the dataset, wherein the first number is greater than the number of global verification blocks in the dataset.

2. The method according to claim 1, characterized in that, The steps for obtaining the global verification block and the local verification block include: Determine the first matrix with dimensions r+1, where r is a positive integer; Based on any r data rows in the first matrix, r derivative rows are generated accordingly; the r derivative rows are used to encode r local check blocks, and each derivative row contains elements from the data rows used to generate the derivative row. The second matrix is ​​obtained by replacing all data rows in the first matrix except for any r data rows with the r derived rows; By appending an identity matrix to the second matrix, the encoding matrix is ​​obtained. The data blocks in the dataset are encoded using the encoding matrix to obtain a global check block and local check blocks corresponding to each of the multiple data groups.

3. The method according to claim 2, characterized in that, In the encoding matrix, any r+1 columns are full-rank matrices.

4. The method according to claim 2 or 3, characterized in that, The first matrix is ​​a Cauchy matrix.

5. The method according to claim 2, characterized in that, Based on any r data rows in the first matrix, generate r derived rows, including: Based on the correspondence between the elements of different columns in the r data rows and the data group during matrix calculation, additional terms are added to the elements of different columns in the r data rows to obtain the r derived rows; Within the same derived row, additional items are added to the elements of the column corresponding to any data group; in different derived rows, the data groups corresponding to the elements to which additional items are added are different.

6. The method according to claim 4, characterized in that, Based on the correspondence between the elements of different columns in the r data rows and the data group during matrix calculation, additional items are added to the elements of different columns in the r data rows, including: For the element in the j-th column of the i-th data row among the r data rows, the additional item of the element in the j-th column of the i-th data row is determined based on the element in the j-th column of the r+1 row of the first matrix; Add the additional item to the element in the j-th column of the i-th data row to obtain the derived row corresponding to the i-th data row, where i and j are positive integers.

7. The method according to claim 6, characterized in that, The summation of the elements in the r derived rows is equal to the summation of the elements in the (r+1)th row of the first matrix; wherein the summation includes an XOR operation on the elements.

8. A server, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it can implement the data partial recovery method according to any one of claims 1-7.

Images