A data recovery method, device, computer device and storage medium

Abstract

The present invention relates to the field of distributed storage, and in particular to a data recovery method, device, computer equipment, and storage medium. The method comprises: dividing the data related to the recovery of a certain erroneous data in each cabinet into the same number of slices as the number of cabinets, and assigning a different slice number to each slice for each cabinet; the control hardware corresponding to the multiple cabinets respectively and in parallel perform multiplication operations on the respective multiple slices in sequence, each beat corresponding to a slice number, and the same beat corresponding to different cabinets having different slice numbers; the control hardware corresponding to the multiple cabinets accumulates the multiplication operation results corresponding to the same slice number in different cabinets in the order of the first beat to the last beat to obtain an accumulated result corresponding to each slice number; all accumulated results are spliced based on the slice number to generate recovery data. The solution of the present invention maximizes the use of hardware resources for data transmission and operation, significantly reducing data recovery time.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular to a data recovery method, device, computer equipment and storage medium. Background Art

[0002] Faced with the storage requirements of massive data, distributed storage, with its advantages of low cost and good scalability, has gradually replaced the dominant position of unified storage and has received increasing attention in both theoretical research and practical applications. Distributed storage systems often use inexpensive disks as storage nodes, and the reliability of each storage node is often not very high. On the other hand, a distributed storage system usually contains many nodes, and node failures are common in the system due to software and hardware failures, human errors, and other reasons. To improve the data reliability of distributed storage systems and ensure that data collection nodes can reconstruct the original files with a high probability, it is necessary to store a certain amount of redundancy in addition to the original data. This ensures that even if some nodes fail, the system can still operate normally and the data collection nodes can still decode and recover the original files. At the same time, to maintain system reliability, failed nodes need to be repaired in a timely manner. Therefore, designing a good node repair mechanism is very important.

[0003] At present, the simple structure of distributed storage is as follows Figure 1A As shown, Figure 1A In the example, there are four cabinets, such as Figure 1A There are cabinets 1234 in the example, and each cabinet has multiple hard disks. The hard disks have different chunk sizes based on the operation. Each chunk is the basic unit of storage operation, such as Figure 1A As shown in the gray block in the middle cabinet, when operating in the upper control, the visible operation unit is the block. Figure 1A In the system, each cabinet will have its own control hardware, which connects and controls multiple disks through PCIE and other means. After receiving the needs of the upper layer, it divides the data into different numbers of blocks for corresponding encoding and decoding. The cabinets are connected to each other through network cables and other means, and then the corresponding commands are transferred and controlled by plugging into the switch. The upper layer of the switch is the host, which is user-oriented and breaks down different user needs into different commands, which are issued separately for corresponding control. Assuming that in the current example, a disk data error occurs in cabinet 4 (disk error, power failure, disk removal, etc.), it needs to be recovered based on erasure (RS or RAID), which is expressed as As can be seen from the above formula, the principle of data recovery is: use all the surviving data, multiply by the corresponding calculation parameters, and then add them together. The way to implement it in existing distributed storage is as follows Figure 1BAs shown, when data in cabinet 4 is erroneous and needs to be recovered, data is read from cabinet 123 respectively, and then relevant operations are performed on the hardware corresponding to cabinet 4. The results are then stored in the disk corresponding to the cabinet or returned to the host.

[0004] When data in cabinet 4 experiences an error and requires recovery, data is read from cabinets 1, 2, and 3, and then the corresponding hardware in cabinet 4 performs the relevant operations. The results are then stored in the corresponding disk in the cabinet in the second step, or returned to the host. In this case, we assume that the amount of data on the disk in cabinet 4 is wr_m, the data transmission speed supported by the switch protocol is sw_sp, the data transmission speed under the hardware transmission protocol within the cabinet is ra_sp, and the hardware operation speed is hw_sp. In this scenario, with k data cabinets and only one cabinet experiencing an error, the time required to complete data recovery is expressed as:

[0005]

[0006] This shows that traditional distributed system cabinet error data recovery only relies on the cabinet control hardware where the error occurred. In other words, the serial working mode does not fully utilize the bandwidth for data transmission, resulting in a certain speed loss and low recovery efficiency, and therefore urgently needs to be improved. Summary of the Invention

[0007] In view of this, it is necessary to provide a data recovery method, device, computer equipment and storage medium to address the above technical problems.

[0008] According to a first aspect of the present invention, there is provided a data recovery method, which is applied to a distributed storage system comprising a plurality of cabinets, each cabinet corresponding to a control hardware for performing operations, the method comprising the following steps:

[0009] Divide the data related to recovering a certain error data in each cabinet into shards equal to the number of cabinets, and assign a different shard number to each shard in each cabinet;

[0010] The control hardware corresponding to the multiple cabinets performs multiplication operations on their respective multiple shards in parallel and in sequence, where each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers;

[0011] The control hardware corresponding to the multiple cabinets accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain the accumulated results corresponding to each shard number.

[0012] All the accumulated results are concatenated based on the shard number to generate the recovered data.

[0013] In some embodiments, the step of dividing the data related to recovering a certain erroneous data in each cabinet into shards having the same number as the number of cabinets, and assigning a different shard number to each shard in each cabinet includes:

[0014] For each enclosure, do the following:

[0015] Obtain the total number of bits wr_m of data related to recovering a certain error data and the number k of all cabinets;

[0016] Divide the data into k slices according to the order of each bit of data;

[0017] Assign fragment numbers wr_m1, ..., wr_m to the obtained k fragments in sequence k .

[0018] In some embodiments, each slice includes wr_m / k bits, and the bits in the same slice are continuous.

[0019] In some embodiments, the step of dividing the data related to recovering a certain erroneous data in each cabinet into shards having the same number as the number of cabinets, and assigning a different shard number to each shard in each cabinet further includes:

[0020] If wr_m / k is not an integer, any k-1 shards are evenly divided according to the maximum ratio of wr_m / k, and the remaining data is treated as a separate shard.

[0021] In some embodiments, the control hardware corresponding to the plurality of cabinets sequentially performs multiplication operations on the respective plurality of shards in parallel in separate beats, wherein each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers, including the following steps:

[0022] Set one of the cabinet control hardware to calculate wr_m1, ..., wr_m in order from the first beat to the kth beat. k Multiplication operation of the corresponding slice;

[0023] Refer to the slice numbers corresponding to each beat of the control hardware of the cabinet that has been set up, and set the control hardware of other cabinets to synchronously calculate the multiplication operations corresponding to different slice numbers in each beat.

[0024] In some embodiments, the control hardware corresponding to the multiple cabinets sequentially accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain the accumulated result corresponding to each shard number, including:

[0025] For each enclosure, do the following:

[0026] After completing a multiplication operation, the slice number wr_m corresponding to the multiplication operation performed in the current shot is obtained. i ;

[0027] Get the second-beat segment number wr_m i The cabinet i corresponding to the multiplication operation is executed;

[0028] Set the fragment number wr_m i The corresponding multiplication result is sent from the control hardware of the current cabinet to cabinet i;

[0029] The control hardware corresponding to cabinet i is used to match the second slice number wr_m i Perform the multiplication operation on the received fragment number wr_m i The corresponding multiplication results are added;

[0030] The control hardware corresponding to cabinet i sends the addition result to the third pair of slices wr_m i The cabinet corresponding to the multiplication operation is performed;

[0031] Until the last beat of the fragment number wr_m i The control hardware that performs the multiplication operation performs the addition operation and then outputs it.

[0032] In some embodiments, the step of concatenating all accumulated results based on the shard number to generate restored data includes:

[0033] Get the shard number corresponding to the last multiplication operation performed by the control hardware of each cabinet;

[0034] The accumulated results of the last beat output of the control hardware of each cabinet are sorted in the order of the slice numbers wr_m1, ..., wr_m k Splicing is performed to obtain the recovered data.

[0035] According to a second aspect of the present invention, there is provided a data recovery device for use in a distributed storage system comprising a plurality of cabinets, each cabinet corresponding to a control hardware for performing operations, the device comprising:

[0036] A sharding module configured to divide data related to recovery of a certain erroneous data in each cabinet into shards having the same number as the number of cabinets, and assigning a different shard number to each shard based on each cabinet;

[0037] A multiplication module is configured to execute multiplication operations on the respective shards in parallel and in steps by the control hardware corresponding to the multiple cabinets, wherein each step corresponds to a shard number, and the same step in different cabinets corresponds to different shard numbers;

[0038] An accumulation module is configured to accumulate the multiplication results corresponding to the same shard number in different cabinets in sequence from the first beat to the last beat by the control hardware corresponding to the multiple cabinets, so as to obtain an accumulation result corresponding to each shard number one by one;

[0039] The splicing module is configured to splice all the accumulated results based on the shard numbers to generate restored data.

[0040] According to a third aspect of the present invention, a computer device is further provided, the computer device comprising:

[0041] at least one processor; and

[0042] The memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned data recovery method when executing the program.

[0043] According to a fourth aspect of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the aforementioned data recovery method is executed.

[0044] The above-mentioned data recovery method designs a parallel pipeline for data recovery based on the existing distributed storage structure, divides data transmission into inside and outside the cabinet, and effectively combines data calculations based on the characteristics of the hardware acceleration calculation module of each cabinet. The data is sharded inside each cabinet, and the control hardware of each cabinet performs calculations in parallel, so that the hardware resources for data transmission and calculation can be maximized, thereby significantly reducing the time required for data recovery and improving the calculation speed and data recovery efficiency.

[0045] In addition, the present invention also provides a data recovery device, a computer device and a computer-readable storage medium, which can also achieve the above technical effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0047] Figure 1A This is a schematic diagram of the traditional distributed storage system structure;

[0048] Figure 1B Schematic diagram of the data recovery process for an error in one of the distributed storage cabinets;

[0049] Figure 2 A flowchart of a data recovery method provided by one embodiment of the present invention;

[0050] Figure 3 A schematic diagram of data transmission between cabinets when recovering data using the method of the present invention is provided in one embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of processing data of each shard when recovering a single error using the method of the present invention;

[0052] Figure 5 A schematic diagram of a data recovery device according to another embodiment of the present invention;

[0053] Figure 6 This is a diagram of the internal structure of a computer device in another embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0055] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. Subsequent embodiments will not explain this one by one.

[0056] In one embodiment, please refer to Figure 2 and Figure 3 As shown, the present invention provides a data recovery method 100, which is applied to a distributed storage system including multiple cabinets, each cabinet corresponding to a control hardware for performing operations, and the method includes the following steps:

[0057] Step 101: Divide the data related to restoring a certain erroneous data in each cabinet into shards equal in number to the number of cabinets, and assign a different shard number to each shard based on each cabinet;

[0058] Step 102: The control hardware corresponding to the multiple cabinets performs multiplication operations on the respective multiple shards in parallel and in sequence, wherein each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers.

[0059] Step 103: The control hardware corresponding to the multiple cabinets sequentially accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain an accumulation result corresponding to each shard number.

[0060] Step 104: concatenate all accumulated results based on the shard numbers to generate restored data.

[0061] The above-mentioned data recovery method designs a parallel pipeline for data recovery based on the existing distributed storage structure, divides data transmission into inside and outside the cabinet, and effectively combines data calculations based on the characteristics of the hardware acceleration calculation module of each cabinet. The data is sharded inside each cabinet, and the control hardware of each cabinet performs calculations in parallel, so that the hardware resources for data transmission and calculation can be maximized, thereby significantly reducing the time required for data recovery and improving the calculation speed and data recovery efficiency.

[0062] In some embodiments, the aforementioned step 101 of dividing the data related to recovering a certain erroneous data in each cabinet into shards equal in number to the number of cabinets, and assigning a different shard number to each shard in each cabinet specifically includes:

[0063] For each enclosure, do the following:

[0064] Obtain the total number of bits wr_m of data related to recovering a certain error data and the number k of all cabinets;

[0065] Divide the data into k slices according to the order of each bit of data;

[0066] Assign fragment numbers wr_m1, ..., wr_m to the obtained k fragments in sequence k .

[0067] Preferably, each slice includes wr_m / k bits, and the bits in the same slice are continuous.

[0068] Preferably, the aforementioned step 101 of dividing the data related to recovering a certain erroneous data in each cabinet into shards having the same number as the number of cabinets, and assigning a different shard number to each shard in each cabinet also includes:

[0069] If wr_m / k is not an integer, any k-1 shards are split evenly according to the maximum wr_m / k ratio, and the remaining data is treated as a separate shard. For example, if there are three cabinets and the total number of bits wr_m required to recover a particular error is 8, bits 1 through 3 can be used as one shard, bits 4 through 6 as another, and bits 7 and 8 as a third.

[0070] In some embodiments, the aforementioned step 102, in which the control hardware corresponding to the multiple cabinets respectively performs multiplication operations on the respective multiple shards in parallel and in sequence, wherein each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers, specifically includes:

[0071] Set one of the cabinet control hardware to calculate wr_m1, ..., wr_m in order from the first beat to the kth beat. k Multiplication operation of the corresponding slice;

[0072] Refer to the slice numbers corresponding to each beat of the control hardware of the cabinet that has been set up, and set the control hardware of other cabinets to synchronously calculate the multiplication operations corresponding to different slice numbers in each beat.

[0073] In some embodiments, the aforementioned step 103, in which the control hardware corresponding to the plurality of cabinets sequentially accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain an accumulated result corresponding to each shard number, specifically includes:

[0074] For each enclosure, do the following:

[0075] After completing a multiplication operation, the slice number wr_m corresponding to the multiplication operation performed in the current shot is obtained. i ;

[0076] Get the second-beat segment number wr_m i The cabinet i corresponding to the multiplication operation is executed;

[0077] Set the fragment number wr_m i The corresponding multiplication result is sent from the control hardware of the current cabinet to cabinet i;

[0078] The control hardware corresponding to cabinet i is used to match the second slice number wr_m i Perform the multiplication operation on the received fragment number wr_m i The corresponding multiplication results are added;

[0079] The control hardware corresponding to cabinet i sends the addition result to the third pair of slices wr_m i The cabinet corresponding to the multiplication operation is performed;

[0080] Until the last beat of the fragment number wr_m i The control hardware that performs the multiplication operation performs the addition operation and then outputs it.

[0081] In some embodiments, the aforementioned step 104 of concatenating all accumulated results based on the shard numbers to generate restored data specifically includes:

[0082] Get the shard number corresponding to the last multiplication operation performed by the control hardware of each cabinet;

[0083] The accumulated results of the last beat output of the control hardware of each cabinet are sorted in the order of the slice numbers wr_m1, ..., wr_mk Splicing is performed to obtain the recovered data.

[0084] In another embodiment, to facilitate understanding of the solution of the present invention, a distributed storage system including four cabinets is taken as an example. This embodiment provides a data recovery method. To facilitate understanding, a single error is used as an example. The specific implementation method is as follows:

[0085] The data volume of wr_m is divided into k data slices based on the number of cabinets k. The number of each data slice is wr_m / k. The cabinets are allowed to transmit data to each other in the order of 1→2→...→(k-1)→k. The corresponding data of the received cabinets are calculated respectively. Taking the above example, the data involved in the calculation and transmission of each cabinet is divided into four slices, denoted as wr_m1, wr_m2, wr_m3 and wr_m4. Figure 4 The data is transmitted to each other in the manner shown: cabinet 1 calculates the multiplication of the fragment numbers wr_m1, wr_m2, wr_m3, and wr_m4 from the first beat to the fourth beat, cabinet 2 calculates the multiplication of wr_m2, wr_m3, wr_m4, and wr_m1 from the first beat to the fourth beat, cabinet 3 calculates the multiplication of wr_m3, wr_m4, wr_m1, and wr_m2 from the first beat to the fourth beat, cabinet 4 calculates wr_m4, wr_m1, and wr_m2 from the first beat to the fourth beat, and cabinet 5 calculates wr_m6, wr_m7, wr_m8, wr_m9, wr_m10, wr_m110, wr_m120, wr_m121, wr_m122, wr_m13, wr_m14, wr_m15, wr_m16, wr_m17, wr_m18, wr_m19, wr_m223, wr_m24, wr_m3, wr_m4, wr_m110, wr_m124, wr_m19, wr_m110, wr_m125 The multiplication operation of r_m1 and wr_m2 is performed. Taking the transmission process of wr_m4, the first slice of data involved in the operation, wr_m1, as an example, the fourth cabinet first calculates the product of the wr_m4 slice in the first cycle and passes this product to cabinet 3, which performs the multiplication operation on wr_m4 in the second cycle. The control hardware of cabinet 3 adds its calculated product to the product passed by cabinet 4 and then passes it to cabinet 2. This continues until cabinet 1, which outputs the result of its addition. The transmission process of the wr_m1, wr_m2, and wr_m3 slices follows the same steps. Finally, the control hardware of cabinet 2 outputs the accumulated result of the wr_m1 slice, the control hardware of cabinet 3 outputs the accumulated result of the wr_m2 slice, and the control hardware of cabinet 4 outputs the accumulated result of the wr_m3 slice.

[0086] like Figure 4As shown, the data to be transferred within each cabinet is first divided into wr_m / k slices based on the data to be calculated and the number of cabinets. The data to be transferred is then sequentially divided into wr_m / k slices. In the example above, the data to be calculated within cabinets 1-4 is divided into four slices, wr_m1-wr_m4. The dashed boxes represent the multiplication of data by parameters. Because the PCIe transmission speed within the hardware or the operating speed of the hardware module is inherently faster than the cross-cabinet data transfer under switch control, the two can be pipelined. Specifically, the algorithm-based multiplication operation is first performed based on the size of the data slices divided within the cabinet. The data is then transferred sequentially. After the cross-cabinet data is transferred, an XOR operation is performed. This pipelined operation is repeated to complete all the operations. For example, in cabinet 4, the data to be calculated within cabinet 4 is first divided into four slices, wr_m1-wr_m4. In the first beat, a multiplication operation is performed on wr_m4 of cabinet 4, which is shown as the dotted box. The multiplied parameter is the parameter configured by the erasure algorithm (RS or RAID) for the calculation cabinet 4. The time required at this time is:

[0087]

[0088] The multiplication operation time in the second beat is completed in advance because the data transmission time is longer. The corresponding time required at this time is:

[0089]

[0090] The time of the third and fourth beats is equal to the second beat. All cabinets perform calculations and transmissions at the same time. The completed data is finally calculated according to Figure 4 The total recovery time for a single error is expressed as:

[0091]

[0092] For the sake of comparison, the difference between the time loss of the traditional data recovery method and the data recovery method of the present invention is obtained by subtracting the formula 1 from the formula 4, and the difference is expressed as:

[0093] The difference between Formula 1 and Formula 4

[0094]

[0095] It is understandable that in the case of multiple errors, the computation time at this time is increased by x times because x intermediate values need to be generated. Since the number of errors is x times, the data transmission between cabinets and the data transmission within the cabinet are also x times. Therefore, when all are increased by x times, the traditional data recovery method also increases by x times because of the cross-cabinet transmission and computation. Therefore, the time loss can be obtained as follows:

[0096]

[0097] The data recovery method of the present invention has at least the following beneficial technical effects: based on the user needs or scenario requirements under distributed storage when downgraded reading, data recovery and other operations are required, based on the existing distributed storage structure, hardware design and workflow, a parallel pipeline for data recovery is designed, which divides data transmission into inside and outside the cabinet, and effectively combines data calculations based on the characteristics of the hardware acceleration calculation module of each cabinet, so that the hardware resources for data transmission and calculation can be maximized, thereby achieving the advantage of reducing working time and improving calculation speed.

[0098] In another embodiment, please refer to Figure 5 As shown, this embodiment provides a data recovery device 200, which is applied to a distributed storage system including multiple cabinets, each cabinet corresponding to a control hardware for performing operations, and the device 200 includes:

[0099] A sharding module 201 is configured to divide the data related to recovering a certain error data in each cabinet into shards of the same number as the number of cabinets, and assign a different shard number to each shard based on each cabinet;

[0100] The multiplication module 202 is configured to execute multiplication operations on the respective shards in parallel and in steps by the control hardware corresponding to the multiple cabinets, wherein each step corresponds to a shard number, and the same step in different cabinets corresponds to different shard numbers;

[0101] The accumulation module 203 is configured to accumulate the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat by the control hardware corresponding to the multiple cabinets, so as to obtain an accumulation result corresponding to each shard number.

[0102] The splicing module 204 is configured to splice all the accumulated results based on the fragment numbers to generate restored data.

[0103] The above-mentioned data recovery device designs a parallel pipeline for data recovery based on the existing distributed storage structure, divides data transmission into inside and outside the cabinet, and effectively combines data calculations based on the characteristics of the hardware acceleration calculation module of each cabinet. The data is sharded inside each cabinet, and the control hardware of each cabinet performs calculations in parallel, so that the hardware resources for data transmission and calculation can be maximized, thereby significantly reducing the time required for data recovery and improving the calculation speed and data recovery efficiency.

[0104] In some embodiments, the sharding module 201 is further configured to:

[0105] For each enclosure, do the following:

[0106] Obtain the total number of bits wr_m of data related to recovering a certain error data and the number k of all cabinets;

[0107] Divide the data into k slices according to the order of each bit of data;

[0108] Assign fragment numbers wr_m1, ..., wr_m to the obtained k fragments in sequence k .

[0109] In some embodiments, each slice includes wr_m / k bits, and the bits in the same slice are continuous.

[0110] In some embodiments, the sharding module 201 is further configured to:

[0111] If wr_m / k is not an integer, any k-1 shards are evenly divided according to the maximum ratio of wr_m / k, and the remaining data is treated as a separate shard.

[0112] In some embodiments, the multiplication module 202 is further configured to:

[0113] Set one of the cabinet control hardware to calculate wr_m1, ..., wr_m in order from the first beat to the kth beat. k Multiplication operation of the corresponding slice;

[0114] Refer to the slice numbers corresponding to each beat of the control hardware of the cabinet that has been set up, and set the control hardware of other cabinets to synchronously calculate the multiplication operations corresponding to different slice numbers in each beat.

[0115] In some embodiments, the accumulation module 203 is further configured to:

[0116] For each enclosure, do the following:

[0117] After completing a multiplication operation, the slice number wr_m corresponding to the multiplication operation performed in the current shot is obtained. i ;

[0118] Get the second-beat segment number wr_m i The cabinet i corresponding to the multiplication operation is executed;

[0119] Set the fragment number wr_m i The corresponding multiplication result is sent from the control hardware of the current cabinet to cabinet i;

[0120] The control hardware corresponding to cabinet i is used to match the second slice number wr_m i Perform the multiplication operation on the received fragment number wr_m iThe corresponding multiplication results are added;

[0121] The control hardware corresponding to cabinet i sends the addition result to the third pair of slices wr_m i The cabinet corresponding to the multiplication operation is performed;

[0122] Until the last beat of the fragment number wr_m i The control hardware that performs the multiplication operation performs the addition operation and then outputs it.

[0123] In some embodiments, the splicing module 204 is further configured to:

[0124] Get the shard number corresponding to the last multiplication operation performed by the control hardware of each cabinet;

[0125] The accumulated results of the last beat output of the control hardware of each cabinet are sorted in the order of the slice numbers wr_m1, ..., wr_m k Splicing is performed to obtain the recovered data.

[0126] It should be noted that the specific definition of the data recovery device can be found in the definition of the data recovery method above and will not be repeated here. The various modules in the above-mentioned data recovery device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0127] According to another aspect of the present invention, a computer device is provided. The computer device may be a server. Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the data recovery method described above is implemented, which is applied to a distributed storage system including multiple cabinets, each cabinet corresponding to a control hardware for performing operations, and the method includes the following steps:

[0128] Divide the data related to recovering a certain error data in each cabinet into shards equal to the number of cabinets, and assign a different shard number to each shard in each cabinet;

[0129] The control hardware corresponding to the multiple cabinets performs multiplication operations on their respective multiple shards in parallel and in sequence, where each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers;

[0130] The control hardware corresponding to the multiple cabinets accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain the accumulated results corresponding to each shard number.

[0131] All the accumulated results are concatenated based on the shard number to generate the recovered data.

[0132] According to another aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the data recovery method described above. The method is applied to a distributed storage system including multiple cabinets, each cabinet corresponding to a control hardware for performing operations. The method comprises the following steps:

[0133] Divide the data related to recovering a certain error data in each cabinet into shards equal to the number of cabinets, and assign a different shard number to each shard in each cabinet;

[0134] The control hardware corresponding to the multiple cabinets performs multiplication operations on their respective multiple shards in parallel and in sequence, where each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers;

[0135] The control hardware corresponding to the multiple cabinets accumulates the multiplication results corresponding to the same shard number in different cabinets in the order of the first beat to the last beat to obtain the accumulated results corresponding to each shard number.

[0136] All the accumulated results are concatenated based on the shard number to generate the recovered data.

[0137] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0138] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data recovery method, characterized in that: Applied to a distributed storage system comprising multiple cabinets, each cabinet corresponding to a control hardware for performing operations, the method comprises the following steps: The data related to the recovery of a certain error data in each cabinet is divided into the same number of slices as the number of cabinets, and a different slice number is assigned to each slice in each cabinet. The step of dividing the data related to the recovery of a certain error data in each cabinet into the same number of slices as the number of cabinets, and a different slice number is assigned to each slice in each cabinet includes the following operations for each cabinet: obtaining the total number of bits of data related to the recovery of a certain error data and the number of all cabinets ; Divide the data into fragments; for the obtained The shards are assigned shard numbers in turn ; Set the control hardware of one of the multiple cabinets to follow the order from first beat to Calculate the beat order separately Multiplication operations of corresponding shards; referring to the shard numbers corresponding to each beat of the control hardware of the already configured cabinet, the control hardware of other cabinets in the multiple cabinets is configured to synchronously calculate the multiplication operations corresponding to different shard numbers in each beat, where each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers; The control hardware corresponding to multiple cabinets performs the following operations on each cabinet: After completing a multiplication operation, the shard number corresponding to the multiplication operation performed on the current multiplication operation is obtained. ; Get the second-shot split number The cabinet that performs the multiplication operation ; Set the shard number The corresponding multiplication result is sent from the control hardware of the current cabinet to the cabinet ; By the cabinet The corresponding control hardware pairs the second beat pair fragment number Perform the multiplication operation on the received fragment number The corresponding multiplication results are added; The corresponding control hardware sends the addition result to the third beat pair slice number Execute the cabinet corresponding to the multiplication operation; until the last beat of the shard number The control hardware that performs the multiplication operation outputs the addition operation after it completes the addition operation; Get the slice number corresponding to the multiplication operation performed by the control hardware of each cabinet in the last beat; add the cumulative results of the control hardware of each cabinet in the last beat in the order of the slice number Splicing is performed to obtain the recovered data.

2. The method according to claim 1, characterized in that Each fragment contains the number of bits and all bits within the same slice are continuous.

3. The method according to claim 1, characterized in that The step of dividing the data related to recovering a certain erroneous data in each cabinet into slices having the same number as the number of cabinets, and assigning a different slice number to each slice based on each cabinet further includes: like If it is not an integer, take any Shards according to The largest proportion of data is evenly divided into shards, and the remaining data is treated as a separate shard.

4. A data recovery device, characterized in that: Applied to a distributed storage system comprising multiple cabinets, each cabinet corresponding to a control hardware for performing operations, the device comprises: The sharding module is configured to divide the data related to the recovery of a certain error data in each cabinet into a number of shards equal to the number of cabinets, and assign a different shard number to each shard based on each cabinet. The step of dividing the data related to the recovery of a certain error data in each cabinet into a number of shards equal to the number of cabinets, and assigning a different shard number to each shard based on each cabinet includes the following operations for each cabinet: obtaining the total number of bits of data related to the recovery of a certain error data; and the number of all cabinets ; Divide the data into fragments; for the obtained The shards are assigned shard numbers in turn ; The multiplication module is configured to set the control hardware of one of the multiple cabinets to be in the order from the first beat to the second beat. Calculate the beat order separately Multiplication operations of corresponding shards; referring to the shard numbers corresponding to each beat of the control hardware of the already configured cabinet, the control hardware of other cabinets in the multiple cabinets is configured to synchronously calculate the multiplication operations corresponding to different shard numbers in each beat, where each beat corresponds to a shard number, and the same beat in different cabinets corresponds to different shard numbers; The accumulation module is configured to use the control hardware corresponding to multiple cabinets to perform the following operations on each cabinet: each time a multiplication operation is completed, the slice number corresponding to the multiplication operation performed on the current beat is obtained. ; Get the second-shot split number The cabinet that performs the multiplication operation ; Set the shard number The corresponding multiplication result is sent from the control hardware of the current cabinet to the cabinet ; By the cabinet The corresponding control hardware pairs second beat pair fragment number Perform the multiplication operation on the received fragment number The corresponding multiplication results are added; The corresponding control hardware sends the addition result to the third beat pair slice number Execute the cabinet corresponding to the multiplication operation; until the last beat of the shard number The control hardware that performs the multiplication operation outputs the addition operation after it completes the addition operation; The splicing module is configured to obtain the slice number corresponding to the multiplication operation performed by the last beat of the control hardware of each cabinet; the cumulative result of the last beat output of the control hardware of each cabinet is sorted in the order of the slice number Splicing is performed to obtain the recovered data.

5. A computer device, characterized in that: include: at least one processor; as well as A memory storing a computer program executable in the processor, wherein the processor executes the method according to any one of claims 1 to 3 when executing the program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is performed.

Images