A fast disk repair method and system for high-density storage service servers

Abstract

This invention discloses a fast disk repair method and system for high-density storage service servers, belonging to the field of computer storage. It includes: determining all stripes requiring repair based on the faulty disk; reading all surviving blocks of the stripes requiring repair from the normal disk; calculating the time required for each surviving block to be read from the disk into memory; constructing all possible [a,b] combinations; calculating the disk repair time T for each [a,b] combination based on the time required for each surviving block to be read from the disk into memory; and using the [a,b] combination with the minimum disk repair time to repair the faulty disk; where 'a' represents the intra-strip parallelism, 'b' represents the inter-strip parallelism, and a*b = the number of data blocks that can be held in memory. This invention selects reasonable intra-strip parallelism and inter-strip parallelism based on the current transfer speed of all disks in the server, minimizing disk repair time and thus repairing lost data as quickly as possible, improving repair performance and system reliability.

Description

Technical Field

[0001] This invention belongs to the field of computer storage, and more specifically, relates to a fast disk repair method and system for high-density storage service servers. Background Technology

[0002] High-density storage servers (servers with a large number of disks) are currently used in data centers to reduce costs (physical space, electricity, maintenance, etc.). In storage systems, erasure coding is a widely used and efficient storage method that guarantees data reliability. There are many types of erasure codes, with Reed-Solomon (RS) codes being the most popular and widely used in industrial production. Specifically, RS codes encode k original data blocks into n blocks (including k original data blocks and m check blocks), and store these n blocks on n different storage nodes (e.g., disks, servers, etc.). According to the properties of RS codes, any k blocks can reconstruct all n blocks. However, RS codes suffer from the classic problem of high repair overhead; that is, RS codes require reading k data blocks to repair one data block, which introduces significant transmission overhead (network transmission, disk I / O, etc.).

[0003] Currently, there is considerable research on erasure coding repair, but most of this research focuses on optimizing for limited network resources in distributed storage systems. These studies leverage the advantages of distributed storage systems—numerous storage nodes, ample memory, and abundant computing resources—by decomposing the repair process into sub-repair processes and distributing these sub-repair processes across various child nodes in parallel. However, unlike distributed storage systems, high-density storage servers utilize a bus for transmission, and the bus is often not the bottleneck. The bottleneck for high-density storage servers lies in their limited memory, which is quite scarce. Furthermore, high-density storage servers often have a large number of disks. When data is read from numerous disks in parallel, insufficient memory capacity (memory contention) will limit the performance of erasure coding repair in high-density storage servers. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a fast disk repair method and system for high-density storage service servers, aiming to solve the problem that existing repair methods cannot solve memory contention.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a fast disk repair method for high-density storage servers, wherein the high-density storage server employs RS erasure coding, and the method includes:

[0006] Identify all stripes that need repair based on the faulty disk;

[0007] Read all surviving blocks from the normal disk that require stripe repair;

[0008] Calculate the time required for each surviving block to be read from disk into memory;

[0009] Construct all possible combinations of [a, b];

[0010] Based on the time required for each surviving block to be read from the disk into memory, calculate the disk repair time T for each [a,b] combination;

[0011] The faulty disk is repaired using the [a,b] combination that corresponds to the shortest disk repair time.

[0012] Where 'a' represents the parallelism within a stripe, 'b' represents the parallelism between stripes, and 'a*b' equals the number of data blocks that can be accommodated in memory.

[0013] Preferably, the calculation of the disk repair time T under each [a,b] combination is as follows:

[0014] Construct a two-dimensional array, wherein the element in the i-th row and j-th column of the two-dimensional array represents the time required for the j-th surviving block of the i-th strip to be read from the disk into memory;

[0015] Sort the elements of each row of the two-dimensional array in ascending order;

[0016] For each row of the two-dimensional array, group every 'a' elements and take the maximum value of each group. Sum the maximum values ​​of all groups and replace the original row with the sum to obtain a one-dimensional array.

[0017] Sort the elements in a one-dimensional array in ascending order;

[0018] A sliding window of length b is used to slide the one-dimensional array. After each slide, the minimum value within the sliding window is selected, and the minimum value is subtracted from all elements within the window. The sliding window moves forward one unit. When the sliding window reaches the end, the maximum value within the sliding window is taken. The minimum value within the sliding window and the last maximum value are added together to obtain the disk repair time T.

[0019] It should be noted that the present invention preferably determines the disk repair time T in the above manner. Since the disk repair time can be theoretically optimized by following the above theoretical analysis process, this calculation method can minimize the disk repair time.

[0020] Preferably, the calculation of the time required for each surviving block to be read from the disk into memory is as follows:

[0021] (1) Obtain the current transfer speed of each normal disk;

[0022] (2) Divide the size of the surviving block by the corresponding disk transfer speed to get the time required for the disk to transfer the surviving block.

[0023] Preferably, an active testing mechanism is used to obtain the current transfer speed of each normal disk.

[0024] Preferably, after identifying the stripes that need repair, and before reading the surviving blocks, the same stripes on different disks are deduplicated.

[0025] It should be noted that this invention deduplicates identical stripes. Since reading k surviving blocks from a stripe can simultaneously recover all damaged blocks of that stripe, deduplication can be performed on the same stripe from different damaged disks. Without deduplication, more redundant disk I / O would be performed, reducing repair performance.

[0026] Preferably, the value of a ranges from [1, k], where k represents the number of original data blocks in each stripe of the RS erasure code.

[0027] To achieve the above objectives, in a second aspect, the present invention provides a fast disk repair system for high-density storage servers, comprising: a processor and a memory;

[0028] The memory is used to store computer programs or instructions;

[0029] The processor is used to execute the computer program or instructions in the memory, causing the method described in the first aspect to be performed.

[0030] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0031] This invention provides a fast disk repair method and system for high-density storage service servers. Based on the current transfer speed of all disks in the server, it selects an appropriate degree of parallelism within and between stripes, thereby minimizing disk repair time and recovering lost data as quickly as possible, thus improving repair performance and system reliability. Attached Figure Description

[0032] Figure 1 This invention provides a flowchart of a fast disk repair method for high-density storage service servers.

[0033] Figure 2 This is a schematic diagram of a high-density storage server with faulty disks, provided in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of a two-dimensional array provided in an embodiment of the present invention.

[0035] Figure 4A schematic diagram illustrating the calculation process of repair time under the [4,1] combination and [2,2] combination provided in the embodiments of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] Figure 1 This invention provides a flowchart of a fast disk repair method for high-density storage service servers. Figure 1 As shown, the method includes:

[0038] (S1) An active testing mechanism is adopted to obtain the current transfer speed of all non-damaged disks. That is, a small portion of data in the disk is read first, and the reading time is counted to calculate the disk transfer speed. The data block size is divided by the disk transfer speed to obtain the time required for the disk to transfer one data block.

[0039] (S2) Form a two-dimensional array from all the stripes contained in the damaged disks. Preferably, duplicate stripes on different disks are deduplicated. Each row of the two-dimensional array represents a stripe, and each stripe is a one-dimensional array containing k elements, where the k elements represent the time required for the k surviving blocks of the stripe to be read from the disk into memory.

[0040] (S3) Sort each row in the array in ascending order.

[0041] (S4) Assume the parallelism within a stripe is 'a', and the parallelism between stripes is 'b' (b = memory capacity / a). Take the maximum value of every 'a' elements in each row of the above two-dimensional array, and sum these 'a' maximum values. The sum replaces the original row. In this way, the original two-dimensional array is reduced to a one-dimensional array.

[0042] (S5) Sort the one-dimensional array in ascending order.

[0043] (S6) Create a sliding window of size b, starting from the first b elements of the one-dimensional array. Each time, select the minimum value within the sliding window and subtract that minimum value from all elements within the window. Move the sliding window one unit to the right. When the sliding window reaches the end, take the maximum value M within the sliding window. Finally, add all the minimum values ​​and the last maximum value to obtain the disk repair time T.

[0044] (S7) Iterate through the intra-strip parallelism from 2 to k, calculating the disk repair time T each time according to steps S4-S6. Finally, select the intra-strip parallelism when T reaches its minimum value, which is the desired result. After obtaining the optimal intra-strip parallelism, the optimal inter-strip parallelism can be obtained through mathematical calculation.

[0045] (S8) Repair can be performed based on the calculated optimal intra-strip parallelism and inter-strip parallelism.

[0046] Example

[0047] In a storage system based on Reed-solomon codes (RS codes), for each stripe, all k data points are encoded using RS codes to generate nk parity blocks. These n blocks form a stripe, and any k blocks from these n blocks can be used to recover the original data.

[0048] Figure 2 This is a schematic diagram of a high-density storage server with faulty disks, provided as an embodiment of the present invention. Figure 2 As shown in the example (n,k)=(5,4), C0~C9 are 10 data blocks, where C0~C4 is the first stripe, C5~C9 is the second stripe, and each disk stores one block from each stripe. When data blocks C0 and C9 are lost, C1~C4 and C5~C8 can be used to recover the original data C0 and C9.

[0049] Step S1: Obtain the transfer speed v1-v8 from disk to memory for C1-C8, and calculate the transfer time t1-t8.

[0050] Step S2. Construct a 2*4 two-dimensional array Figure 3 This is a schematic diagram of a two-dimensional array provided in an embodiment of the present invention. Figure 3 As shown, the two-dimensional array is

[0051] Step S3. Sort in ascending order

[0052] Step S4. Assume that the number of original data blocks in each stripe of the RS erasure code is 4, and the possible combinations of [a,b] are [1,4], [2,2], and [4,1]. Taking the combination [2,2] as an example, the following explanation is provided.

[0053] Figure 4 This is a schematic diagram illustrating the calculation process of repair time under the [4,1] and [2,2] combinations provided in embodiments of the present invention. Figure 4As shown, for each row of the two-dimensional array, the maximum value is taken in pairs, resulting in 1 and 3; these two maximum values ​​are then added together, and the sum 4 replaces the original row. In this way, the original two-dimensional array is reduced to a one-dimensional array.

[0054] Step S5. Sort in ascending order.

[0055] Step S6. Create a sliding window of size 2. The sliding window starts from the first two elements of the one-dimensional array. Each time, select the minimum value of 4 within the sliding window and subtract the minimum value of 4 from all elements within the window to obtain... The sliding window moves forward one unit. When the sliding window reaches the end, the maximum value within the sliding window, 1, is taken. Finally, all the minimum values ​​of 4 are added to the last maximum value of 1, and the result of 5 is the disk repair time.

[0056] Step S7. Similarly, the disk repair time is 8 for the [1,4] combination and 7 for the [4,1] combination. Select the [2,2] combination when T reaches its minimum value.

[0057] Step S8. Perform repair using a parallelism of 2 within a stripe and a parallelism of 2 between stripes.

[0058] When a disk failure occurs, this invention rationally selects the intra-strip parallelism and inter-strip parallelism based on the current transfer speed of all disks, and then performs repair according to the optimal intra-strip parallelism and inter-strip parallelism, thereby minimizing disk repair time, improving repair performance, and increasing system reliability.

[0059] In the implementation, the above-mentioned fast disk repair method for high-density storage servers can reasonably select intra-strip parallelism and inter-strip parallelism according to the current disk transfer status.

[0060] This method can speed up the repair of a single disk, and it can also be used to speed up the repair of multiple disks.

[0061] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fast disk repair method for high-density storage servers, wherein the high-density storage server employs RS erasure coding, characterized in that... The method includes: Identify all stripes that need repair based on the faulty disk; Read all surviving blocks from the normal disk that require stripe repair; Calculate the time required for each surviving block to be read from disk into memory; Construct all possible combinations of [a, b]; Based on the time required for each surviving block to be read from the disk into memory, calculate the disk repair time T for each [a,b] combination; The faulty disk is repaired using the [a,b] combination that corresponds to the shortest disk repair time. Where 'a' represents the parallelism within a stripe, and 'b' represents the parallelism between stripes, and... = The number of data blocks that can be held in memory; The calculation of disk repair time T for each [a,b] combination is as follows: Construct a two-dimensional array, wherein the element in the i-th row and j-th column of the two-dimensional array represents the time required for the j-th surviving block of the i-th strip to be read from the disk into memory; Sort the elements of each row of the two-dimensional array in ascending order; For each row of the two-dimensional array, group every 'a' elements and take the maximum value of each group. Sum the maximum values ​​of all groups and replace the original row with the sum to obtain a one-dimensional array. Sort the elements in a one-dimensional array in ascending order; A sliding window of length b is used to slide the one-dimensional array. After each slide, the minimum value within the sliding window is selected, and the minimum value is subtracted from all elements within the window. The sliding window moves forward one unit. When the sliding window reaches the end, the maximum value within the sliding window is taken. The minimum value within the sliding window and the last maximum value are added together to obtain the disk repair time T.

2. The method as described in claim 1, characterized in that, The time required for each surviving block to be read from disk into memory is calculated as follows: (1) Obtain the current transfer speed of each normal disk; (2) Divide the size of the surviving block by the corresponding disk transfer speed to get the time required for the disk to transfer the surviving block.

3. The method as described in claim 2, characterized in that, An active testing mechanism is used to obtain the current transfer speed of each normal disk.

4. The method as described in claim 1, characterized in that, After identifying the stripes that need repair, deduplication is performed on the same stripes on different disks before reading the surviving blocks.

5. The method according to any one of claims 1 to 4, characterized in that, The value of a ranges from [1, k], where k represents the number of original data blocks in each stripe of the RS erasure code.

6. A fast disk repair system for high-density storage service servers, characterized in that, include: Processor and memory; The memory is used to store computer programs or instructions; The processor is configured to execute the computer program or instructions in the memory, such that the method of any one of claims 1 to 5 is performed.

Images