Data recovery method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a data recovery method and device, electronic equipment and a readable medium, and relates to the technical field of distributed storage, the method comprises the following steps: after a disk in a distributed cluster storage pool fails, determining the number of missing disks of each PG; the recovery priority of each PG is determined according to the number of the missing disks of each PG, and the recovery priority corresponding to the PG with the larger number of the missing disks is higher; sequentially writing the PGs into a recovery queue according to the recovery priorities from high to low, wherein the PGs with the high recovery priorities are arranged in front of the PGs with the low recovery priorities; and sequentially recovering the data of each PG in the recovery queue from front to back. According to the invention, reliable recovery of the PG data can be realized.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a data recovery method, a data recovery device, an electronic device and a computer-readable medium. Background Art

[0002] With the continuous advancement of network technology, users have higher and higher requirements for data storage management. Traditional network storage systems have gradually been unable to meet user needs. In this environment, distributed storage systems have emerged. Distributed storage systems store data in a dispersed manner on multiple independent devices.

[0003] At the same time, in order to ensure the reliability of data in the distributed storage system, the distributed storage system generally has redundant data, which is mainly achieved through copy redundancy and error correction redundancy. When a disk in the distributed storage system fails, the PG to be restored can be written into the recovery queue, and then the data can be restored in sequence according to the recovery queue. However, if the disk is damaged continuously, the PG may suffer further disk damage before completing data recovery, resulting in the problem that the data cannot be recovered. Summary of the invention

[0004] The embodiment of the present invention provides a data recovery method, device, electronic device and computer-readable storage medium to solve the problem that PG data cannot be reliably recovered when the disk is continuously damaged.

[0005] The embodiment of the present invention discloses a data recovery method, comprising:

[0006] After a disk failure occurs in a distributed cluster storage pool, determine the number of missing disks for each PG;

[0007] Determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher recovery priority;

[0008] Writing each PG into the recovery queue in sequence according to the recovery priority from high to low; the PG with high recovery priority is placed before the PG with low recovery priority;

[0009] The data of each PG in the recovery queue is restored in sequence from front to back.

[0010] In one embodiment, determining the recovery priority of each PG according to the number of missing disks of each PG includes:

[0011] Determine the number of surviving copies of each PG according to the number of missing disks of each PG;

[0012] The recovery priority of each PG is determined according to the number of surviving copies of each PG, wherein the PG with fewer surviving copies has a higher recovery priority.

[0013] In one embodiment, determining the recovery priority of each PG according to the number of missing disks of each PG includes:

[0014] Obtaining the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool;

[0015] The recovery priority of each PG is determined based on the number of missing disks and the number of redundant blocks of each PG.

[0016] In one embodiment, determining the recovery priority of each PG according to the number of missing disks and the number of redundant blocks of each PG includes:

[0017] Obtain the difference between the number of missing disks and the number of redundant blocks of each of the PGs;

[0018] When the difference is not greater than 0, it is determined that among the PGs whose difference is not greater than 0, the PGs with more missing disks have a higher recovery priority;

[0019] When the difference is greater than 0, it is determined that the PG with the difference greater than 0 cannot perform data recovery.

[0020] In one embodiment, the method further comprises:

[0021] Check whether there is data being restored currently;

[0022] If so, stop restoring the data and restore the PG with the highest recovery priority.

[0023] In one embodiment, stopping the data being recovered and recovering the PG with the highest recovery priority includes:

[0024] Detecting whether the recovery priority of the data being recovered is the same as the highest recovery priority;

[0025] If they are the same, the PG with the highest recovery priority is placed after the data being recovered, and the PG with the highest recovery priority is restored after the data being recovered is completed;

[0026] If not, the data being restored will be stopped and the PG with the highest recovery priority will be restored.

[0027] In one embodiment, the method further comprises:

[0028] Scanning the recovery queue at preset intervals;

[0029] Detect whether each of the PGs in the recovery queue is arranged from high to low according to the recovery priority;

[0030] If not, the PGs in the recovery queue are rearranged in descending order of the recovery priority.

[0031] The embodiment of the present invention further discloses a data recovery device, comprising:

[0032] A first determination module is used to determine the number of missing disks of each PG after a disk failure occurs in a distributed cluster storage pool;

[0033] A second determination module is used to determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher recovery priority;

[0034] A writing module, used for writing each of the PGs into the recovery queue in sequence according to the recovery priority from high to low, and the PG with a high recovery priority is arranged before the PG with a low recovery priority;

[0035] The recovery module is used to restore the data of each PG in the recovery queue in sequence from front to back.

[0036] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0037] The memory is used to store computer programs;

[0038] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0039] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0040] The embodiments of the present invention include the following advantages:

[0041] After continuous disk failures occur in the storage pool, the criticality level of each PG can be determined based on the number of missing disks in each data group PG. The more missing disks, the higher the criticality level. The recovery priority of each PG is determined based on the criticality level of each PG. The higher the criticality level, the higher the corresponding recovery priority. In this way, PGs with high criticality levels can be restored first. Even if disk damage continues to occur, it will not affect the data recovery of PG, thereby improving the reliability of data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a step flow of a data recovery method provided in an embodiment of the present invention. Figure 1 ;

[0043] Figure 2 This is a step flow of a data recovery method provided in an embodiment of the present invention. Figure 2 ;

[0044] Figure 3 PG recovery diagram provided in the embodiment of the present invention Figure 1 ;

[0045] Figure 4 PG recovery diagram provided in the embodiment of the present invention Figure 2 ;

[0046] Figure 5 PG recovery diagram provided in the embodiment of the present invention Figure 3 ;

[0047] Figure 6 PG recovery diagram provided in the embodiment of the present invention Figure 4 ;

[0048] Figure 7 is a structural block diagram of a data recovery device provided in an embodiment of the present invention;

[0049] Figure 8 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0050] Fig. 9 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] As a well-known distributed storage system, Ceph uses two data redundancy methods to achieve cluster data reliability. One is the copy method. When a storage pool using this data redundancy method is used, each data object in the storage cluster will be copied and distributed to different disks (disks are called OSDs in the cluster). When a disk fails and is damaged, all the data stored on the disk will be lost. Since there are exactly the same data copies on other disks, no actual data loss will occur. The other method is the erasure code method. When a storage pool using this data redundancy method is used, a data object in the storage pool will be divided into k data blocks. These k data blocks are used as the input of a certain algorithm to generate m redundant blocks. These k+m blocks will be distributed and stored on different disks. When a disk fails and is damaged, all the data stored on the disk will be lost. Since there are blocks calculated by the algorithm on other disks, if the number of damaged blocks does not exceed m at the same time, the storage cluster can recover the lost blocks through the same algorithm (this is a mathematical guarantee based on the algorithm), so no actual data loss will occur.

[0053] While implementing two data redundancy methods, the distributed storage system Ceph also implements a self-healing mechanism. When a disk is damaged and all data on the disk is lost, the storage cluster will sense the situation and automatically recover the missing data. Specifically, the data stored on the disk is actually divided into groups. A data object uniquely corresponds to a data group, called PG. In the replica mode, any PG has a unique ID. The PG entity will exist on different OSDs and be completely consistent. In the erasure code mode, the PG with the same ID will exist in k+m blocks. When the disk fails, the storage cluster will use the data group PG on other disks to recover the missing data. In terms of process, the replica mode and the erasure code mode are slightly different. In the storage pool of the replica mode, because all PGs are the same, the lost PG can be directly copied from other replica PGs to recover the lost PG. The storage pool of the erasure code mode needs to obtain k blocks of PG as input according to the algorithm and recalculate the lost PG data. The final result is that the entire data group of the missing data will be restored to the current normal disk. The whole process is automatically performed by Ceph and cannot be manually intervened.

[0054] In an actual production environment, disk damage may occur continuously. For example, in a storage pool with 3 copies of redundancy, 2 disks are damaged continuously, or in a storage pool with k+m of 4+2, 2 disks are damaged continuously. In this case, although the data is still reliable and not lost, if another disk is damaged before the lost data is fully recovered, some data will be permanently lost and cannot be recovered. In the implementation of Ceph, PGs of data groups with different degrees of damage are restored equally, and due to the speed limit of recovery, all PGs cannot be restored together. Therefore, Ceph uses the recovery queue method for sorting. However, this default recovery method has hidden dangers. Data with fewer missing PGs may be restored first, while data with more missing PGs may be further damaged by disks and lost before data recovery is completed.

[0055] In order to solve the above technical problems, the present disclosure proposes a data recovery method, device, electronic device and readable medium.

[0056] First, the technical terms used in this disclosure are introduced:

[0057] Distributed storage: A storage method that stores data in multiple nodes.

[0058] PG (placement group): The group of data stored on the disk of the distributed storage cluster Ceph. All data objects in the cluster uniquely correspond to one PG.

[0059] Erasure Code: A coding technology used for data redundancy and fault tolerance. It divides data into multiple fragments and generates some redundant fragments, so that even if some of the fragments are damaged or lost, data can still be recovered through the redundant fragments.

[0060] Reference Figure 1 , shows a flow chart of the steps of a data recovery method provided in an embodiment of the present invention, which may specifically include the following steps S101-S104:

[0061] S101. After a disk in a distributed cluster storage pool fails, determine the number of missing disks of each PG.

[0062] After a disk failure occurs in a distributed cluster storage pool, all PGs in the storage pool are immediately checked to determine the number of missing disks for each PG.

[0063] S102. Determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher corresponding recovery priority.

[0064] Specifically, since the distributed storage system has two redundancy modes, it is necessary to determine the grading method of the recovery priority by analyzing the redundancy mode used by the storage pool.

[0065] When the redundancy mode is the copy mode, the recovery priority of each PG is determined according to the number of missing disks of each PG, including the following sub-steps A1-A2:

[0066] A1. Determine the number of surviving copies of each PG based on the number of missing disks of each PG.

[0067] A2. Determine the recovery priority of each PG according to the number of surviving copies of each PG, where the PG with fewer surviving copies has a higher recovery priority.

[0068] For storage pools with replica redundancy, calculate the number of replicas and prioritize PG recovery based on the number of missing replicas.

[0069] For example: if the number of replicas in the replica redundancy mode is 3, then the level when only 1 of the surviving replicas is the highest, and the level when only 2 of the surviving replicas is the second highest. In other words, the level when only 2 of the missing replicas is the highest, and the level when only 1 of the missing replicas is the second highest.

[0070] Because, if the PG with 2 surviving copies is restored first, and the PG with 1 surviving copy is placed in the recovery queue waiting, if a disk failure continues to occur during the recovery, the PG with 1 surviving copy may lose the remaining copy and be unable to recover the data. Therefore, during the recovery, the PG with 1 surviving copy can be restored first, and the PG with 2 surviving copies can be placed in the recovery queue waiting. In this way, even if a disk failure continues to occur during the recovery, the PG with 2 surviving copies may lose another copy. At this time, the number of surviving copies becomes 1, but data recovery can be achieved without data loss even if its number of surviving copies becomes 1.

[0071] When the redundancy mode is the erasure coding mode, the recovery priority of each PG is determined according to the number of missing disks of each PG, including the following sub-steps B1-B2:

[0072] B1. Obtain the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool.

[0073] For the erasure code method, the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool can be obtained first.

[0074] The number of redundant blocks in the erasure code can be obtained by the following scheme:

[0075] If the total number of data blocks n and the number of original data blocks k are known, the number of redundant blocks can be calculated by the formula m = nk. For example, there are 8 data blocks in total, of which 6 are original data blocks, then the number of redundant blocks m = 8-6 = 2.

[0076] In a storage system, the configuration parameters of erasure coding are usually recorded in a configuration file. These parameters can be obtained by viewing the storage system's management interface or directly accessing the storage device's configuration file. Different storage systems may have different configuration file formats and locations. For example, in the Ceph storage system, its configuration file stores detailed information about the Erasure-Coded Pool, including the parameters of the encoding scheme.

[0077] Many storage systems provide an application programming interface (API) to query information related to erasure coding. Taking the OpenStack Swift storage system as an example, you can use its API to query the erasure coding configuration in object storage. By sending a specific request to the Swift API, you can get the erasure coding parameters used by the bucket (Bucket) or object (Object), including the number of redundant blocks.

[0078] Some programming languages ​​provide libraries specifically for handling storage and erasure codes. For example, in Python, there are some third-party libraries that can interact with storage systems, such as boto3, which is used to interact with the S3 storage service. If the S3 bucket uses erasure coding technology, the boto3 library can be used to query the relevant erasure code parameters to obtain the number of redundant blocks.

[0079] B2. Determine the recovery priority of each PG based on the number of missing disks and redundant blocks of each PG.

[0080] Specifically, according to the number of missing disks and redundant blocks of each PG, the recovery priority of each PG is determined, including the following sub-steps B11-B13:

[0081] B11. Get the difference between the number of missing disks and the number of redundant blocks of each PG.

[0082] The more disks are missing, the more difficult it is to recover and the greater the risk. The fewer disks are missing, the easier it is to recover and the less risk. Therefore, after obtaining the number of redundant blocks in the erasure code, the difference between the number of missing disks and the number of redundant blocks of each PG can be obtained.

[0083] B12. When the difference is not greater than 0, it is determined that among the PGs whose difference is not greater than 0, the PG with a greater number of missing disks has a higher recovery priority.

[0084] B13. When the difference is greater than 0, it is determined that the PG with the difference greater than 0 cannot perform data recovery.

[0085] When the difference is greater than 0, it indicates that the number of missing disks cannot recover the PG, and the PG does not need to be written into the recovery queue. When the difference is not greater than 0, it indicates that the number of missing disks can still recover the PG, and the PG can be written into the recovery queue.

[0086] In the present disclosure, the more missing disks a PG has, the higher the corresponding recovery priority. Taking the erasure code 4+2 as an example, as long as the number of missing disks is greater than or equal to 4, data recovery can be achieved. If the number of missing disks is 4, the calculated difference is 2. If the number of missing disks is 2, the calculated difference is 0. If the number of missing disks is 1, the calculated difference is -1. When the difference is 2, it indicates that data recovery cannot be performed and it will not be written to the recovery queue. When the difference is 2, if another disk is damaged next, the data at this time cannot be recovered. For the case where the difference is -1, if another disk is damaged next, the data at this time can also be recovered. Therefore, in the present disclosure, the recovery priority of the PG with a difference of 2 will be set to the highest, and it will be placed first in the recovery queue to restore the PG first.

[0087] In the present disclosure, since the recovery queue of Ceph does not perform reliability grading on the PG to be recovered, the present disclosure uses an algorithm to perform reliability statistics and grading on the PG (the reliability here represents the recovery priority in the above embodiment, the lower the reliability, the higher the recovery priority, and the higher the reliability, the lower the recovery priority). The present disclosure will first obtain the values ​​of the erasure code ratios k and m of the storage pool, and then count all the PGs in the entire storage pool, and grade them according to the number of missing PG copies. The recovery priority of the PG that is missing m blocks has the highest grading level, and the recovery priority of the PG that is missing m-1 blocks has the second highest grading level, and so on. Set the recovery priority of the PG with the highest recovery priority to the highest (255), and immediately seize the operating resources of the PG currently being recovered, and immediately recover. Set the recovery priority of the PG with the second highest recovery priority to the second highest (254), but do not seize the operating resources of the PG being recovered, but wait for data recovery.

[0088] S103. Write each PG into the recovery queue in sequence from high to low recovery priority.

[0089] The PG with a high recovery priority is placed ahead of the PG with a low recovery priority, that is, the PG with the highest recovery priority is at the first position in the recovery queue.

[0090] The recovery queue in the present disclosure is a queue for storing tasks or data items waiting for recovery operations. During recovery, the recovery is performed in sequence from the front to the back of the recovery queue.

[0091] It is worth noting that if multiple PGs have the same recovery priority, the PGs with the same recovery priority will be arranged in the recovery queue in the order of detection time.

[0092] S104. Restore the data of each PG in the recovery queue in order from front to back.

[0093] In the present disclosure, after continuous disk failures occur in the storage pool, the criticality of each PG can be determined according to the number of missing disks of each data group PG, and the recovery priority of each PG can be determined based on the criticality of each PG. The higher the criticality, the higher the corresponding recovery priority. In this way, PGs with high criticality can be restored first, and even if disk damage continues to occur, it will not affect the data recovery of PGs with a large number of missing disks, thereby improving the reliability of data recovery.

[0094] In one embodiment, Figure 2 As shown, the above method further includes the following sub-steps S105-S107:

[0095] S105 , scanning the recovery queue at preset intervals.

[0096] Scanning at preset intervals means that the system will check the recovery queue at preset intervals.

[0097] For example, in a software system, a timer can be used to implement scanning at set intervals. For example, in programming languages, many high-level languages ​​(such as Java) provide timer-related classes (java.util.Timer and java.util.TimerTask). The recovery queue can be scanned by setting the interval of the timer and writing the code to scan the recovery queue in the run method of the timer task.

[0098] In some system-level recovery operations, the operating system's task scheduler can be used to implement interval scanning. By configuring the task scheduling policy, the system can start the process of scanning the recovery queue at a specific time interval. For example, in a Linux system, you can use the cron task scheduling tool to set the execution rules of the scheduled task to regularly execute the script of the scan recovery queue.

[0099] S106. Check whether each PG in the recovery queue is arranged from high to low according to the recovery priority.

[0100] Check the recovery queue at a preset time interval to detect whether the PGs in the recovery queue are arranged from high to low according to the recovery priority.

[0101] S107. If not, then rearrange the PGs in the recovery queue in descending order of recovery priority.

[0102] If it is detected that the PGs in the recovery queue are not arranged from high to low according to the recovery priority, that is, the order is chaotic, the recovery queue will be re-sorted to put the PGs with high recovery priority in front of the PGs with low recovery priority, so that the PGs with high recovery priority can restore data first.

[0103] In the present disclosure, the entire recovery queue is scanned periodically to check whether the PGs in the current queue are arranged according to the level of grading. If there is a change, the grading is immediately re-enforced to ensure that PGs with higher levels can achieve faster recovery.

[0104] After PG grading, the recovery queue is scanned and maintained regularly to ensure the effectiveness of grading recovery.

[0105] In the present disclosure, after scanning the PG with the highest recovery priority, it will also be detected whether there is any data currently being recovered; if so, the data being recovered will be stopped and the PG with the highest recovery priority will be restored.

[0106] Specifically: check whether the recovery priority of the data being recovered is the same as the highest recovery priority; if so, put the PG with the highest recovery priority after the data being recovered, wait until the recovery of the data being recovered is complete, and then restore the PG with the highest recovery priority; if not, stop restoring the data being recovered and restore the PG with the highest recovery priority.

[0107] Check whether the recovery priority of the data being recovered is the highest. If the recovery priority of the data being recovered is the highest, do not preempt the current recovery resources, but wait until the recovery is complete before recovering the PG with the highest recovery priority. If it is detected that the recovery priority of the data being recovered is not the highest, directly preempt the current recovery resources and recover the PG with the highest recovery priority.

[0108] The present disclosure combines Ceph's data recovery mechanism. When a disk failure occurs in a storage pool and automatic data recovery is initiated, the queue priority of the PG being recovered is intervened, and the recovery priority of the PG is graded, so that more critical data is restored first, thereby reducing the risk of the storage pool. The solution in the present disclosure is described in detail below through specific examples:

[0109] Taking erasure code 4+2 as an example, when a disk in the storage pool is damaged, by default, Figure 3 As shown, the PGs involved are missing 1 block at this time, and the recovery mechanism of Ceph is automatically started. The PGs involved are put into a recovery queue, and then the data is recovered from the head of the recovery queue. At the same time, the number of PGs recovering data is 1.

[0110] When Figure 4 In the case shown, another disk is damaged in the storage pool. At this time, the number of missing blocks of some PGs reaches 2. At this time, the order of the recovery queue does not change, and recovery continues according to the previous recovery queue. That is, the difference between the number of missing disks of all PGs and the number of redundant codes 2 is no more than 0.

[0111] If in Figure 4 The priority of each PG in the recovery queue is not changed. When the following occurs Figure 5 In the case shown, Figure 4 If the missing data in the storage pool has not been fully restored, and the storage pool has another disk failure, the number of missing PG blocks exceeds 2, resulting in permanent data loss. Figure 5 In this case, the difference between the number of missing disks of PG and the number of redundant codes 2 is greater than 0.

[0112] Through the method disclosed in this disclosure, Figure 4 The recovery queue is reordered. The recovery priority of the PG with 2 missing blocks is higher than that of the PG with 1 missing block. The PG with 2 missing blocks is written in front of the recovery queue, and the PG with 1 missing block is written behind the PG with 2 missing blocks in the recovery queue. When recovering data, the data of the PG with 2 missing blocks is restored first. Figure 6 shown.

[0113] exist Figure 3-Figure 6 In the process of recovery, recovering indicates that the process is in progress, and recover-wait indicates that the process is waiting to be recovered.

[0114] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0115] Reference Figure 7, shows a structural block diagram of a data recovery device provided in an embodiment of the present invention, which may specifically include the following modules:

[0116] A first determination module 11 is used to determine the number of missing disks of each PG after a disk failure occurs in a distributed cluster storage pool;

[0117] When performing data recovery, after a disk failure occurs in a distributed cluster storage pool, the first determination module 11 will immediately detect all PGs in the storage pool to determine the number of missing disks of each PG.

[0118] The second determination module 12 is used to determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher recovery priority;

[0119] The second determination module 12 determines the recovery priority of each PG according to the number of missing disks of each PG determined by the first determination module 11. In the present disclosure, the PG with more missing disks has a higher recovery priority.

[0120] The writing module 13 is used to write each PG into the recovery queue in sequence according to the recovery priority from high to low; the PG with high recovery priority is arranged before the PG with low recovery priority;

[0121] The writing module 13 writes the recovery priorities of the PGs determined by the second determining module 12 into the recovery queue in descending order of the recovery priorities.

[0122] The recovery module 14 is used to restore the data of each PG in the recovery queue in sequence from front to back.

[0123] After the writing module 13 writes each PG in the recovery queue in order from high to low recovery priority, the recovery module 14 will restore each PG in the recovery queue in order from high to low recovery priority.

[0124] In one embodiment, the second determining module is specifically configured to:

[0125] Determine the number of surviving copies of each PG according to the number of missing disks of each PG;

[0126] The recovery priority of each PG is determined according to the number of surviving copies of each PG, wherein the PG with fewer surviving copies has a higher recovery priority.

[0127] In one embodiment, the second determining module is specifically configured to:

[0128] Obtaining the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool;

[0129] The recovery priority of each PG is determined based on the number of missing disks and the number of redundant blocks of each PG.

[0130] In one embodiment, in determining the recovery priority of each PG according to the number of missing disks and the number of redundant blocks of each PG, the second determination module is specifically used to:

[0131] Obtain the difference between the number of missing disks and the number of redundant blocks of each of the PGs;

[0132] When the difference is not greater than 0, it is determined that among the PGs whose difference is not greater than 0, the PGs with more missing disks have a higher recovery priority;

[0133] When the difference is greater than 0, it is determined that the PG with the difference greater than 0 cannot perform data recovery.

[0134] In one embodiment, the apparatus further comprises:

[0135] A first detection module is used to detect whether there is data being restored currently;

[0136] The stop module is used to stop the data being restored and restore the PG with the highest restoration priority if the first detection module detects that there is data currently being restored.

[0137] In one embodiment, the stopping module is specifically used to:

[0138] Detecting whether the recovery priority of the data being recovered is the same as the highest recovery priority;

[0139] If they are the same, the PG with the highest recovery priority is placed after the data being recovered, and the PG with the highest recovery priority is restored after the data being recovered is completed;

[0140] If not, the data being restored will be stopped and the PG with the highest recovery priority will be restored.

[0141] In one embodiment, the apparatus further comprises:

[0142] A scanning module, used for scanning the recovery queue at preset intervals;

[0143] A second detection module is used to detect whether each of the PGs in the recovery queue is arranged from high to low according to the recovery priority;

[0144] The sorting module is used to rearrange the PGs in the recovery queue in order from high to low according to the recovery priority if the detection module detects that the PGs in the recovery queue are not arranged in order from high to low according to the recovery priority.

[0145] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0147] Memory 1303, used for storing computer programs;

[0148] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0149] After a disk failure occurs in a distributed cluster storage pool, determine the number of missing disks for each PG;

[0150] Determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher corresponding recovery priority;

[0151] Writing each PG into the recovery queue in sequence according to the recovery priority from high to low; the PG with high recovery priority is placed before the PG with low recovery priority;

[0152] The data of each PG in the recovery queue is restored in sequence from front to back.

[0153] In one embodiment, determining the recovery priority of each PG according to the number of missing disks of each PG includes:

[0154] Determine the number of surviving copies of each PG according to the number of missing disks of each PG;

[0155] The recovery priority of each PG is determined according to the number of surviving copies of each PG, wherein the PG with fewer surviving copies has a higher recovery priority.

[0156] In one embodiment, determining the recovery priority of each PG according to the number of missing disks of each PG includes:

[0157] Obtaining the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool;

[0158] The recovery priority of each PG is determined based on the number of missing disks and the number of redundant blocks of each PG.

[0159] In one embodiment, determining the recovery priority of each PG according to the number of missing disks and the number of redundant blocks of each PG includes:

[0160] Obtain the difference between the number of missing disks and the number of redundant blocks of each of the PGs;

[0161] When the difference is not greater than 0, it is determined that among the PGs whose difference is not greater than 0, the PGs with more missing disks have a higher recovery priority;

[0162] When the difference is greater than 0, it is determined that the PG with the difference greater than 0 cannot perform data recovery.

[0163] In one embodiment, it further includes:

[0164] Check whether there is data being restored currently;

[0165] If so, stop restoring the data and restore the PG with the highest recovery priority.

[0166] In one embodiment, stopping the data being recovered and recovering the PG with the highest recovery priority includes:

[0167] Detecting whether the recovery priority of the data being recovered is the same as the highest recovery priority;

[0168] If they are the same, the PG with the highest recovery priority is placed after the data being recovered, and the PG with the highest recovery priority is restored after the data being recovered is completed;

[0169] If not, the data being restored will be stopped and the PG with the highest recovery priority will be restored.

[0170] In one embodiment, it further includes:

[0171] Scanning the recovery queue at preset intervals;

[0172] Detect whether each of the PGs in the recovery queue is arranged from high to low according to the recovery priority;

[0173] If not, the PGs in the recovery queue are rearranged in descending order of the recovery priority.

[0174] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0175] The communication interface is used for communication between the above terminal and other devices.

[0176] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0177] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0178] like Fig. 9 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the data recovery method described in the above embodiment.

[0179] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data recovery method described in the above embodiment.

[0180] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0181] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0182] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data recovery method, characterized in that: include: After a disk failure occurs in a distributed cluster storage pool, determine the number of missing disks for each PG; Determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher recovery priority; Writing each PG into the recovery queue in sequence according to the recovery priority from high to low; the PG with high recovery priority is placed before the PG with low recovery priority; The data of each PG in the recovery queue is restored in sequence from front to back.

2. The method according to claim 1, characterized in that Determining the recovery priority of each PG according to the number of missing disks of each PG includes: Determine the number of surviving copies of each PG according to the number of missing disks of each PG; The recovery priority of each PG is determined according to the number of surviving copies of each PG, wherein the PG with fewer surviving copies has a higher recovery priority.

3. The method according to claim 1, characterized in that Determining the recovery priority of each PG according to the number of missing disks of each PG includes: Obtaining the number of redundant blocks in the erasure code corresponding to the distributed cluster storage pool; The recovery priority of each PG is determined based on the number of missing disks and the number of redundant blocks of each PG.

4. The method according to claim 3, characterized in that: Determining the recovery priority of each PG according to the number of missing disks and the number of redundant blocks of each PG includes: Obtain the difference between the number of missing disks and the number of redundant blocks of each of the PGs; When the difference is not greater than 0, it is determined that among the PGs whose difference is not greater than 0, the PGs with more missing disks have a higher recovery priority; When the difference is greater than 0, it is determined that the PG with the difference greater than 0 cannot perform data recovery.

5. The method according to claim 1, characterized in that The method further comprises: Check whether there is data being restored currently; If so, stop restoring the data and restore the PG with the highest recovery priority.

6. The method according to claim 5, characterized in that The stopping of the data being recovered and recovering the PG with the highest recovery priority includes: Detecting whether the recovery priority of the data being recovered is the same as the highest recovery priority; If they are the same, the PG with the highest recovery priority is placed after the data being recovered, and the PG with the highest recovery priority is restored after the data being recovered is completed; If not, the data being restored will be stopped and the PG with the highest recovery priority will be restored.

7. The method according to claim 1, characterized in that The method further comprises: Scanning the recovery queue at preset intervals; Detect whether each of the PGs in the recovery queue is arranged from high to low according to the recovery priority; If not, the PGs in the recovery queue are rearranged in descending order of the recovery priority.

8. A data recovery device, characterized in that: include: A first determination module is used to determine the number of missing disks of each PG after a disk failure occurs in a distributed cluster storage pool; A second determination module is used to determine the recovery priority of each PG according to the number of missing disks of each PG, wherein the PG with more missing disks has a higher recovery priority; A writing module, used for writing each of the PGs into the recovery queue in sequence according to the recovery priority from high to low, and the PG with a high recovery priority is arranged before the PG with a low recovery priority; The recovery module is used to restore the data of each PG in the recovery queue in sequence from front to back.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images