Data recovery method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data recovery method and device, computer equipment, a readable storage medium and a program product. According to the method, in response to a progress query instruction and an operation request instruction, the size relation between a key corresponding to the operation request instruction and a repaired position pointer can be obtained, and under the condition that the key corresponding to the operation request instruction is smaller than the repaired position pointer, data operation is executed on all copies in an operation field; inputting a corresponding operation log; and when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, executing data operation on the main copy in the operation field, inputting the operation log, and inputting the operation log in the to-be-recovered copy in the operation field, so as to avoid blocking the service request, and also consider the consistency and integrity of the data.

Description

Technical Field

[0001] The present application relates to the technical field of data recovery, and in particular to a data recovery method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] Ceph (distributed storage system) is an open source distributed storage system. As a powerful, flexible and scalable distributed storage system, it has been widely used and recognized in various application scenarios. Among them, Ceph achieves data reliability through a series of mechanisms such as automatic data recovery, thereby ensuring that data will not be lost or damaged during storage and transmission.

[0003] In traditional technologies, Ceph data recovery is achieved through incremental recovery or full recovery. However, in order to ensure data consistency, there will be a problem of blocking business requests during the data recovery process. Summary of the invention

[0004] Based on this, it is necessary to provide a data recovery method, apparatus, computer equipment, readable storage medium and program product that can continuously support business requests during the data recovery process in order to address the above technical problems.

[0005] In a first aspect, the present application provides a data recovery method, which is applied to a distributed storage system, and the method includes:

[0006] In response to the progress query instruction, obtaining a pointer to a repaired position of the object;

[0007] In response to the operation request instruction, obtaining a key corresponding to the operation request instruction;

[0008] When the key corresponding to the operation request instruction is smaller than the repaired position pointer, data operations are performed on all replicas in the operation field, and corresponding operation logs are recorded; wherein the operation field is the field pointed to by the key;

[0009] When the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, the data operation is performed on the primary copy in the operation field and the operation log is recorded, and the operation log is also recorded in the copy to be restored in the operation field.

[0010] In one embodiment, after receiving the operation request instruction, the method further includes:

[0011] Determine whether the object is in recovery state;

[0012] When the object is not in the recovery state, data operations are performed on the primary copy in the operation field and the operation log is recorded, and the operation log is also recorded in the copy to be recovered in the operation field.

[0013] In one embodiment, the method further comprises:

[0014] When the object is in the recovery state, proceed to step "when the key corresponding to the operation request instruction is smaller than the repaired position pointer, perform data operations on all copies in the operation field and record the corresponding operation log."

[0015] In one embodiment, the method further comprises:

[0016] In the case where no operation request instruction is received, each field is restored in the direction of increasing key based on the preset key corresponding to each field.

[0017] In one embodiment, after the steps of performing data operations on the primary copy in the operation field and recording the operation log, and recording the operation log in the copy to be restored in the operation field, the method further includes:

[0018] When the repaired position pointer is greater than the key corresponding to the operation request instruction, a data operation is performed on the copy to be restored in the operation field.

[0019] In a second aspect, the present application also provides a data recovery device, the device comprising:

[0020] A first acquisition module, configured to acquire a repaired position pointer of the object in response to a progress query instruction;

[0021] A second acquisition module, configured to acquire a key corresponding to the operation request instruction in response to the operation request instruction;

[0022] A first operation module, configured to perform data operations on all replicas in an operation field and record corresponding operation logs when the key corresponding to the operation request instruction is smaller than the repaired position pointer; wherein the operation field is the field pointed to by the key;

[0023] The second operation module is used to perform data operations on the primary copy in the operation field and record the operation log when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, and to record the operation log in the copy to be restored in the operation field.

[0024] In one embodiment, the above device further comprises:

[0025] A judgment module, used to judge whether the object is in a recovery state;

[0026] The third operation module is used to perform data operations on the primary copy in the operation field and record the operation log when the object is not in the recovery state, and record the operation log in the copy to be recovered in the operation field.

[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements any step of the above-mentioned data recovery method when executing the computer program.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any step of the above-mentioned data recovery method.

[0029] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any step of the above-mentioned data recovery method.

[0030] The above-mentioned data recovery method, device, computer equipment, readable storage medium and program product; the method responds to the progress query instruction and the operation request instruction, can obtain the size relationship between the key corresponding to the operation request instruction and the repaired position pointer, and when the key corresponding to the operation request instruction is smaller than the repaired position pointer, perform data operations on all copies in the operation field and record the corresponding operation log; when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, perform data operations on the primary copy in the operation field and record the operation log, and record the operation log in the copy to be recovered in the operation field to avoid blocking business requests, while also taking into account the consistency and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 An application environment diagram of a data recovery method in an embodiment;

[0033] Figure 2 is a schematic diagram of an incremental recovery method in one embodiment;

[0034] Figure 3 A schematic diagram of a full recovery method in an embodiment;

[0035] Figure 4A schematic diagram of a data recovery method in one embodiment;

[0036] Figure 5 One of the schematic diagrams of object fragment recovery of a distributed storage system in one embodiment;

[0037] Figure 6 The second schematic diagram of object fragment recovery of a distributed storage system in one embodiment;

[0038] Figure 7 The third schematic diagram of object fragment recovery of a distributed storage system in one embodiment;

[0039] Figure 8 is a structural block diagram of a data recovery device in one embodiment;

[0040] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] Each write operation issued by the user will record a placement group log, which includes: operation type, object name of the operation, and corresponding version information. Since the placement group log stores a limited number of entries, there are two data recovery methods, namely incremental recovery and full recovery, depending on whether data can be recovered through the log. Figure 2 As shown in Figure 1, incremental recovery builds a list of missing objects to be restored by comparing the missing logs between the recovery copy and the authoritative log, and then restores these objects one by one. Figure 3 As shown in the figure, full recovery is to copy all object data of the authoritative copy to the restored copy. However, this traditional recovery method will increase network bandwidth overhead and disk read and write amplification due to the large granularity of object recovery; if there are business requests during the recovery process, it will also cause business requests to be blocked.

[0043] The data recovery method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown in the figure, in the Ceph architecture, the core component of the Ceph cluster is RADOS (Reliable Autonomic Distributed ObjectStore), which stores data through OSD. LIBRADOS (Librados library) is an interface provided to upper-layer applications for direct access to RADOS. The Client interface layer implements different types of storage services through components such as Librados, RBD (RADOS Block Device), and RGW (RADOS Gateway). Data is organized in pools (objs in pools) and managed by PG groups (Placement Groups), which are determined by Crush rulesets to determine how data is distributed. Monitors (MONs) are responsible for monitoring the status of the entire cluster and maintaining the robustness of the cluster. Therefore, the data recovery method is executed for data recovery of objects in objs in pools, wherein Bucket Shard data recovery is performed strictly in order from small to large according to the keys in the object. Therefore, in response to a progress query instruction, a repaired position pointer of the object can be obtained; in response to an operation request instruction, a key corresponding to the operation request instruction can be obtained, by comparing the size relationship between the key of the fragment pointed to by the repaired position pointer of the object and the key corresponding to the operation request instruction, and when the key corresponding to the operation request instruction is smaller than the repaired position pointer, data operations are performed on all copies in the operation field, and corresponding operation logs are recorded; when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, data operations are performed on the primary copy in the operation field, and the operation logs are recorded, and operation logs are recorded in the copy to be recovered in the operation field, so as to ensure data consistency without blocking business requests.

[0044] In an exemplary embodiment, Figure 4 As shown, a data recovery method is provided, comprising:

[0045] S202: In response to the progress query instruction, obtain a pointer to a repaired position of the object.

[0046] The progress query instruction is to send the omap_recovered_to function, and omap_recovered_to is used to record the current recovery progress of the object. That is, according to omap_recovered_to, the key corresponding to the segment currently pointed to by the repaired position pointer of the object can be obtained.

[0047] S204, in response to the operation request instruction, obtain the key corresponding to the operation request instruction.

[0048] The operation request instruction is an instruction carrying a client op (operation) request. In response to the operation request instruction, the key corresponding to the segment pointed to by the operation request instruction can be obtained.

[0049] S206, when the key corresponding to the operation request instruction is less than the repaired position pointer, perform a data operation on all replicas in the operation field and record the corresponding operation log; where the operation field is the field pointed to by the key.

[0050] S208, when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, perform a data operation on the primary replica in the operation field, record the operation log, and record the operation log in the replicas to be recovered in the operation field.

[0051] By comparing the size relationship between the key corresponding to the operation request instruction and the repaired position pointer, it can be determined whether the segment corresponding to the key pointed to by the operation request instruction has been recovered. When the key corresponding to the operation request instruction is less than the repaired position pointer, it indicates that the segment (operation field) corresponding to the key pointed to by the operation request instruction has been recovered, and all replicas can perform the op operation, that is, perform a data operation on all replicas in the operation field and record the corresponding operation log; when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, it indicates that the operation field has not been recovered. To avoid blocking of business requests, an updated data operation is performed on the primary replica in the operation field, the operation log is recorded, and the operation log is recorded in the replicas to be recovered in the operation field.

[0052] Specifically, as Figure 5 shown, if in response to the progress query instruction, it is determined that omap_recovered_to is F, and in response to the updated operation request instruction, it is determined that the key at this time is B. From Figure 5 it can be seen that B < F. From this, it can be known that the segment corresponding to B has completed the recovery operation. Therefore, an updated data operation can be performed on all replicas in the operation field and the corresponding operation log can be recorded, thus ensuring the consistency of the data in the replicas.

[0053] Such as Figure 6As shown, if it is determined that omap_recovered_to is F in response to a progress query instruction, and it is determined that the key is Ba at this time in response to an inserted operation request instruction, then Figure 6 it can be seen that Ba < F. Therefore, data insertion operations can be performed on all replicas in the operation field, and the corresponding operation logs can be recorded, thereby ensuring data consistency in the replicas.

[0054] As Figure 5 shown, if it is determined that omap_recovered_to is F in response to a progress query instruction, and it is determined that the key is H at this time in response to an updated operation request instruction, then Figure 5 it can be seen that H > F. From this, it can be known that the segment corresponding to H has not completed the recovery operation. Therefore, data update operations can be performed on the primary replica in the operation field, and the operation logs can be recorded, and the operation logs can also be recorded in the replicas to be recovered in the operation field, so as to avoid blocking of business requests.

[0055] As Figure 7 shown, if it is determined that omap_recovered_to is F in response to a progress query instruction, and it is determined that the key is Ha at this time in response to an inserted operation request instruction, then Figure 6 it can be seen that Ha > F. Therefore, data update operations can be performed on the primary replica in the operation field, and the operation logs can be recorded, and the operation logs can also be recorded in the replicas to be recovered in the operation field, so as to avoid blocking of business requests.

[0056] Therefore, for the above data recovery method, in response to a progress query instruction and an operation request instruction, the size relationship between the key corresponding to the operation request instruction and the repaired position pointer can be obtained. When the key corresponding to the operation request instruction is less than the repaired position pointer, data operations are performed on all replicas in the operation field, and the corresponding operation logs are recorded; when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, data operations are performed on the primary replica in the operation field, and the operation logs are recorded, and the operation logs are also recorded in the replicas to be recovered in the operation field, so as to avoid blocking business requests, while also taking into account data consistency and integrity.

[0057] In an exemplary embodiment, after receiving an operation request instruction, the above method further includes:

[0058] Determine whether the object is in a recovery state.

[0059] When the object is not in a recovery state, data operations are performed on the primary replica in the operation field, and the operation logs are recorded, and the operation logs are also recorded in the replicas to be recovered in the operation field.

[0060] If the object is not in the recovery state, perform data operations on the primary copy in the operation field and record the operation log, and record the operation log in the copy to be recovered in the operation field to avoid blocking business requests, thereby improving the data recovery efficiency of the distributed storage system.

[0061] In an exemplary embodiment, the method further includes:

[0062] When the object is in the recovery state, proceed to step "when the key corresponding to the operation request instruction is smaller than the repaired position pointer, perform data operations on all copies in the operation field and record the corresponding operation log."

[0063] If it is determined that the object is in a recovery state, further judgment is performed based on the size relationship between the key corresponding to the operation request instruction and the repaired position pointer, thereby improving the reliability of the data recovery method.

[0064] In an exemplary embodiment, the method further includes:

[0065] In the case where no operation request instruction is received, the fields are restored in the direction of increasing keys based on the preset keys corresponding to the fields, wherein the preset keys are determined according to the preset restoration order of the fields.

[0066] When applied in the Ceph architecture, each field is restored in the direction of increasing key, which can match the Ceph architecture.

[0067] In an exemplary embodiment, after the steps of performing data operations on the primary copy in the operation field and recording the operation log, and recording the operation log in the copy to be restored in the operation field, the method further includes:

[0068] When the repaired position pointer is greater than the key corresponding to the operation request instruction, a data operation is performed on the copy to be restored in the operation field.

[0069] As the recovery of each field in the object is executed sequentially according to the preset keys corresponding to each field, there is a situation where the repaired position pointer is greater than the key corresponding to the operation request instruction. In the case where the repaired position pointer is greater than the key corresponding to the operation request instruction, the corresponding data operation is performed on the copy to be recovered in the operation field, thereby ensuring the consistency and integrity of the data.

[0070] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0071] Based on the same inventive concept, the embodiment of the present application also provides a data recovery device for implementing the data recovery method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data recovery device embodiments provided below can refer to the limitations on the data recovery method above, and will not be repeated here.

[0072] In an exemplary embodiment, Figure 8 As shown, a data recovery device 800 is provided, comprising: a first acquisition module 802, a second acquisition module 804, a first operation module 806 and a second operation module 808, wherein:

[0073] The first acquisition module 802 is used to obtain a repaired position pointer of the object in response to a progress query instruction.

[0074] The second acquisition module 804 is used for acquiring a key corresponding to the operation request instruction in response to the operation request instruction.

[0075] The first operation module 806 is used to perform data operations on all copies in the operation field and record the corresponding operation log when the key corresponding to the operation request instruction is smaller than the repaired position pointer; wherein the operation field is the field pointed to by the key.

[0076] The second operation module 808 is used to perform data operations on the primary copy in the operation field and record the operation log when the key corresponding to the operation request instruction is greater than or equal to the repaired location pointer, and to record the operation log in the copy to be restored in the operation field.

[0077] In an exemplary embodiment, the data recovery device 800 further includes: a judgment module and a third operation module.

[0078] Wherein, the judging module is used to judge whether the object is in a recovery state;

[0079] The third operation module is used to perform data operations on the primary copy in the operation field and record the operation log when the object is not in the recovery state, and to record the operation log in the copy to be recovered in the operation field.

[0080] In an exemplary embodiment, the data recovery device 800 further includes: a state determination module.

[0081] The state judgment module is used to enter the step of "when the key corresponding to the operation request instruction is less than the repaired position pointer, perform data operations on all copies in the operation field and enter the corresponding operation log" when the object is in a recovery state.

[0082] In an exemplary embodiment, the data recovery device 800 further includes: a sequential recovery module.

[0083] The sequential recovery module is used to recover each field in the direction of increasing keys based on the preset keys corresponding to each field when no operation request instruction is received.

[0084] In an exemplary embodiment, the data recovery device 800 further includes: a to-be-recovered copy execution module.

[0085] The copy-to-be-restored execution module is used to execute data operations on the copy-to-be-restored in the operation field when the repaired position pointer is greater than the key corresponding to the operation request instruction.

[0086] Each module in the above data recovery device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0087] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset keys corresponding to each field in the object. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data recovery method is implemented.

[0088] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of any one of the above data recovery methods.

[0090] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned data recovery methods are implemented.

[0091] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of any one of the above data recovery methods when executed by a processor.

[0092] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0094] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data recovery method, characterized in that: The method is applied to a distributed storage system, and the method comprises: In response to the progress query instruction, obtaining a pointer to a repaired position of the object; In response to the operation request instruction, obtaining a key corresponding to the operation request instruction; In the case where the key corresponding to the operation request instruction is smaller than the repaired position pointer, performing data operations on all copies in the operation field and recording corresponding operation logs; wherein the operation field is the field pointed to by the key; When the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, data operation is performed on the primary copy in the operation field and the operation log is recorded, and the operation log is also recorded in the copy to be restored in the operation field.

2. The method according to claim 1, characterized in that After receiving the operation request instruction, the method further includes: Determining whether the object is in a recovery state; In the case that the object is not in a recovery state, data operations are performed on the primary copy in the operation field and the operation log is recorded, and the operation log is also recorded in the copy to be recovered in the operation field.

3. The method according to claim 2, characterized in that The method further comprises: When the object is in the recovery state, proceed to step "when the key corresponding to the operation request instruction is smaller than the repaired position pointer, perform data operations on all copies in the operation field and record the corresponding operation log".

4. The method according to claim 1, characterized in that: The method further comprises: In the case that the operation request instruction is not received, the fields are restored in the direction of increasing keys based on the preset keys corresponding to the fields.

5. The method according to any one of claims 1 to 4, characterized in that: After the steps of performing data operations on the primary copy in the operation field and recording the operation log, and recording the operation log in the copy to be restored in the operation field, the method further includes: When the repaired position pointer is greater than the key corresponding to the operation request instruction, a data operation is performed on the copy to be restored in the operation field.

6. A data recovery device, characterized in that: The device comprises: A first acquisition module, configured to acquire a repaired position pointer of the object in response to a progress query instruction; A second acquisition module, configured to acquire, in response to the operation request instruction, a key corresponding to the operation request instruction; A first operation module, configured to perform data operations on all copies in an operation field and record corresponding operation logs when the key corresponding to the operation request instruction is smaller than the repaired position pointer; wherein the operation field is the field pointed to by the key; The second operation module is used to perform data operations on the primary copy in the operation field and record the operation log when the key corresponding to the operation request instruction is greater than or equal to the repaired position pointer, and to record the operation log in the copy to be restored in the operation field.

7. The device according to claim 6, characterized in that The device also includes: A judging module, used for judging whether the object is in a recovery state; The third operation module is used to perform data operations on the primary copy in the operation field and record the operation log when the object is not in a recovery state, and to record the operation log in the copy to be recovered in the operation field.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images