Data recovery method and device, electronic equipment and storage medium

Abstract

The invention provides a data recovery method and device, electronic equipment and a storage medium. During a Peering period, a main OSD is determined from OSDs associated with a PG, an Acting Set and an Up Set of the PG are determined, a target OSD is determined according to the Acting Set and the Up Set of the PG when the PG is in an Active period, and when the PG is in a Backfilling period, according to metadata of each object stored in the target OSD and metadata of each object stored in the main OSD, a data recovery result is obtained according to the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD. And determining a to-be-recovered object and performing data recovery. Since the data recovery is performed in the state that the PG can perform the read-write service, the PGLOG does not need to be acquired from each OSD associated with the PG during the Peering process, and the Peering time can be reduced, so that the service interruption time is reduced, and the service continuity and availability of the distributed storage system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data recovery method, device, electronic device and storage medium. Background Art

[0002] In a distributed storage system, when a user stores a file through a client, the file to be stored will be cut into multiple objects according to the preset size, and each object will be mapped to a placement group PG (PlacementGroup). PG is a logical collection of objects. Objects in PG will be replicated according to the preset number of copies. Multiple copies of the object are stored in different object storage devices OSD (Object Storage Device), which include master OSD and slave OSD.

[0003] When the state of the OSD associated with the PG changes, such as when the OSD fails or recovers from a failure, the data needs to be restored and balanced according to its redundancy strategy. Before data recovery and balancing, the missing objects on the OSD need to be determined through the state synchronization Peering process. However, read and write services are not allowed to be issued during the Peering process. Therefore, the speed of determining the missing objects of the OSD during the Peering process directly affects the service interruption time of the distributed storage system.

[0004] In the related art, during the Peering process, the placement group log PG_LOG (Placement Group Log) is obtained from each OSD associated with the PG, and the missing objects on the OSD are determined based on the PG_LOG. However, since the PG_LOG in the distributed storage system is large, it usually takes a long time to obtain the PG_LOG, which results in a long Peering process time, which in turn results in a long service interruption time of the distributed storage system, and low service continuity and availability. Summary of the invention

[0005] In view of this, the present application provides a data recovery method, device, electronic device and storage medium to address the deficiencies in the related art.

[0006] In a first aspect of the present application, a data recovery method is provided, which is applied to a distributed storage system including a placement group PG, comprising:

[0007] When the state of the OSD associated with any PG changes, triggering the PG to execute the state synchronization Peering process, during the Peering process, the primary OSD is determined from the object storage device OSD associated with the PG, and the Acting Set and Up Set of the PG are determined;

[0008] While the PG is in the active state Active, determining a target OSD for data recovery according to the Acting Set and Up Set of the PG;

[0009] While the PG is in the background filling state, the objects to be recovered in the target OSD are determined according to the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD, and the objects to be recovered in the target OSD are recovered using the main OSD.

[0010] According to one embodiment of the present application, the metadata includes an identifier and a version number of the target object, and the version number of the target object is the version number of the object storage device mapping OSDMap in the distributed storage system when the OSD associated with the PG receives a write request for the target object and stores the target object in the OSD.

[0011] According to an embodiment of the present application, determining the object to be restored in the target OSD according to the metadata of each object stored in the target OSD and the metadata of each object stored in the primary OSD includes:

[0012] Determine missing objects in the target OSD according to identifiers of each object stored in the target OSD and identifiers of each object stored in the primary OSD;

[0013] For objects having the same identifier in the target OSD and the primary OSD, determining the unupdated objects in the target OSD according to the version numbers of the objects;

[0014] According to the missing objects and the unupdated objects, objects to be restored in the target OSD are determined.

[0015] According to one embodiment of the present application, the method further includes:

[0016] When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if a read request for an object in the PG is received, the data of the object is read from the primary OSD associated with the PG;

[0017] If a write request for an object within the PG is received, the data and metadata of the object are stored in the primary OSD associated with the PG, and the data and metadata of the object are copied from the primary OSD to the corresponding slave OSD for storage.

[0018] According to an embodiment of the present application, the OSD state associated with any PG changes, triggering the PG to execute a state synchronization Peering process, including:

[0019] In the case that an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the OSD recovers from the failure within a preset time, the PG is triggered to execute the Peering process;

[0020] or,

[0021] When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the failure time of the OSD is greater than the preset time, the PG is triggered to execute the Peering process.

[0022] According to an embodiment of the present application, determining the primary OSD from the OSDs associated with the PG includes:

[0023] Obtain placement group information PG_INFO from the OSDs associated with the PG that have not failed respectively;

[0024] The primary OSD is determined based on the comparison results of the specified fields in each PG_INFO.

[0025] According to an embodiment of the present application, determining a target OSD for data recovery according to the Acting Set and Up Set of the PG includes:

[0026] For each OSD in the Up Set of the PG, determine whether the OSD exists in the Acting Set of the PG;

[0027] If not, the OSD is determined to be the target OSD.

[0028] In a second aspect of the present application, a data recovery device is provided, which is applied to a distributed storage system including a placement group PG, and the device includes:

[0029] A first determining unit is configured to, when the state of an OSD associated with any PG changes, triggering the PG to execute a state synchronization Peering process, determine the primary OSD from the object storage device OSD associated with the PG during the Peering process, and determine the Acting Set and Up Set of the PG;

[0030] A second determining unit is used to determine a target OSD for data recovery according to the ActingSet and Up Set of the PG while the PG is in an active state Active;

[0031] A data recovery unit is used to determine the objects to be recovered in the target OSD based on the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD when the PG is in the background filling state, and use the main OSD to recover the objects to be recovered in the target OSD.

[0032] In a third aspect of the present application, an electronic device is provided, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the steps of the method proposed in the above embodiment.

[0033] In a fourth aspect of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the steps of the method proposed in the above embodiment are implemented.

[0034] In a fifth aspect of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method proposed in the above embodiment when executed by a processor.

[0035] It can be seen from the above technical solution that when the state of the OSD associated with any PG changes, triggering the PG to execute the state synchronization Peering process, during the Peering process, the main OSD is determined from the object storage device OSD associated with the PG, and the active set Acting Set and the upset Up Set of the PG are determined. When the PG is in the active state Active, the target OSD for data recovery is determined according to the Acting Set and Up Set of the PG. When the PG is in the background filling state Backfilling, the objects to be recovered in the target OSD are determined according to the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD, and the main OSD is used to recover the objects to be recovered in the target OSD. During the Active and Backfilling periods when read and write services can be performed, the objects to be recovered of the OSD are determined according to the metadata of each object stored in the OSD and data is recovered. Since the primary OSD, as well as the Acting Set and Up Set can be determined during the Peering process without obtaining PG_LOG from each OSD associated with the PG, the Peering time can be reduced, thereby reducing the business interruption time of the distributed storage system and improving the business continuity and availability of the distributed storage system.

[0036] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of an addressing process of a distributed storage system provided in an embodiment of the present application;

[0038] Figure 2 It is a flowchart of a data recovery method provided by an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of an OSD state provided in an embodiment of the present application;

[0040] Figure 4 It is a structural schematic diagram of a data recovery device provided in an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0044] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0045] First, some technical abbreviations and key terms involved in this application are explained:

[0046] PG: Placement Group. PG is a logical structure used to manage data distribution and fault recovery in a distributed storage system. PG is a logical grouping, and each PG can manage a group of objects, which are mapped to a group of OSDs.

[0047] OSD: Object Storage Device. OSD is a data storage unit in a distributed storage system. It stores data in the form of objects and is responsible for maintaining the consistency and availability of these data. Each OSD has a unique identifier and independent data storage space. It collaborates with other OSDs to achieve data redundancy, fault recovery, and load balancing, thereby ensuring the reliability and performance of the distributed storage system.

[0048] PG_INFO: Placement Group Information, PG_INFO is a collection of detailed information used to record and report PG in a distributed storage system. It includes PG status information, history, statistics, object distribution, health status, and other related information.

[0049] PG_LOG: Placement Group Log, PG_LOG is a log used to record all PG operations, including object creation, update, deletion and query, which can be used for data recovery.

[0050] OSDMap: Object Storage Device Map. OSDMap contains information about all OSDs in the cluster, including OSD status, OSD composition in the cluster, OSD weight changes, and other OSD-related metadata. When the status of OSDs in the cluster changes, such as addition, deletion, failure, and failure recovery of OSDs, these changes will be monitored and reflected in OSDMap.

[0051] Peering: state synchronization, Peering refers to the process of establishing communication between all OSDs in a PG and reaching consensus on the state of objects and their metadata in the PG to ensure the state consistency between the PG's master OSD and all slave OSDs. Usually, when the state of the OSD associated with the PG changes, the PG can be triggered to execute the Peering process. When the Peering process is completed, the PG enters the Active state.

[0052] Active: Active state. Active means that all OSDs associated with PG are in normal working state. At this time, PG can normally process read and write requests from clients.

[0053] Backfilling: Background filling state. Backfilling is a data recovery process that can migrate data from one OSD to another to restore data integrity and redundancy. For example, when an OSD failure is recovered or a new OSD joins the cluster, the missing objects are copied from other healthy OSDs to these OSDs during Backfilling.

[0054] Degraded: Degraded state. When a PG's OSD fails, such as OSD downtime, network unreachability, or OSD process crash, the PG copy stored on the OSD will not be accessible or synchronized. If the number of surviving copies of PG is less than the minimum number of copies configured in the storage pool, PG will enter a degraded state. In the degraded state, PG can still provide services, but its data redundancy is reduced and the system's fault tolerance is affected.

[0055] like Figure 1 As shown, Figure 1 This is a schematic diagram of the addressing process of a distributed storage system provided by an embodiment of the present application. When a user stores a file through a client, the distributed storage system can cut the file into multiple objects according to a preset size, and each object can be mapped to a certain PG through a hash algorithm. Objects in a PG will be replicated according to the preset number of copies, and then stored in different OSDs, including primary OSDs and replica OSDs.

[0056] In the related technology, when the state of the OSD associated with the PG changes, such as when the OSD fails or recovers from a failure, before data recovery, the Peering process is used to determine the missing objects on the OSD. The specific steps are: during the Peering process, obtain the PG_LOG from each OSD associated with the PG, and determine the missing objects on each OSD based on the PG_LOG.

[0057] However, since the PG_LOG in the distributed storage system is large, it usually takes a long time to obtain the PG_LOG, so the Peering process in the related art takes a long time. Since read and write services are not allowed to be issued during the Peering process, the service interruption time of the distributed storage system in the related art is long, and the service continuity and availability are low.

[0058] In view of this, an embodiment of the present application discloses a data recovery method to address the deficiencies in the related art.

[0059] like Figure 2 As shown, Figure 2 1 is a flow chart of a data recovery method provided in an embodiment of the present application, and the data recovery method can be applied to a distributed storage system including a placement group PG. Exemplarily, the above-mentioned distributed storage system can include Ceph.

[0060] The data recovery method may include the following steps:

[0061] S201: When the state of an OSD associated with any PG changes, triggering the PG to execute a state synchronization Peering process, during the Peering process, the primary OSD is determined from the object storage device OSD associated with the PG, and the Acting Set and Up Set of the PG are determined.

[0062] In some embodiments, the distributed storage system can manage and store data in the form of objects, and each object will be assigned to a corresponding PG. For each object in the PG, the object can be replicated according to a preset number of copies, and the multiple copies of the object will be stored in different OSDs, which include a master OSD and multiple slave OSDs.

[0063] The number of replicas determines how many replicas of each object need to be stored in the distributed storage system. The number of replicas can be preset according to actual needs. For example, the number of replicas can be preset to 3, 4, etc.

[0064] Assuming the preset number of replicas is 3, for each object in PG, the distributed storage system will create 2 additional replicas, so that there are a total of 3 identical copies.

[0065] In a distributed storage system, each PG can be mapped to at least two OSDs, one of which is the master OSD and the other OSDs are slave OSDs. The master OSD is responsible for receiving and processing read and write requests. When an object is written to the master OSD, the master OSD can synchronize the object to the slave OSD according to the preset number of copies to ensure data availability and redundancy. The slave OSD is responsible for storing copies of the object and participating in data recovery and reconstruction when necessary.

[0066] In some embodiments, the failed OSD may include one or more OSDs, and the failed OSD may include a PG-associated master OSD and / or a slave OSD.

[0067] In some embodiments, the change in the status of the OSD associated with the PG may include a failure of the OSD associated with the PG, deletion of the OSD associated with the PG, or association of the PG to a newly added OSD.

[0068] In some embodiments, when an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the OSD recovers from the failure within a preset time, the PG is triggered to execute the Peering process.

[0069] The OSD failure may include a physical failure at the hardware level of the OSD, such as hard disk damage, memory failure, etc., or an OSD offline due to non-hardware reasons, such as network shock, system power on and off, etc.

[0070] After any OSD associated with a PG fails, the replicas stored on the failed OSD will not be accessible or synchronized, causing the number of surviving replicas of the PG to be less than the configured minimum number of replicas, thereby triggering the PG to enter a degraded state. While the PG is in a degraded state, if it is detected that the failed OSD recovers within the preset time, the PG will be triggered to execute the Peering process.

[0071] The preset time can be set according to actual needs, and the embodiments of the present application do not specifically limit this. For example, the preset time can be 15 minutes, 30 minutes, etc.

[0072] In some embodiments, the distributed storage system can detect the OSD status at specified intervals. If the OSD failure recovery is detected at the first time, and it is determined that the difference between the first time and the time when the OSD failure occurs is less than or equal to the above-mentioned preset time, the PG can be triggered to execute the Peering process at the first time.

[0073] For example, assuming that the preset time is 30 minutes, the distributed storage system detects the OSD status every 2 minutes, the OSD fails at 15:10, and the OSD recovers at 15:21. The distributed storage system can detect that the OSD has recovered at 15:22 (the first time). Since the difference between the first time and the time when the OSD fails is 12 minutes, which is less than the preset time, the PG can be triggered to execute the Peering process at 15:22 (the first time).

[0074] In some embodiments, when an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the failure time of the OSD is greater than the preset time, the PG is triggered to execute the Peering process.

[0075] After any OSD associated with a PG fails, the replicas stored on the failed OSD will not be accessible or synchronized, causing the number of surviving replicas of the PG to be less than the configured minimum number of replicas, thereby triggering the PG to enter a degraded state. While the PG is in a degraded state, if it is detected that the failure time of the failed OSD is greater than the above-mentioned preset time, the PG is triggered to execute the Peering process.

[0076] In some embodiments, the distributed storage system can detect the OSD status at specified intervals. If it is detected that the OSD is still faulty at the second time, and it is determined that the difference between the second time and the time when the OSD fails is greater than the preset time, the PG can be triggered to execute the Peering process at the second time.

[0077] For example, assuming that the preset time is 30 minutes, the distributed storage system detects the OSD status every 2 minutes, the OSD fails at 15:10, and the distributed storage system detects that the OSD is still faulty at 15:42 (the second time). Since the difference between the second time and the time when the OSD fails is 32 minutes, which is greater than the preset time, the PG can be triggered to execute the Peering process at 15:42 (the second time).

[0078] In some embodiments, when an OSD associated with any PG fails, triggering the PG to enter a degraded state, if a read request for an object in the PG is received, the data of the object is read from the primary OSD associated with the PG;

[0079] If a write request for an object within the PG is received, the data and metadata of the object are stored in the primary OSD associated with the PG, and the data and metadata of the object are copied from the primary OSD to the corresponding slave OSD for storage.

[0080] In an embodiment of the present application, when an OSD associated with any PG fails, triggering the PG to enter a degraded state, data is read from the master OSD associated with the PG, and when written, data is written to the master OSD and the corresponding slave OSD without reconstructing the data. If the failure time of the OSD exceeds the preset time, the PG is triggered to execute the Peering process, and data is reconstructed after the Peering process is completed. This can avoid invalid data reconstruction in the event of network oscillation or power on and off of the distributed storage system, thereby improving the performance and stability of the distributed storage system.

[0081] In some embodiments, PG_INFO can be obtained from the OSDs associated with the PG respectively, and the main OSD can be determined based on the comparison results of specified fields in each PG_INFO.

[0082] PG_INFO is a collection of detailed information about PGs in a distributed storage system. It includes PG status information, history records, statistics, and other related information. Each OSD can store the PG_INFO of the PG associated with the OSD.

[0083] Exemplarily, the above-mentioned designated fields may include last_update, last_complete, etc., wherein the last_update field may indicate the timestamp of the last PG status update, and the last_complete field may indicate the timestamp of the last PG completed operation.

[0084] In some embodiments, the above-mentioned designated fields may include a designated field, which can effectively reflect the activity level and operation completion status of the PG, and the main OSD can be determined based on the comparison results of the designated fields in each PG_INFO.

[0085] For example, by comparing the sizes of the designated field values ​​in each PG_INFO, it is determined that the OSD corresponding to the PG_INFO with the largest designated field value is the primary OSD.

[0086] In some embodiments, the above-mentioned designated field may include multiple designated fields, and a weight value may be pre-assigned to each designated field. The comprehensive evaluation of each OSD may be determined based on the comparison results of multiple designated fields in each PG_INFO and the weight value of each designated field, and the master OSD may be determined based on the comprehensive evaluation.

[0087] In some embodiments, during the Peering process, the Acting Set and Up Set of the PG can be determined based on a preset algorithm for data distribution and data location calculation in the distributed storage system (such as the CRUSH algorithm in the Ceph distributed storage system) and the OSDMap in the distributed storage system.

[0088] Each PG corresponds to an Acting Set and an Up Set, where the Acting Set is the OSD set that currently participates in processing the read / write requests of the PG and storing the copies of the PG objects.

[0089] Up Set is an OSD set calculated based on a preset algorithm for data distribution and data location calculation in a distributed storage system (such as the CRUSH algorithm in a Ceph distributed storage system).

[0090] In an embodiment of the present application, when the state of an OSD associated with any PG changes, triggering the PG to execute a Peering process, during the Peering process, after determining the master OSD from the OSD associated with the PG, and determining the Acting Set and Up Set of the PG, the PG can enter the Active state to provide services externally. Since it is not necessary to obtain PG_LOG from each OSD associated with the PG during the Peering process, the Peering time can be reduced, thereby improving the business continuity of the distributed storage system.

[0091] S202: While the PG is in the active state, determine a target OSD for data recovery according to the Acting Set and Up Set of the PG.

[0092] In an embodiment of the present application, after the Peering process in S201 is completed, the PG enters the Active state, at which time the PG can process read and write requests normally. While the PG is in the Active state, the target OSD for data recovery can be determined based on the Acting Set and Up Set of the PG.

[0093] In some embodiments, for each OSD in the Up Set of the PG, it is determined whether the OSD exists in the Acting Set of the PG. If not, the OSD is determined to be the target OSD for data recovery.

[0094] Each PG corresponds to an Acting Set and an Up Set. Under normal circumstances, the UP Set and Acting Set of a PG will be consistent to ensure data reliability and availability. For example, under normal circumstances, the Acting Set and Up Set of a PG are (OSD1, OSD2, OSD3).

[0095] When the status of the OSD associated with the PG changes, the Acting Set and Up Set of the PG will also change. Among them, when the OSD associated with the PG fails, the failed OSD will not be removed from the UP Set of the PG immediately, but it will be removed from the Acting Set of the PG immediately. If the failed OSD recovers within the preset time, then during the Peering process, the Acting Set of the determined PG does not contain the recovered OSD, but the Up Set contains the recovered OSD. When the PG is Active, according to the Acting Set and Up Set of the PG, it can be determined that the target OSD for data recovery is the recovered OSD.

[0096] If the failure time of the failed OSD is greater than the above-mentioned preset time, the distributed storage system will calculate a new Up Set according to the preset algorithm for data distribution and data location calculation. Then during the Peering process, the newly added OSD in the Up Set does not exist in the Acting Set of the determined PG. While the PG is Active, based on the ActingSet and Up Set of the PG, it can be determined that the target OSD for data recovery is the OSD newly added in the Up Set.

[0097] For example, under normal circumstances, the Acting Set and UP Set of a PG are both (OSD1, OSD2, OSD3). When OSD2 associated with the PG fails, the Acting Set of the PG is (OSD1, OSD3) and the UP Set is (OSD1, OSD2 (failure), OSD3). After OSD2 fails, it will wait for OSD2 to recover within the preset time.

[0098] If OSD2 recovers from the fault within the preset time, then during the Peering process, the ActingSet of the PG determined is (OSD1, OSD3), and the UP Set is (OSD1, OSD2, OSD3). While the PG is Active, for each OSD in the Up Set, it is determined whether the OSD exists in the Acting Set, thereby determining that the target OSD for data recovery is OSD2.

[0099] If the failure time of OSD2 exceeds the preset time, the distributed storage system can mark OSD2 as no longer running, and calculate a new Up Set based on the preset algorithm for data distribution and data location calculation (such as the CRUSH algorithm in the Ceph distributed storage system). Then, during the Peering process, the Acting Set of the determined PG is (OSD1, OSD3), and the UP Set is (OSD1, OSD4 (newly added OSD), OSD3). While the PG is Active, for each OSD in the Up Set, it is determined whether the OSD exists in the Acting Set, thereby determining that the target OSD for data recovery is OSD4.

[0100] S203: While the PG is in the backfilling state, determine the objects to be recovered in the target OSD based on the metadata of each object stored in the target OSD and the metadata of each object stored in the primary OSD, and use the primary OSD to recover the objects to be recovered in the target OSD.

[0101] In an embodiment of the present application, after determining the target OSD for data recovery while the PG is in the Active state, the PG can be triggered to enter Backfilling, and data recovery can be performed on the target OSD while the PG is in the Backfilling state. During the Backfilling period, the objects to be recovered in the target OSD can be determined based on the metadata of each object stored in the target OSD and the metadata of each object stored in the master OSD, and then the master OSD can be used to recover the objects to be recovered in the target OSD.

[0102] The distributed storage system can manage and store data in the form of objects, each object will be assigned to a corresponding PG, and then the object will be stored in the OSD associated with the PG. In an embodiment of the present application, the object stored in the OSD includes the object's data and the object's metadata.

[0103] In some embodiments, the metadata of an object includes an identifier of the object and a version number of the object.

[0104] The object identifier can be a string or number that uniquely identifies each object in the distributed storage system. The object identifier is used to distinguish different objects and ensure that each object has a unique identity in the distributed storage system.

[0105] The version number of the object is the version number of the current OSDMap in the distributed storage system when the OSD associated with the PG receives a write request for the object and stores the object in the OSD.

[0106] The version number of OSDMap in the distributed storage system is automatically generated by the distributed storage system. Whenever the status of OSD in the cluster changes (such as addition, deletion, failure, failure recovery and configuration change of OSD), the distributed storage system will generate a new OSDMap to reflect these changes. Each OSDmap has a corresponding OSDmap version number. The version number of OSDmap will increase monotonically in sequence. The size of OSDmap version number directly reflects the newness of OSDMap. The larger the version number, the newer the information contained in OSDMap, which reflects the latest status of the cluster.

[0107] In some embodiments, after the primary OSD associated with a PG receives a write request for an object within the PG, the data of the object can be stored in the primary OSD, and the version number of the current OSDMap in the distributed storage system can be obtained at the same time. If it is determined that the object is written for the first time, the metadata of the object and the object data are stored together in the primary OSD, and the metadata of the object includes the identifier of the object and the version number of the object (that is, the version number of the current OSDMap); if it is determined that the object is not written for the first time, it is determined whether the version number in the metadata of the object is consistent with the version number of the current OSDMap. If they are consistent, there is no need to update the version number in the metadata of the object. If they are inconsistent, the version number in the metadata of the object is updated to the version number of the above-mentioned current OSDMap.

[0108] After the object is written to the primary OSD, the primary OSD may synchronize the object (including the object's data and the object's metadata) to the corresponding secondary OSDs according to a preset number of replicas.

[0109] In an embodiment of the present application, there is no need to record PG_LOG for the write operation of each object. Instead, the version number of the object is recorded in the metadata of the object, that is, the version number of the OSDMap in the distributed storage system is recorded in the metadata of the object. Since the version number of the object is only updated when the object is written for the first time, or when the object is not written for the first time but the version number of the current OSDMap in the distributed storage system is inconsistent with the version number of the object, and the metadata of the object does not need to be updated during the append write process, the number of metadata updates can be reduced, thereby improving the performance of the distributed storage system.

[0110] In some embodiments, the missing objects in the target OSD may be determined based on identifiers of the respective objects stored in the target OSD and identifiers of the respective objects stored in the primary OSD;

[0111] For objects having the same identifier in the target OSD and the primary OSD, determining the unupdated objects in the target OSD according to the version numbers of the objects;

[0112] According to the missing objects and the unupdated objects, objects to be restored in the target OSD are determined.

[0113] In an embodiment of the present application, when the state of an OSD associated with any PG changes, triggering the PG to execute a state synchronization Peering process, during the Peering process, the master OSD is determined from the object storage device OSD associated with the PG, and the active set Acting Set and the upset Up Set of the PG are determined; while the PG is in the active state Active, the target OSD for data recovery is determined based on the Acting Set and Up Set of the PG; while the PG is in the background filling state Backfilling, the objects to be recovered in the target OSD are determined based on the metadata of each object stored in the target OSD and the metadata of each object stored in the master OSD, and the objects to be recovered in the target OSD are recovered using the master OSD. During the Active and Backfilling periods when read and write services can be performed, the objects to be recovered of the OSD are determined according to the metadata of each object stored in the OSD and data is recovered. Since the primary OSD, as well as the Acting Set and Up Set can be determined during the Peering process without obtaining PG_LOG from each OSD associated with the PG, the Peering time can be reduced, thereby reducing the business interruption time of the distributed storage system and improving the business continuity and availability of the distributed storage system.

[0114] In some embodiments, when no OSD in the distributed storage system fails, that is, all OSDs in the distributed storage system are in a state of Figure 3 In the IN state shown, the distributed storage system can perform normal IO processes.

[0115] During normal IO, after receiving a write request for target data, the distributed storage system can divide the target data into multiple objects according to the preset size, and each object will be mapped to a corresponding PG. For each object in the PG, the object can be replicated according to the preset number of copies, and the multiple copies of the object will be stored in a master OSD and multiple slave OSDs respectively. The copies stored in the OSD include the object's data and the object's metadata, where the object's metadata includes the object's identifier and the object's version number, and the object's version number refers to the version number of the current OSDMap in the distributed storage system when the object's data is stored in the master OSD.

[0116] In the case of a failure of an OSD associated with a PG in a distributed storage system, that is, the OSD is Figure 3 The IN state shown changes to the DOWN state, and the distributed storage system will automatically generate a new OSDMap to reflect the above changes. The OSDmap has a corresponding OSDmap version number, which is monotonically increasing in sequence based on the version number of the previous OSDmap. At the same time, the OSD will be immediately removed from the Acting Set of the PG, but will not be immediately removed from the UP Set of the PG. Due to the OSD failure, the PG copies stored on the OSD will not be accessible or synchronized. The number of surviving copies of the PG is less than the minimum number of copies configured in the storage pool, and the PG enters a degraded state.

[0117] In the degraded state, PG can still perform read and write services, but its data redundancy is reduced and the fault tolerance of the system is affected. In the degraded state, the IO process follows the master OSD distribution process, that is, for read requests, the object data is read from the master OSD, and for write requests, after the master OSD receives the write request for the object, it stores the object data in the master OSD and obtains the version number of the current OSDMap in the distributed storage system. If it is determined that the object is written for the first time, the metadata of the object and the object data are stored together in the master OSD. The metadata of the object includes the identifier of the object and the version number of the object (that is, the version number of the current OSDMap); if it is determined that the object is not written for the first time, it is determined whether the version number in the metadata of the object is consistent with the version number of the current OSDMap. If they are consistent, there is no need to update the version number in the metadata of the object. If they are inconsistent, the version number in the metadata of the object is updated to the version number of the current OSDMap. After writing the object to the master OSD, the master OSD can synchronize the object (including the object data and the object metadata) to the corresponding slave OSD according to the preset number of copies.

[0118] In an embodiment of the present application, when the distributed storage system performs normal IO processes and when the PG enters a degraded state, it is not necessary to record PG_LOG for the write operation of each object. Instead, the version number of the object is recorded in the metadata of the object, that is, the version number of the OSDMap in the distributed storage system is recorded in the metadata of the object. Since the version number of the object is only updated when the object is written for the first time, or when the object is not written for the first time but the version number of the current OSDMap in the distributed storage system is inconsistent with the version number of the object, and the metadata of the object does not need to be updated during the append write process, the number of metadata updates can be reduced, thereby improving the performance of the distributed storage system.

[0119] In the degraded state, if it is detected that the OSD recovers from the fault within the preset time, that is, the OSD Figure 3 After the DOWN state shown in the figure is restored to the IN state, the PG can be triggered to execute the Peering process when the OSD fault is restored; or, if it is detected that the OSD fault time exceeds the preset time, the OSD can be restored from the Figure 3 The DOWN state shown is marked as OUT state, triggering PG to execute the Peering process.

[0120] In an embodiment of the present application, when an OSD associated with any PG fails, triggering the PG to enter a degraded state, data is read from the master OSD associated with the PG, and when written, data is written to the master OSD and the corresponding slave OSD without reconstructing the data. If the failure time of the OSD exceeds the preset time, the PG is triggered to execute the Peering process, and data is reconstructed after the Peering process is completed. This can avoid invalid data reconstruction in the event of network oscillation or power on and off of the distributed storage system, thereby improving the performance and stability of the distributed storage system.

[0121] During the Peering process, PG_INFO is obtained from the OSDs associated with the PG that have not failed. The primary OSD is determined based on the comparison results of the specified fields in each PG_INFO. The Acting Set and Up Set of the PG are determined based on the preset algorithm for data distribution and data location calculation in the distributed storage system (such as the CRUSH algorithm in the Ceph distributed storage system) and the current OSDMap, and the Peering process of the PG is completed.

[0122] After the Peering process is completed, PG enters the Active state to provide external services. While PG is in the Active state, the distributed storage system can perform read and write services.

[0123] While the PG is in the Active state, for each OSD in the PG's Up Set, determine whether the OSD exists in the PG's Acting Set. If not, determine that the OSD is the target OSD that needs to perform data recovery.

[0124] If there is a target OSD, the PG is triggered to enter backfilling. During the backfilling period, the missing objects in the target OSD can be determined based on the identifiers of each object stored in the target OSD and the identifiers of each object stored in the primary OSD associated with the PG. For objects with the same identifier in the target OSD and the primary OSD, the unupdated objects in the target OSD can be determined based on the version number of the object. Then, based on the missing objects and unupdated objects, the objects to be restored in the target OSD can be determined. After determining the objects to be restored in the target OSD, copy the objects to be restored from the primary OSD of the PG to the target OSD for storage.

[0125] In an embodiment of the present application, during the Active and Backfilling periods when read and write services can be performed after the Peering process, the objects to be recovered of the OSD are determined according to the metadata of each object stored in the OSD, and data recovery is performed. Since only the primary OSD, as well as the Acting Set and Up Set need to be determined during the Peering process, and there is no need to obtain PG_LOG from each OSD associated with the PG, the Peering time can be reduced, thereby reducing the business interruption time of the distributed storage system, and further improving the business continuity and availability of the distributed storage system.

[0126] The above content describes the method provided by the present application. The following describes the device provided by the present application:

[0127] See also Figure 4 , is a structural diagram of a data recovery device provided in an embodiment of the present application.

[0128] like Figure 4 As shown, the device may include:

[0129] A first determining unit 410 is configured to, when the state of an OSD associated with any PG changes, triggering the PG to execute a state synchronization Peering process, determine a primary OSD from the object storage device OSD associated with the PG during the Peering process, and determine an Acting Set and an Up Set of the PG;

[0130] A second determining unit 420 is configured to determine a target OSD for data recovery according to an ActingSet and an Up Set of the PG while the PG is in an active state Active;

[0131] The data recovery unit 430 is used to determine the objects to be recovered in the target OSD based on the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD during the period when the PG is in the background filling state Backfilling, and use the main OSD to recover the objects to be recovered in the target OSD.

[0132] In some embodiments, the data recovery unit 430 is specifically used to:

[0133] The metadata includes an identifier and a version number of the target object. The version number of the target object is the version number of the object storage device mapping OSDMap in the distributed storage system when the OSD associated with the PG receives a write request for the target object and stores the target object in the OSD.

[0134] In some embodiments, the data recovery unit 430 is specifically used to:

[0135] Determine missing objects in the target OSD according to identifiers of each object stored in the target OSD and identifiers of each object stored in the primary OSD;

[0136] For objects having the same identifier in the target OSD and the primary OSD, determining the unupdated objects in the target OSD according to the version numbers of the objects;

[0137] According to the missing objects and the unupdated objects, objects to be restored in the target OSD are determined.

[0138] In some embodiments, the device further comprises a reading and writing unit, specifically configured to:

[0139] When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if a read request for an object in the PG is received, the data of the object is read from the primary OSD associated with the PG;

[0140] If a write request for an object within the PG is received, the data and metadata of the object are stored in the primary OSD associated with the PG, and the data and metadata of the object are copied from the primary OSD to the corresponding slave OSD for storage.

[0141] In some embodiments, the first determining unit 410 is specifically configured to:

[0142] In the case that an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the OSD recovers from the failure within a preset time, the PG is triggered to execute the Peering process;

[0143] or,

[0144] When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the failure time of the OSD is greater than the preset time, the PG is triggered to execute the Peering process.

[0145] In some embodiments, the first determining unit 410 is specifically configured to:

[0146] Obtain placement group information PG_INFO from the OSDs associated with the PG that have not failed respectively;

[0147] The primary OSD is determined based on the comparison results of the specified fields in each PG_INFO.

[0148] In some embodiments, the second determining unit 420 is specifically configured to:

[0149] For each OSD in the Up Set of the PG, determine whether the OSD exists in the Acting Set of the PG;

[0150] If not, the OSD is determined to be the target OSD.

[0151] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0152] The present application embodiment also provides a hardware structure. Figure 5 , Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0153] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0154] Exemplarily, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0155] It should be noted that, in this article, relational terms such as target and target are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0156] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data recovery method, characterized in that: Applied to a distributed storage system including a placement group PG, the method includes: When the state of the OSD associated with any PG changes, triggering the PG to execute the state synchronization Peering process, during the Peering process, the primary OSD is determined from the object storage device OSD associated with the PG, and the Acting Set and Up Set of the PG are determined; While the PG is in the active state Active, determining a target OSD for data recovery according to the Acting Set and Up Set of the PG; While the PG is in the background filling state, the objects to be recovered in the target OSD are determined based on the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD, and the objects to be recovered in the target OSD are recovered using the main OSD.

2. The method according to claim 1, characterized in that The metadata includes an identifier and a version number of the target object. The version number of the target object is the version number of the object storage device mapping OSDMap in the distributed storage system when the OSD associated with the PG stores the target object to the OSD after receiving a write request for the target object.

3. The method according to claim 2, characterized in that The determining the to-be-recovered object in the target OSD according to the metadata of each object stored in the target OSD and the metadata of each object stored in the primary OSD comprises: Determine the missing objects in the target OSD according to the identifiers of the respective objects stored in the target OSD and the identifiers of the respective objects stored in the primary OSD; For objects having the same identifier in the target OSD and the primary OSD, determining the unupdated objects in the target OSD according to the version numbers of the objects; According to the missing objects and the unupdated objects, objects to be restored in the target OSD are determined.

4. The method according to claim 1, characterized in that: The method further comprises: When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if a read request for an object in the PG is received, the data of the object is read from the primary OSD associated with the PG; If a write request for an object within the PG is received, the data and metadata of the object are stored in the primary OSD associated with the PG, and the data and metadata of the object are copied from the primary OSD to the corresponding slave OSD for storage.

5. The method according to claim 1, characterized in that The OSD state associated with any PG changes, triggering the PG to execute the state synchronization Peering process, including: In the case that an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the OSD recovers from the failure within a preset time, the PG is triggered to execute the Peering process; or, When an OSD associated with any PG fails, triggering the PG to enter a degraded state, if it is detected that the failure time of the OSD is greater than the preset time, the PG is triggered to execute the Peering process.

6. The method according to claim 1, characterized in that The determining the primary OSD from the OSDs associated with the PG includes: Obtain placement group information PG_INFO from the OSDs associated with the PG that have not failed respectively; The primary OSD is determined based on the comparison results of the specified fields in each PG_INFO.

7. The method according to claim 1, characterized in that Determining a target OSD for data recovery according to the Acting Set and Up Set of the PG includes: For each OSD in the Up Set of the PG, determine whether the OSD exists in the Acting Set of the PG; If not, the OSD is determined to be the target OSD.

8. A data recovery device, characterized in that: Applied to a distributed storage system including a placement group PG, the device comprises: A first determining unit is configured to, when the state of an OSD associated with any PG changes, triggering the PG to execute a state synchronization Peering process, determine the primary OSD from the object storage device OSD associated with the PG during the Peering process, and determine the Acting Set and Up Set of the PG; A second determining unit is used to determine a target OSD for data recovery according to an Acting Set and an Up Set of the PG while the PG is in an active state Active; A data recovery unit is used to determine the objects to be recovered in the target OSD based on the metadata of each object stored in the target OSD and the metadata of each object stored in the main OSD when the PG is in the background filling state, and use the main OSD to recover the objects to be recovered in the target OSD.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images