Method for recovering attribution group PG and storage device

Abstract

The invention provides a home group PG recovery method and storage device.The method comprises the steps that when a target PG is recovered, the first utilization rate of a target object storage device OSD scheduled by the target PG is obtained; the first utilization rate is compared with a first threshold value, and the recovery data flow direction of the target OSD is obtained; and determining whether to perform recovery operation on the target OSD or not according to the comparison result and the recovery data flow direction. Through the method, the problems of expansion process interruption failure and service interruption caused by the fact that the OSD is fully written in advance in the recovery process can be effectively prevented.

Description

Technical Field

[0001] This specification relates to the field of communication technology, and in particular to a method for recovering a home group (PG) and a storage device. Background Technology

[0002] Ceph: Ceph is a software-defined storage technology. It is an open-source project that provides a software-defined, unified storage solution for block storage, object storage, and file storage. It features a scalable, high-performance, and single-point-of-failure distributed storage system, providing massive, undifferentiated, and unified distributed storage services to external users.

[0003] OSD: Object Storage Device. In Ceph cluster applications, a hard drive in a Ceph cluster typically corresponds to one OSD daemon process.

[0004] PG: Placement Group. A PG is a logical unit in Ceph used to manage data distribution and replication. A PG contains multiple data objects and has a mapping relationship with OSDs based on storage redundancy policies.

[0005] Crush: CRUSH is an abbreviation for Controlled Replication Under Scalable Hashing. It is a data distribution algorithm, similar to consistent hashing, used to control data distribution in RADOS storage clusters. Crush consists of two steps: first, mapping objects to PGs, and second, mapping PGs to OSDs.

[0006] PG peering: PG peering is a mechanism in Ceph used to ensure that the metadata of each PG remains consistent across different OSDs (Object Storage Daesel). When changes occur in the Ceph cluster (OSDs are added, deleted, fail, etc.), a PG enters the peering state.

[0007] Ceph is a software-defined distributed storage system. It integrates the hardware storage resources of general-purpose servers to provide customers with unified, distributed storage services. When using a distributed storage system, as data is continuously written, the storage utilization of the entire system will increase. When the storage utilization becomes too high, the storage system is often expanded to provide space for subsequent data writes. For a Ceph cluster, this involves adding new cluster nodes or adding new OSDs to existing nodes.

[0008] For Ceph distributed storage systems, after the cluster expands with new nodes or existing nodes are expanded with new OSDs, the CRUSH algorithm will re-process the data. The distribution of PGs is calculated. When the OSD corresponding to a PG changes, the PG enters the peering state. The peering process generally includes confirming the master-slave OSD relationship and determining which object data needs to be restored and synchronized by comparing logs and metadata. After peering is completed, the PG will begin to repair inconsistent data objects on each replica until all replicas of all PGs reach consistency.

[0009] During the process of a PG modifying a data object, from the perspective of the OSD, the PG to which the OSD belongs changes. During the data recovery process, the OSD has data reads and writes, and after the data recovery is completed, there is also the situation of data space being released.

[0010] In general applications, expansion operations often occur when cluster utilization is high. However, in the Ceph distributed storage system, the data balancing order of PGs is not strictly controllable during the data recovery process after expansion. Often, when the utilization of existing OSDs in the cluster is high, the balancing process causes these OSDs to be filled first, resulting in expansion interruptions and service interruptions. Summary of the Invention

[0011] To overcome the problems existing in related technologies, this specification provides a method for recovering the home group (PG) and a storage device.

[0012] According to a first aspect of the embodiments of this specification, a method for restoring a Group of Attribution (PG) is provided, the method comprising: When the target PG is restored, the first utilization rate of the target object storage device OSD scheduled by the target PG is obtained; The first utilization rate is compared with the first threshold, and the recovery data flow of the target OSD is obtained; Determine whether to perform a recovery operation on the target OSD based on the comparison results and the recovery data flow.

[0013] The recovery data flow of the target OSD includes: a data flow written to the OSD or a data flow read from the OSD.

[0014] The step of determining whether to perform a recovery operation on the target OSD based on the comparison results and the recovery data flow includes: If the comparison result shows that the first utilization rate is greater than the first threshold, and the recovery data flow of the target OSD is the data flow written to the OSD, then the data recovery action of the target OSD is delayed; or, If the comparison result is that the first utilization rate is less than or equal to the first threshold, and the recovery data flow of the target OSD is the data flow written to the OSD, then the data recovery action of the target OSD is executed.

[0015] Optionally, determining whether to perform a recovery operation on the target OSD based on the comparison result and the recovery data flow includes: If the target OSD's recovery data flow is to read the OSD's data flow, then the target OSD's data recovery action is executed.

[0016] The data recovery action of the delayed target OSD includes: Add a Low-Priority status flag to the target OSD, and delay the data recovery action of the target OSD based on the Low-Priority status flag.

[0017] As can be seen from the above embodiments, the solution provided in this disclosure can perform reasonable sequential scheduling of PG recovery during Ceph storage cluster expansion, and effectively prevent the problem of OSDs being prematurely filled during the recovery process, leading to expansion process interruption and service interruption.

[0018] According to a second aspect of the embodiments of this specification, a storage device is provided, wherein a Ceph cluster is deployed in the storage device, the storage device comprising: The acquisition module is used to acquire the first utilization rate of the target object storage device OSD scheduled by the target PG when the target PG is restored; The comparison module is used to compare the first utilization rate with the first threshold and obtain the recovery data flow of the target OSD; The processing module is used to determine whether to perform a recovery operation on the target OSD based on the comparison results and the recovery data flow.

[0019] The recovery data flow of the target OSD includes: a data flow written to the OSD or a data flow read from the OSD.

[0020] Specifically, the processing module is configured to delay the data recovery action of the target OSD when the comparison result shows that the first utilization rate is greater than the first threshold, and the recovery data flow direction of the target OSD is the data flow written to the OSD; or, If the comparison result is that the first utilization rate is less than or equal to the first threshold, and the recovery data flow of the target OSD is the data flow written to the OSD, then the data recovery action of the target OSD is executed.

[0021] Specifically, the processing module is used to perform data recovery actions on the target OSD when the recovery data flow of the target OSD is a data flow that reads the OSD.

[0022] Specifically, the processing module is used to add a Low-Priority status identifier to the target OSD and delay the data recovery action of the target OSD according to the Low-Priority status identifier.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0025] Figure 1 This is a schematic diagram illustrating a method for restoring the group of ownership (PG) according to an exemplary embodiment of this specification. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0027] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] As shown in Table 1, after expanding OSD4, the PG mapping changes. The newly added OSD4 is written with data during the data recovery process. For the previous OSD1-OSD3 in the cluster, corresponding recovery data will also be written due to the PG change.

[0030] Table 1 As shown in Table 2, taking OSD1 as an example, OSD1 will release data space after PG1 and PG4 are restored, and OSD1 will write data space during PG6 restoration. However, the restoration order of PG1, PG4, and PG6 is not fixed. In practice, OSD1 is often filled first because data repair is performed first by PG6, causing the entire expansion process to fail.

[0031] Table 2 To address the aforementioned technical problems, embodiments of this disclosure provide a method for recovering the Group of Attribution (PG), such as... Figure 1 As shown, the method includes: S101 When restoring the target PG, obtain the first utilization rate of the target object storage device OSD scheduled by the target PG; S102 compares the first utilization rate with the first threshold and obtains the recovery data flow direction of the target OSD; S103 determines whether to perform a recovery operation on the target OSD based on the comparison results and the recovery data flow.

[0032] In this embodiment, after the Ceph cluster is expanded, Crush recalculates the mapping relationship between PGs and OSDs. Some old OSDs in the cluster are removed from the PG mapping (data migration out), while others are calculated into the mapping of other PGs (data migration in). Through the embodiments provided in this disclosure, the PG recovery scheduling order of the corresponding OSDs can be dynamically controlled and adjusted according to the flow of recovery data in different PGs (writing data or releasing space) and the utilization rate of the OSD itself, so as to avoid the problem of OSDs being filled with recovery data during the expansion process, which would cause the expansion to be interrupted.

[0033] In this embodiment, the Mon node maintains PGMAP and OSDMAP, allowing it to obtain the mapping distribution of PGs and basic OSD information in real time. In Ceph, when the cluster topology changes, the Mon updates PGMAP and OSDMAP and synchronizes the information to the OSDs. During PG recovery, the Primary OSD in the PG is responsible for initiating the recovery operation. It identifies which OSDs can participate in the recovery process and the data flow (i.e., which OSDs will write recovery data) based on the current PGMAP and OSDMAP.

[0034] In this embodiment, a first threshold for the recovery of a PG in the OSD can be set first. This first threshold can be the utilization rate, for example, 85%. After Peering is completed, the Primary OSD will start scheduling the recovery of the PG. Specifically, the scheduled PG is called the target PG. Before the target PG is scheduled to start data recovery, the first utilization rate of the target OSD scheduled by the target PG is obtained. In step S102, the Primary OSD queries the storage first utilization rate of the target OSD in the target PG group and compares it with the first threshold.

[0035] The comparison results may include cases where the first utilization rate of the target OSD is greater than a first threshold, or the first utilization rate is less than or equal to the first threshold. In this embodiment, the target OSD recovery data flow direction may also be obtained, for example, the OSD recovery data flow direction may include a write data flow or a read data flow.

[0036] Based on the above comparison results, when the OSD recovery data flow is a read data flow, the above comparison results can be disregarded, and the corresponding recovery operation can be performed directly.

[0037] In another embodiment, when the OSD recovery data flow is a write data flow, the comparison result between the first utilization rate and the first threshold is further considered. Specifically, when the comparison result is that the first utilization rate is greater than the first threshold, or when the comparison result is that the first utilization rate is less than or equal to the first threshold, the data recovery action of the target OSD is performed.

[0038] To illustrate the solutions in this disclosure in detail, the following examples are provided for this embodiment: As shown in Table 3, the first threshold is 85%, and it can be seen that the usage rate of OSD1 and OSD2 is greater than the first threshold.

[0039] Table 3 As shown in Table 4, it can be seen that the first utilization rate of each target OSD in PG1, PG4, PG5, PG7 and PG8 is less than the first threshold and the data flow is writing, so recovery operation can be performed. However, the first utilization rate of PG2 and PG6 is greater than the first threshold and the data flow is writing, so recovery operation is delayed.

[0040] Table 4 In this embodiment, the delayed operation can be achieved by setting the Low-Priority state for the OSD that needs to perform the recovery operation with a delay.

[0041] In this embodiment, after each PG is restored, the OSDs that no longer carry the data of this PG in the corresponding old OSD map relationship will release the data. Therefore, the utilization rate of the current OSD needs to be retrieved in real time again when each PG is restored, and the latest OSD utilization rate is compared with the first threshold.

[0042] As shown in Table 5, after PG1 is normally restored, OSD1 no longer carries the data of PG1 and will release space. After the space is released, it is possible that the utilization rate of OSD1 will be lower than A. Then PG6, which was previously set to the low-priority state, will be temporarily set to the state where it can be normally restored.

[0043] Table 5 Similarly, during the recovery process of the PG, when the PG is restored, the utilization rate of the OSDs that newly carry the data of this PG in the corresponding new OSD map relationship will increase, and it is possible that it will exceed the first threshold.

[0044] As shown in Table 6, if PG8 is not scheduled until after PG7 is restored, at this time the utilization rate of OSD3 has exceeded A. Therefore, PG8 will be updated and set to the low-priority state and will not be restored temporarily. Table 6 In the specific implementation, when there are only Low-Priority PGs to be restored in the queue, the MON node compares the utilization rates of the OSDs with data migration in these PGs, and starts to restore the PGs from the one with the lowest utilization rate until all PGs are restored.

[0045] As shown in Tables 7 and 8, after all the normally restored PGs are restored, only the PGs in the Low-Priority state remain, and the real-time utilization rates of the OSDs where the restored data is written in these PGs are all greater than the first threshold.

[0046] Table 7 Table 8 Then the recovery order is to restore PG9. After the restoration, there will be space released on OSD3. If the utilization rate of OSD3 becomes <A, then PG8 can be normally scheduled for restoration; if it is still greater than A, then compare the utilization rates of OSD3 and the OSDs with data migration in other PGs to be restored, and still start to restore the PG corresponding to the OSD with the lowest utilization rate.

[0047] As can be seen from the above embodiments, by performing differentiated scheduling and recovery of PGs, the problem of OSDs being filled in advance during the recovery process, which leads to the failure of the expansion process and the interruption of services, is effectively prevented.

[0048] Based on the above method embodiments, this disclosure also provides a storage device, in which a Ceph cluster is deployed, the storage device comprising: The acquisition module is used to acquire the first utilization rate of the target object storage device OSD scheduled by the target PG when the target PG is restored; The comparison module is used to compare the first utilization rate with the first threshold and obtain the recovery data flow of the target OSD; The processing module is used to determine whether to perform a recovery operation on the target OSD based on the comparison results and the recovery data flow.

[0049] The recovery data flow of the target OSD includes: a data flow written to the OSD or a data flow read from the OSD.

[0050] Specifically, the processing module is configured to delay the data recovery action of the target OSD when the comparison result shows that the first utilization rate is greater than the first threshold, and the recovery data flow direction of the target OSD is the data flow written to the OSD; or, If the comparison result is that the first utilization rate is less than or equal to the first threshold, and the recovery data flow of the target OSD is the data flow written to the OSD, then the data recovery action of the target OSD is executed.

[0051] Specifically, the processing module is used to perform data recovery actions on the target OSD when the recovery data flow of the target OSD is a data flow that reads the OSD.

[0052] Specifically, the processing module is used to add a Low-Priority status identifier to the target OSD and delay the data recovery action of the target OSD according to the Low-Priority status identifier.

[0053] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0054] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0055] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0056] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0057] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for recovering a home group (PG), characterized by, The method comprises: When the target PG is recovered, a first usage rate of a target object storage device OSD scheduled by the target PG is acquired; The first usage rate is compared with a first threshold, and a recovery data flow direction of the target OSD is acquired; Whether a recovery operation is performed on the target OSD is determined according to the comparison result and the recovery data flow direction.

2. The method of claim 1, wherein, The recovery data flow direction of the target OSD comprises a data flow of writing into the OSD or a data flow of reading from the OSD.

3. The method of claim 2, wherein, The determination whether the recovery operation is performed on the target OSD according to the comparison result and the recovery data flow direction comprises: When the comparison result is that the first usage rate is greater than the first threshold and the recovery data flow direction of the target OSD is the data flow of writing into the OSD, a data recovery action of the target OSD is delayed; or When the comparison result is that the first usage rate is less than or equal to the first threshold and the recovery data flow direction of the target OSD is the data flow of writing into the OSD, the data recovery action of the target OSD is performed.

4. The method of claim 1, wherein, The determination whether the recovery operation is performed on the target OSD according to the comparison result and the recovery data flow direction comprises: When the recovery data flow direction of the target OSD is the data flow of reading from the OSD, the data recovery action of the target OSD is performed.

5. The method of claim 3, wherein, The delaying of the data recovery action of the target OSD comprises: A Low-Priority state identifier is added to the target OSD, and the data recovery action of the target OSD is delayed according to the Low-Priority state identifier.

6. A storage device, characterized by The storage device is deployed with a Ceph cluster, and the storage device comprises: An acquisition module, configured to acquire a first usage rate of a target object storage device OSD scheduled by a target PG when the target PG is recovered; A comparison module, configured to compare the first usage rate with a first threshold, and acquire a recovery data flow direction of the target OSD; A processing module, configured to determine whether a recovery operation is performed on the target OSD according to the comparison result and the recovery data flow direction.

7. The storage device of claim 6, wherein, The recovery data flow direction of the target OSD comprises a data flow of writing into the OSD or a data flow of reading from the OSD.

8. The storage device of claim 7, wherein The processing module is specifically configured to delay the data recovery action of the target OSD when the comparison result is that the first usage rate is greater than the first threshold and the recovery data flow direction of the target OSD is the data flow of writing into the OSD; or perform the data recovery action of the target OSD when the comparison result is that the first usage rate is less than or equal to the first threshold and the recovery data flow direction of the target OSD is the data flow of writing into the OSD.

9. The storage device of claim 6, wherein The processing module is specifically configured to perform the data recovery action of the target OSD when the recovery data flow direction of the target OSD is the data flow of reading from the OSD.

10. The storage device of claim 8, wherein The processing module is specifically configured to add a Low-Priority state identifier to the target OSD, and delay the data recovery action of the target OSD according to the Low-Priority state identifier.

Images