Hard disk isolation method and device, electronic equipment and storage medium

Abstract

This application provides a hard disk isolation method, apparatus, electronic device, and storage medium. In this application, by first determining the pool state of the hard disk pool to which the hard disk to be detected belongs, then detecting whether the hard disk pool is in a steady state, and finally combining the isolation determination method based on fault redundancy type and the number of abnormal devices, the problem of low isolation efficiency caused by repeated isolation determinations triggered by reliance on PG status can be effectively avoided in existing methods. This improves isolation efficiency and minimizes the impact on hard disk-related services.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to hard disk isolation methods, devices, electronic devices and storage media. Background Technology

[0002] In a distributed storage system, when abnormal events such as slow disks or input / output (IO) timeouts are detected on a hard drive, the hard drive needs to be actively isolated in order to ensure the normal and stable operation of the business. Before actively isolating the hard drive, it is necessary to determine whether the hard drive can be isolated. Only hard drives that are determined to be isolable are then isolated.

[0003] Currently, in practical applications, hard drive isolation is typically achieved by relying on the status of the Placement Group (PG) to which the hard drive is located. However, during the isolation process, if the PG's status does not meet the isolation criteria (such as the PEERING status, which indicates that the PG replicas are negotiating data versions and synchronizing metadata to confirm data consistency, triggered by data synchronization or reconstruction during the isolation of other disks), a retry will be triggered, i.e., the isolation criteria for that hard drive will be re-evaluated until the required number of retries is reached or the PG's status is detected as meeting the isolation criteria. This can lead to the inability to complete the isolation of the hard drive in a timely manner, reducing isolation efficiency and thus affecting the services associated with that hard drive. Summary of the Invention

[0004] In view of this, this application provides a hard disk isolation method, apparatus, electronic device, and storage medium to improve isolation efficiency.

[0005] This application provides a hard disk isolation method, which is applied to a storage node in a distributed storage system; the method includes: For the hard drive to be tested, if the pool status of the hard drive pool to which the hard drive belongs does not meet the set full state, check whether the hard drive pool is in a steady state; any hard drive pool is in a steady state when the placement group (PG) on each hard drive in the hard drive pool is in the same normal working state. If the disk pool is in a steady state, the disk is isolated; if the disk pool is not in a steady state, the disk is isolated according to the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool; the abnormal devices associated with the disk pool are determined based on the fault redundancy type configured for the disk pool.

[0006] This application embodiment also provides a hard disk isolation device, which is applied to a storage node in a distributed storage system; the device includes: The isolation module is used to detect whether the hard drive pool to which the hard drive belongs is in a steady state when the pool status does not meet the set full state. A hard drive pool is in a steady state when all the placement groups (PGs) on each hard drive in the hard drive pool are in the same normal working state. If the disk pool is in a steady state, the disk is isolated; if the disk pool is not in a steady state, the disk is isolated according to the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool; the abnormal devices associated with the disk pool are determined based on the fault redundancy type configured for the disk pool.

[0007] This application also provides an electronic device, which includes: Processor; and A computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method described above.

[0008] This application also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the above method.

[0009] As can be seen from the above technical solutions, in this embodiment of the application, by first determining the pool status of the hard disk pool to which the hard disk to be detected belongs, then determining whether the hard disk pool is in a steady state, and then combining the isolation determination method of fault redundancy type and number of abnormal devices, the problem of low isolation efficiency caused by repeated isolation determination due to reliance on PG status in the existing solution can be effectively avoided, thereby improving isolation efficiency and minimizing the impact on hard disk-related services.

[0010] Furthermore, in this embodiment, when the disk pool is not in a steady state, isolation is not directly prohibited. Instead, disk isolation is performed based on the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool. Each disk to be tested is processed serially on the same node. This avoids the problem of multiple disks simultaneously requesting isolation, which would prevent the isolation status of other disks from being detected. This effectively avoids the problem of excessive fault redundancy caused by multiple disks simultaneously requesting isolation (i.e., the number of abnormal devices in the devices such as nodes or racks where the data replicas or data shards of the same data in the disk pool are distributed exceeds the range allowed by the fault redundancy of the disk pool). This ensures that the disk isolation operation is always performed within the range allowed by the fault redundancy of the disk pool, thereby effectively preventing data redundancy failure and business continuity interruption caused by excessive fault redundancy, and ensuring the data security and business stability of the distributed storage system. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a hard disk isolation method provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram illustrating the implementation of hard disk isolation provided in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram illustrating the implementation of the disk state machine provided in an embodiment of this application.

[0014] Figure 4 This is a schematic diagram of a hard disk isolation device provided in an embodiment of this application.

[0015] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] In this embodiment, in a distributed storage system, when the storage engine or device management module detects a hard drive failure such as a slow disk or IO timeout, the failed hard drive needs to be actively isolated to ensure the normal and stable operation of the business. In order to take into account the requirements of data redundancy, the hard drive needs to be isolated before active isolation. The actual isolation action is then performed on the hard drive that is determined to be isolable.

[0017] In this embodiment, as an example, the isolation of the hard drive can be achieved in the following manner: One hard disk corresponds to at least one Object Storage Device (OSD) instance. This OSD instance is used for hard disk data management, such as data storage, replication, and recovery. Based on this, in this embodiment, the OSD instance can be used to handle hard disk isolation. For example, after receiving an isolation task from the OSD control module (OSDCTRL module) in the system, the OSD instance first checks whether its own running status is ACTIVE (i.e., active state, indicating that the OSD instance can normally handle data read / write, replication, PG management, and other services), and then retrieves the latest OSDMAP (i.e., OSD cluster state mapping table) cached locally by the OSD instance. It then checks whether the OSD instance's status in the OSDMAP is UP+IN (UP status indicates that the OSD instance is in the O+IN state). The OSD instance is online in the SD cluster; the IN state indicates that the OSD instance is a valid member of the OSD cluster and has the ability to participate in data storage and scheduling in the OSD cluster. If the OSD instance itself is in the ACTIVE state and the OSD instance's state in OSDMAP is UP+IN (+ means AND, i.e., UP and IN state), then the OSD instance itself will handle the isolation task. Otherwise, the OSD instance needs to send the isolation task to the cluster management module (Manager, MGR) in the system so that the MGR can handle the isolation task.

[0018] The specific implementation of the isolation task being handled by the OSD instance itself can be as follows: First, the OSD instance sends a cluster health query request to the MGR to obtain the overall health status of the current OSD cluster.

[0019] If the cluster health status is healthy (that is, the status of PGs on the hard disks corresponding to each OSD instance in the current OSD cluster is in ACTIVE+CLEAN state, ACTIVE state indicates that the PG has normal data read and write service capabilities, and CLEAN state indicates that the number of all PG replicas corresponding to the PG is the same as the total number of redundant replicas indicated by the data redundancy policy configured for the PG, and the data between the PG replicas is complete and consistent), then the isolation result can be directly returned to the OSDCTRL module as isolation allowed.

[0020] Otherwise, based on the latest local OSDMAP, the OSD instance iterates through all storage pools to which it belongs, checking whether the capacity status (i.e., pool status) of each storage pool is FULL (i.e., full capacity, indicating that the used storage space of the storage pool has reached the preset FULL threshold) or NEAR FULL (i.e., close to full capacity, indicating that the used storage space of the storage pool has reached the preset NEAR FULL threshold but has not yet reached the preset FULL threshold, and the preset NEAR FULL threshold is less than the preset FULL threshold). If the capacity status of any storage pool is FULL or NEAR FULL, the isolation result is directly returned to the OSDCTRL module as isolation is not allowed. If the capacity status of all storage pools is not FULL or NEAR FULL, the following OSD redundancy check continues.

[0021] The OSD redundancy check process is as follows: Traverse all PGs on the disk corresponding to the OSD instance (that is, all PGs carried by the OSD instance itself), and take the traversed PG as the current PG; if the current PG is the master PG, it can be isolated directly; if the current PG is the slave PG, an isolation request needs to be constructed and sent to the OSD instance corresponding to the master PG of the slave PG for isolation (for example, it can be based on whether the master PG is on the current node to distinguish between affinity sending and network sending).

[0022] The isolation process for the primary PG is as follows: If the current PG's state is NOT ACTIVE or STALE (NOTACTIVE means inactive, indicating the PG has lost its normal data read / write service capabilities; STALE means disconnected, indicating the PG has not reported its latest status to the cluster control node Monitor for a long time and is disconnected), then the isolation result is determined to be "isolation not allowed". If the current PG's state is ACTIVE and CLEAN, then the isolation result is determined to be "isolation allowed". When the current PG is in the AVTIVE+NOT CLEAN state, it is necessary to first check the ACTING SIZE (i.e., the actual number of valid PG replicas corresponding to the PG) and the MIN SIZE configured on the storage pool to which the current PG belongs (i.e., the minimum number of valid PG replicas that the storage pool is configured to maintain). If the ACTING SIZE is not greater than the MIN SIZE, the isolation result is determined to be "isolation not allowed". If the ACTING SIZE is greater than the MIN SIZE, the MISSING record of the current PG is obtained (the record contains the lost and reconstructed objects in the PG and their corresponding OSD instances, etc.). The MISSING record is traversed to determine whether the OSD instance corresponding to the current PG carries the data that is essential for the reconstruction of these lost objects (i.e., the OSD instance is the only node that stores the data of the lost object, and there are no other replicas. If isolated, the lost objects of the PG cannot be reconstructed and the data will be permanently lost). If not, the isolation result is determined to be "isolation allowed". Otherwise, the isolation result is determined to be "isolation not allowed". Based on this, the OSD instance waits to summarize the isolation results corresponding to all PGs. If the isolation results corresponding to all PGs are allowed to isolate, it returns the isolation result as allowed to isolate to the OSDCTRL module. If the isolation result corresponding to any PG is not allowed to isolate, it returns the isolation result as not allowed to isolate to the OSDCTRL module.

[0023] The specific implementation of the isolation by MGR is as follows: The OSD instance sends an isolation request to the MGR (i.e., the isolation task mentioned above). After receiving the isolation request, the MGR queries all storage pools to which the OSD instance belongs and determines the status of the PGs on all storage pools: If the status of any PG satisfies that the ACTING SIZE is less than the MIN SIZE of the storage pool, then the MGR returns an isolation result to the OSDCTRL module indicating that isolation is not allowed; if the status of any PG is PERRING, CREATING, or ACTIVATING (PERRING state is the peer negotiation state, indicating that the OSD instances associated with each PG replica are in the transition process of negotiating data versions and synchronizing metadata to confirm data consistency, and services are not yet stably provided; CREATING state is the creation state, indicating that the PG is in the process of creation and does not yet have the ability to provide business services; ACTIVATING state is the activation state, indicating that the PG has completed the above PEERING or CREATING process and is in the process of switching to ACTIVE). If the state transition process is complete, a retry is returned, meaning the OSD instance re-attempts mechanical isolation; otherwise, for other cases not described above, the OSDCTRL module returns an isolation result indicating that isolation is allowed.

[0024] However, in the isolation methods described above, since disk isolation is performed by the corresponding OSD instance, the isolation of multiple disks is independent of each other. This means that when multiple disks are isolated almost simultaneously, the isolation status of other disks cannot be detected. Therefore, this could lead to excessive fault redundancy and affect business continuity. Furthermore, because disk isolation relies on the state of the PG (Post Group). If the PG's state does not meet the isolation conditions, retries will be repeatedly triggered, meaning the disk will be isolated again. This can result in disk isolation not being completed in a timely manner, affecting services associated with that disk. In other words, this can lead to multiple disks on the same node triggering isolation almost simultaneously, resulting in untimely isolation and impacting front-end services.

[0025] Based on this, and to solve the above problems, this application further proposes a hard disk isolation method. To make the method provided by this application easier to understand, the method will be described in detail below with reference to the accompanying drawings and embodiments.

[0026] See Figure 1 , Figure 1This is a flowchart illustrating a method provided in an embodiment of this application. In this embodiment, the method can be applied to a storage node in a distributed storage system; optionally, the storage node can be, for example, a storage node in the distributed storage system that has deployed a master OSDMON module (which can be referred to as the master node). The OSDMON module is an OSD monitoring module, and each storage node has one OSDMON module deployed on it. It can be used to monitor the status of OSD instances on the node and report it to the master OSDMON module for aggregation and updating of the maintained global OSDMAP.

[0027] like Figure 1 As shown, the process may include the following steps: Step 101: For the hard drive to be tested, if the state of the hard drive pool to which the hard drive belongs does not meet the set full state, check whether the hard drive pool is in a steady state. If yes, proceed to step 102; otherwise, proceed to step 103.

[0028] In this embodiment, as an example, in a distributed storage system, when any storage node detects an abnormal event such as a slow disk or IO timeout on a hard drive on its own node, it generates an isolation request for the hard drive and sends it to the master node. The master node then performs an isolation determination on the hard drive based on the isolation request to determine whether isolation of the hard drive is allowed, and returns the isolation result to the storage node.

[0029] For example, see Figure 2 As shown, when the OSDCTRL module (denoted as OSDCTRL module 1) in any storage node detects an abnormal event such as a slow disk or IO timeout on a hard drive on its local node, it generates an isolation request for that hard drive and sends it to the OSDMON module (denoted as OSDMON module 1) in the master node. The OSDMON module in the master node then performs an isolation determination on the hard drive based on the isolation request to determine whether isolation is allowed. The isolation result is returned to the OSDCTRL module in the storage node, which then performs subsequent operations based on the received isolation result. If isolation is allowed, the actual isolation operation is performed on the hard drive. The actual isolation operation is not specifically limited here; for example, the state of the OSD instance corresponding to the hard drive can be switched to DOWN state (meaning the OSD instance is offline and unreachable, unable to establish effective communication with other components / modules in the system).

[0030] Based on this, in this embodiment, the master node can process the received isolation requests sequentially according to the order of reception time (that is, process the received isolation requests serially on the master node). Each isolation request at least indicates the identifier of the hard drive to be detected (such as the disk's unique serial number (SN)). Specifically, as an example, for the hard drive to be detected, the pool status of the hard drive pool to which the hard drive belongs can be checked first to see if it meets the set full state. If so, the hard drive is isolated; if not, the hard drive pool is further checked to see if it is in a steady state.

[0031] In this embodiment, the hard disks in the distributed storage system can be divided into multiple hard disk pools, and any hard disk pool contains at least one hard disk from each storage node in the distributed storage system. For example, assuming the distributed storage system contains 2 storage nodes, and each node contains 6 hard disks, if it is to be divided into 2 hard disk pools, then any hard disk pool can contain 3 hard disks from node 1 and 3 hard disks from node 2.

[0032] A stable state for any disk pool can mean that all the PGs (Portable Spectrum Controllers) on the disks in that pool are currently in the same normal operating state. For example, this normal operating state can refer to the ACTIVE+CLEAN state. For the specific meanings of PGs being in the ACTIVE and CLEAN states, please refer to the relevant descriptions above, which will not be repeated here.

[0033] The above-mentioned full state settings can include FULL state and NEAR FULL state. For the specific meanings of FULL state and NEAR FULL state, please refer to the relevant descriptions above, which will not be repeated here.

[0034] Step 102: Isolate the hard drive.

[0035] Step 103: Isolate the hard disk according to the fault redundancy type configured for the hard disk pool and the number of abnormal devices associated with the hard disk pool; the abnormal devices associated with the hard disk pool are determined based on the fault redundancy type configured for the hard disk pool.

[0036] In this embodiment, the fault redundancy type is used to indicate that when storing data, different data copies or data shards of the same data are deployed on the dimension of redundant devices. That is, different data copies or data shards of the same data are distributed on multiple different redundant devices. These data copies or data shards can be mapped to PGs for data management. In other words, different data copies or data shards of the same data can be understood as different PG copies of the same data, and one data copy or data shard can correspond to one PG copy.

[0037] Optionally, fault redundancy types may include node redundancy and rack redundancy. In node redundancy, the redundant device is a storage node, and different data copies or data shards of the same data are distributed across multiple different storage nodes. In rack redundancy, the redundant device is a rack, and different data copies or data shards of the same data are distributed across multiple different racks.

[0038] Based on the above description, as an example, if the fault redundancy type configured for the hard disk pool is node redundancy, then the abnormal devices associated with the hard disk pool under node redundancy are the abnormal storage nodes that have experienced an anomaly. The number of abnormal devices associated with the hard disk pool includes the number of abnormal storage nodes in the storage nodes where each hard disk in the hard disk pool resides. Therefore, in this step, the hard disk is isolated according to the fault redundancy type configured for the hard disk pool and the number of abnormal devices associated with the hard disk pool. Specifically, this can be implemented by isolating the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes allowed to be abnormal under the node redundancy type.

[0039] As another embodiment, if the fault redundancy type configured for the hard disk pool is rack redundancy, then the abnormal devices associated with the hard disk pool under rack redundancy are the abnormal racks that have experienced an anomaly. The number of abnormal devices associated with the hard disk pool includes the number of abnormal racks in the racks where each hard disk in the hard disk pool is located. Based on this, in this step, the hard disk is isolated according to the fault redundancy type configured for the hard disk pool and the number of abnormal devices associated with the hard disk pool. In specific implementation, for example, the hard disk can be isolated according to the number of abnormal racks associated with the hard disk pool and the number of racks allowed to be abnormal under the rack redundancy type.

[0040] Regarding the determination of abnormal racks and abnormal storage nodes, the specific methods for isolating a hard drive based on the number of abnormal storage nodes associated with the hard drive pool and the number of abnormal storage nodes allowed under the node redundancy type, as well as the specific methods for isolating a hard drive based on the number of abnormal racks associated with the hard drive pool and the number of abnormal racks allowed under the rack redundancy type, will be described with examples below and will not be elaborated here.

[0041] This concludes the process. Figure 1 The method flow is shown.

[0042] pass Figure 1As can be seen from the process shown, in this embodiment of the application, by first determining the pool status of the hard disk pool to which the hard disk to be detected belongs, then determining whether the hard disk pool is in a steady state, and then combining the isolation determination method of fault redundancy type and number of abnormal devices, the problem of low isolation efficiency caused by repeated isolation determination due to reliance on PG status in the existing solution can be effectively avoided, thereby improving isolation efficiency and minimizing the impact on hard disk-related services.

[0043] Furthermore, in this embodiment, when the disk pool is not in a steady state, isolation is not directly prohibited. Instead, disk isolation is performed based on the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool. Each disk to be tested is processed serially on the same node. This avoids the problem of multiple disks simultaneously requesting isolation, which would prevent the isolation status of other disks from being detected. This effectively avoids the problem of excessive fault redundancy caused by multiple disks simultaneously requesting isolation (i.e., the number of abnormal devices in the devices such as nodes or racks where the data replicas or data shards of the same data in the disk pool are distributed exceeds the range allowed by the fault redundancy of the disk pool). This ensures that the disk isolation operation is always performed within the range allowed by the fault redundancy of the disk pool, thereby effectively preventing data redundancy failure and business continuity interruption caused by excessive fault redundancy, and ensuring the data security and business stability of the distributed storage system.

[0044] The following example illustrates how to determine whether the aforementioned disk pool is in a steady state: In this embodiment, as an example, in a distributed storage system, each OSD instance periodically reports a message carrying `max_last_epoch_clean` to the OSDMON module in the master node. Here, `max_last_epoch_clean` represents the maximum value among all `last_epoch_clean` values ​​corresponding to all master PGs hosted by the OSD instance itself; the `last_epoch_clean` corresponding to any master PG represents the OSDMAP version number corresponding to the last time that master PG was in the ACTIVE+CLEAN state; the OSDMAP version number here is a globally incrementing integer, and each change in the OSD instance state triggers an update of OSDMAP and the generation of a new OSDMAP version number.

[0045] Based on this, if all OSD instances in the disk pool that are not in a DOWN state report a max_last_epoch_clean value equal to the current global OSDMAP version number, then it is determined that the PGs on each disk in the disk pool are currently in the ACTIVE+CLEAN state, further confirming that the disk pool is in a stable state; otherwise, it is determined that the disk pool is not in a stable state. Here, a non-DOWN state OSD instance can refer to an OSD instance in the UP, RECOVER, or IN state.

[0046] The determination of the aforementioned abnormal racks and abnormal storage nodes is described below: First, combine Figure 3 The OSD state machine shown below provides an example of how the states of an OSD instance are described: See Figure 3 As shown, the states of an OSD instance may include UP, DOWN, OUT, RECOVER, and FAULT states.

[0047] In this context, an OSD instance in the UP (online) state indicates that it is online within the OSD cluster, the data on its corresponding hard drive is complete, and the number of valid PG replicas corresponding to any PG on that hard drive is greater than the MIN SIZE configured on the storage pool to which that PG belongs. In this state, the hard drive corresponding to the OSD instance can be isolated from the cluster at any time.

[0048] When an OSD instance is in a DOWN state (i.e., offline state), it means that the OSD instance is in an offline and unreachable state. It cannot establish effective communication with other OSD instances in the OSD cluster, but the OSD cluster has not yet removed it from the list of valid members. The data on its corresponding hard drive may be incomplete, but the data reconstruction process has not been triggered. Isolating the hard drive corresponding to this OSD instance from the cluster in this state may result in data loss.

[0049] When an OSD instance is in the OUT state (i.e., out of group state), it means that the OSD instance has been removed from the list of valid members of the OSD cluster and no longer has the ability to participate in data storage and scheduling. Furthermore, it has not triggered data reconstruction or completed data migration. Isolating the disks corresponding to the OSD instance from the cluster in this state may result in data loss.

[0050] When an OSD instance is in the RECOVER state (i.e., a reconstruction and recovery state), it indicates a transitional state after recovering from an abnormal state (such as DOWN or OUT). At this time, the OSD instance has reconnected to the OSD cluster and established communication, but it is currently undergoing data reconstruction and recovery. It will transition to the UP state once the data reconstruction and recovery are complete. Isolating the disks corresponding to the OSD instance from the cluster during this state may result in data loss.

[0051] If an OSD instance is in a FAULT state (i.e., a failed state), it means that all local data carried by that OSD instance has been completely migrated to other valid OSD instances through the data reconstruction process. The local data has lost its validity, and the OSD cluster will no longer assign data service tasks such as data storage and backup to it.

[0052] When an OSD instance switches from UP to DOWN state, it may be due to a fault in the hard drive corresponding to the OSD instance, such as being unplugged, slow, or the storage node where the hard drive is located failing.

[0053] When an OSD instance switches from DOWN to OUT state, it may be due to a fault such as a bad disk, strong isolation, or a storage node failure exceeding a set time limit, such as 30 minutes.

[0054] When an OSD instance switches from UP to OUT state, it may be due to a fault in the hard drive corresponding to the OSD instance, such as a bad disk or strong isolation.

[0055] When an OSD instance switches from the OUT state to the FAULT state, it may be because the disk corresponding to the OSD instance has completed a full data reconstruction, and all data on the disk has been migrated to other disks in the cluster.

[0056] When an OSD instance switches from DOWN state to RECOVER state, it may be due to the hard drive corresponding to the OSD instance having completed fault recovery (such as disk reset, pre-read pull-up, or the storage node where the hard drive is located completing fault recovery).

[0057] When an OSD instance switches from the OUT state to the RECOVER state, it may be due to the hard drive corresponding to the OSD instance completing recovery (such as when the hard drive is removed for more than a set time, such as 30 minutes, and then reinserted, or when a forced recovery operation is performed via command line).

[0058] When an OSD instance switches from RECOVER state to UP state, it may be because the disk corresponding to the OSD instance has successfully completed incremental data reconstruction and recovery, and the OSD cluster has returned to a healthy operating state.

[0059] When an OSD instance switches from FAULT state to UP state, it may be because the user performs a forced restart recovery operation via command line after the hard disk corresponding to the OSD instance has completed strong isolation and data reconstruction.

[0060] Based on the above description, in this embodiment, as an example, an OSD instance that meets any of the following conditions is an abnormal OSD instance: The OSD instance is in the first preset state; this first preset state indicates that the OSD instance is offline and unreachable, but the OSD instance has the ability to participate in data processing tasks. The OSD instance is in the second preset state; this second preset state indicates that the OSD instance is offline and unreachable, and does not have the ability to participate in data processing tasks. The OSD instance is in the third preset state; this third preset state indicates that the OSD instance is undergoing local data reconstruction and recovery.

[0061] For example, the state of the OSD instance mentioned above, which is the first preset state, can be understood as the OSD instance switching from the UP state to the DOWN state, but not being OUT, that is, the state of the OSD instance is DOWN and IN (i.e., DOWN+ IN state).

[0062] For example, the above OSD instance's state being the second preset state can be understood as follows: after the OSD instance switches from the UP state to the DOWN state and then to the OUT state, it does not send a Boot message to the OSDCTRL module to start the OSD instance and switch it back to the UP state. It is currently in data reconstruction but has not reached the FAULT state. In other words, the OSD instance's state is both DOWN and OUT (i.e., DOWN + OUT state).

[0063] For example, the above OSD instance's state being the third preset state can be understood as follows: after the OSD instance switches from the UP state to the DOWN state and then to the OUT state, it sends a Boot message to the OSDCTRL module during the data reconstruction process to start the OSD instance. After starting, the OSD instance is in the RECOVER state, that is, the OSD instance's state is RECOVER.

[0064] In this context, the OSD instance status is as follows: DOWN indicates that the OSD instance is offline and unreachable; IN indicates that the OSD instance is a valid member of the OSD cluster and has the ability to participate in data processing tasks such as data storage, replication, and synchronization; OUT indicates that the OSD instance does not have the ability to participate in data processing tasks such as data storage, replication, and synchronization; and RECOVER indicates that the OSD instance is undergoing local data reconstruction and recovery.

[0065] Therefore, in this embodiment, if any OSD instance corresponding to a hard drive is abnormal, it indicates that the hard drive is abnormal; if any storage node has an abnormal hard drive, it indicates that the storage node is abnormal; if any rack has an abnormal storage node, it indicates that the rack is abnormal. This achieves the determination of abnormal racks and abnormal storage nodes.

[0066] It should be noted that the above-described OSD instance status is merely an illustrative example, and this embodiment does not specifically limit it. For details, please refer to the relevant descriptions of OSD instance status in related technologies.

[0067] The following describes how the disk is isolated based on the number of abnormal storage nodes associated with the disk pool and the number of abnormal storage nodes allowed under the redundancy type of the node: In this embodiment, as one example, the above-mentioned isolation of the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type can be implemented in a specific way, for example: First, based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes where the hard disks belonging to the hard disk pool are located among all the hard disks to be detected, the total number of abnormal nodes corresponding to the hard disk pool is determined.

[0068] Optionally, in this embodiment, for example, the number of non-abnormal storage nodes in the storage nodes where the hard drives belonging to the hard drive pool are located among all the hard drives to be detected can be obtained, and the total number of abnormal nodes corresponding to the hard drive pool can be determined based on the sum of the obtained number and the number of abnormal storage nodes associated with the hard drive pool.

[0069] Here, "all hard drives currently under inspection" can refer to the hard drives indicated by all received but not yet processed isolation requests. Subsequently, the hard drive can be isolated based on the data redundancy policy of the hard drive pool, the total number of abnormal nodes corresponding to the hard drive pool, and the number of storage nodes allowed to be abnormal under the node redundancy type.

[0070] Optionally, in this embodiment, isolating the hard disk based on the data redundancy strategy of the data in the hard disk pool, the total number of abnormal nodes corresponding to the hard disk pool, and the number of storage nodes allowed to be abnormal under the node redundancy type can, in specific implementations, include, for example: If the data redundancy strategy in the hard disk pool is not the first preset strategy, the hard disk can be isolated based on the total number of abnormal nodes corresponding to the hard disk pool and the number of storage nodes allowed to be abnormal under the node redundancy type. Specifically, if the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of storage nodes allowed to be abnormal under the node redundancy type, the hard disk is isolated; if the total number of abnormal nodes corresponding to the hard disk pool is greater than the number of storage nodes allowed to be abnormal under the node redundancy type, the hard disk is not isolated.

[0071] The first preset strategy can be, for example, an erasure coding (EC) strategy that sets a limit on the number of storage nodes allowed to malfunction. Therefore, situations where the first preset strategy is not used may include replication strategies or EC strategies that do not set a limit on the number of storage nodes allowed to malfunction, etc., and are not limited here.

[0072] If the data redundancy strategy in the hard disk pool is the first preset strategy, that is, the EC strategy which sets the number of storage nodes that can be abnormal, then for each storage pool into which the hard disk pool is divided, the hard disk is isolated based on the total number of abnormal nodes corresponding to the hard disk pool and the number of abnormal nodes allowed for the storage pool, so as to obtain the isolation result corresponding to the storage pool; and the hard disk is isolated according to the isolation result corresponding to each storage pool into which the hard disk pool is divided. Specifically, for each storage pool into which the disk pool is divided, check whether the total number of abnormal nodes corresponding to the disk pool is less than or equal to the allowed number of abnormal nodes corresponding to the storage pool. If so, determine that the isolation result for the storage pool is to isolate the disk. If not, determine that the isolation result for the storage pool is to isolate the disk when the number of abnormal disks in the disk pool is less than or equal to the allowed number of abnormal disks corresponding to the storage pool. If the number of abnormal disks in the disk pool is greater than the allowed number of abnormal disks corresponding to the storage pool, determine that the isolation result for the storage pool is not to isolate the disk. Then, if the isolation results for each storage pool into which the disk pool is divided all indicate that the disk should be isolated, then the disk should be isolated; otherwise, the disk should not be isolated.

[0073] Optionally, in this embodiment, the number of storage nodes allowed to be abnormal under this node redundancy type and the number of allowed abnormal nodes corresponding to any storage pool can both be determined based on the data redundancy strategy of the data in the disk pool.

[0074] Specifically, as an example, each disk pool can be divided into multiple storage pools, analogous to dividing the storage space provided by each disk in the disk pool into multiple sub-spaces, i.e., storage pools. Each storage pool contains a portion of the storage space of at least one disk on each storage node associated with the disk pool. Based on this, data redundancy in the disk pool is data redundancy at the storage pool level, meaning that different data replicas or data shards of the same data will not exist across storage pools. Therefore, the data redundancy strategy in the disk pool can include the data redundancy strategies of the data in each storage pool into which the disk pool is divided. The strategy type (such as replication strategy or EC strategy) of the data redundancy strategy corresponding to different storage pools is the same, but the strategy content of the data redundancy strategy can be the same or different, which is not specifically limited here.

[0075] Therefore, if the data redundancy strategy corresponding to each storage pool in which the disk pool is divided is not the first preset strategy, such as a replication strategy or an EC strategy that does not set the number of storage nodes allowed to be abnormal, then the minimum value among the allowed number of abnormal nodes corresponding to each storage pool can be used as the allowed number of abnormal storage nodes under that node redundancy type. Specifically, the allowed number of abnormal nodes corresponding to any storage pool can be determined based on the difference between the total number of redundant nodes indicated by the data redundancy strategy corresponding to that storage pool and the minimum number of redundant nodes that must be maintained (i.e., the aforementioned MIN SIZE). For example, this difference can be directly used as the allowed number of abnormal nodes corresponding to that storage pool.

[0076] If the data redundancy policy corresponding to each storage pool in which the disk pool is divided is the first preset policy, that is, the EC policy which sets the number of storage nodes that are allowed to be abnormal, then the number of storage nodes that are allowed to be abnormal indicated by the data redundancy policy corresponding to each storage pool can be used as the number of storage nodes that are allowed to be abnormal in each storage pool. Here, the number of storage nodes that are allowed to be abnormal under the node redundancy type includes the number of storage nodes that are allowed to be abnormal in each storage pool.

[0077] In this embodiment, the number of allowed faulty hard drives corresponding to any storage pool can also be determined based on the data redundancy policy of the data in that hard drive pool. For example, it can be determined based on the number of allowed faulty hard drives indicated by the data redundancy policy corresponding to the storage pool. For example, assuming the EC policy with the allowed number of faulty storage nodes is set to k+m:1, where k represents the number of data shards of the same data, m represents the number of allowed faulty hard drives (which also identifies the number of check shards of the same data), and 1 represents the number of allowed faulty storage nodes, for example, EC policy 4+2:1, which means that under this EC policy, the same data is divided into 4 data shards, of which the number of allowed faulty hard drives is 2 and the number of allowed faulty storage nodes is 1.

[0078] The following describes how the hard drive is isolated based on the number of faulty racks associated with the hard drive pool and the number of faulty racks allowed under the rack redundancy type: In this embodiment, the specific implementation of isolating the hard disk based on the number of abnormal racks associated with the hard disk pool and the number of racks allowed to be abnormal under the rack redundancy type is similar to the specific implementation of isolating the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes allowed to be abnormal under the node redundancy type.

[0079] For example, as an embodiment, the above-mentioned isolation of the hard disk based on the number of abnormal racks associated with the hard disk pool and the number of racks allowed to be abnormal under the rack redundancy type can be implemented in a specific way, such as: First, based on the number of abnormal racks associated with the hard disk pool and the number of racks belonging to the hard disk pool among all the hard disks currently being detected, the total number of abnormal racks corresponding to the hard disk pool is determined.

[0080] Optionally, in this embodiment, for example, the number of non-abnormal racks in the racks where the hard drives belonging to the hard drive pool are located among all the hard drives to be detected can be obtained, and the total number of abnormal racks corresponding to the hard drive pool can be determined based on the sum of the obtained number and the number of abnormal racks associated with the hard drive pool.

[0081] Subsequently, the hard drive can be isolated based on the data redundancy policy of the data in the hard drive pool, the total number of abnormal racks corresponding to the hard drive pool, and the number of racks allowed to be abnormal under the rack redundancy type.

[0082] Optionally, in this embodiment, isolating the hard drive based on the data redundancy strategy of the data in the hard drive pool, the total number of abnormal racks corresponding to the hard drive pool, and the number of racks allowed to be abnormal under the rack redundancy type, may include, for example, the following: If the data redundancy strategy in the disk pool is not the second preset strategy, then the disk will be isolated based on the total number of abnormal racks corresponding to the disk pool and the number of racks allowed to be abnormal under the rack redundancy type. Specifically, if the total number of abnormal racks corresponding to the disk pool is less than or equal to the number of racks allowed to be abnormal under the rack redundancy type, the disk will be isolated; if the total number of abnormal racks corresponding to the disk pool is greater than the number of racks allowed to be abnormal under the rack redundancy type, the disk will not be isolated.

[0083] The second preset policy is an EC policy that sets a certain number of racks that can tolerate exceptions. Therefore, situations where the second preset policy is not used may include, for example, a replication policy or an EC policy that does not set a certain number of storage nodes that can tolerate exceptions; this is not a limitation.

[0084] If the data redundancy strategy in the hard disk pool is the second preset strategy, namely the EC strategy which sets the number of racks allowed for anomalies, then for each storage pool into which the hard disk pool is divided, the hard disk is isolated based on the total number of racks with anomalies corresponding to the hard disk pool and the number of racks with anomalies allowed to the storage pool, so as to obtain the isolation result corresponding to the storage pool; and the hard disk is isolated according to the isolation result corresponding to each storage pool into which the hard disk pool is divided. Specifically, for each storage pool into which the disk pool is divided, check whether the total number of abnormal racks corresponding to the disk pool is less than or equal to the number of allowed abnormal racks corresponding to the storage pool. If so, determine that the isolation result for the storage pool is to isolate the disk. If not, determine that the isolation result for the storage pool is to isolate the disk when the number of abnormal disks in the disk pool is less than or equal to the number of allowed abnormal disks corresponding to the storage pool. If the number of abnormal disks in the disk pool is greater than the number of allowed abnormal disks corresponding to the storage pool, determine that the isolation result for the storage pool is not to isolate the disk. Then, if the isolation results for each storage pool into which the disk pool is divided all indicate that the disk should be isolated, then the disk should be isolated; otherwise, the disk should not be isolated.

[0085] The number of racks allowed to malfunction under this rack redundancy type and the number of racks allowed to malfunction for any storage pool are both determined based on the data redundancy strategy of the disk pool. For details on how these are determined, please refer to the above description of the specific methods for determining the number of storage nodes allowed to malfunction under this node redundancy type and the number of storage nodes allowed to malfunction for any storage pool; these will not be repeated here.

[0086] Based on the above description, in this embodiment, as an example, if it is determined that the hard drive should be isolated, the actual isolation operation is performed directly on the hard drive; if it is determined that the hard drive should not be isolated, an isolation determination for the hard drive can be initiated every set time, such as 30 seconds, until the number of initiations for the hard drive reaches a set number, such as 3 times.

[0087] This concludes the description of the method provided in the embodiments of this application. The apparatus provided in the embodiments of this application will now be described: As an example, this embodiment also provides a hard disk isolation device. For example, see... Figure 4 , Figure 4 This is a schematic diagram of a hard disk isolation device provided in an embodiment of this application. The device corresponds to… Figure 1 The process is shown below. Figure 4 As shown, the hard disk isolation device 400 is applied to a storage node in a distributed storage system. The hard disk isolation device 400 includes: The isolation module 401 is used to: detect whether the hard disk pool to which the hard disk belongs is in a steady state when the pool state does not meet the set full state for the hard disk to be tested; any hard disk pool being in a steady state means that the placement group PG on each hard disk in the hard disk pool is in the same normal working state. If the disk pool is in a steady state, the disk is isolated; if the disk pool is not in a steady state, the disk is isolated according to the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool; the abnormal devices associated with the disk pool are determined based on the fault redundancy type configured for the disk pool.

[0088] As one embodiment, isolating the hard drive based on the fault redundancy type configured for the hard drive pool and the number of faulty devices associated with the hard drive pool includes: If the fault redundancy type configured for the hard disk pool is node redundancy, and the abnormal device associated with the hard disk pool under the node redundancy type is an abnormal storage node that has malfunctioned, then the hard disk is isolated according to the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type. If the fault redundancy type configured for the hard disk pool is rack redundancy, and the abnormal device associated with the hard disk pool under rack redundancy is an abnormal rack that has experienced an anomaly, then the hard disk is isolated based on the number of abnormal racks associated with the hard disk pool and the number of racks that are allowed to be abnormal under rack redundancy.

[0089] As one embodiment, isolating the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes allowed to be abnormal under the node redundancy type includes: Based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes where the hard disks belonging to the hard disk pool are located among all the hard disks to be detected, determine the total number of abnormal nodes corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the first preset policy, then the hard disk will be isolated when the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of storage nodes allowed to be abnormal under the node redundancy type; the first preset policy is the erasure coding EC policy which sets the number of storage nodes allowed to be abnormal. If the data redundancy policy in the hard disk pool is the first preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of allowed abnormal nodes corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of storage nodes allowed to be abnormal under this node redundancy type, the number of abnormal nodes allowed for any storage pool, and the number of abnormal hard disks allowed for any storage pool are all determined based on the data redundancy policy of the data in the hard disk pool; if any OSD instance corresponding to a hard disk is abnormal, it means that the hard disk is abnormal.

[0090] As one embodiment, isolating the hard drive based on the number of faulty racks associated with the hard drive pool and the number of racks allowed to be faulty under the rack redundancy type includes: Based on the number of abnormal racks associated with the hard disk pool and the number of racks in which the hard disks belonging to the hard disk pool are located among all the hard disks currently being detected, determine the total number of abnormal racks corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the second preset policy, then the hard disk will be isolated when the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of racks allowed to be abnormal under the rack redundancy type; the second preset policy is the EC policy which sets the number of racks allowed to be abnormal. If the data redundancy policy in the hard disk pool is the second preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of allowed abnormal racks corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of racks allowed to be abnormal under this rack redundancy type, the number of racks allowed to be abnormal for any storage pool, and the number of hard disks allowed to be abnormal for any storage pool are all determined based on the data redundancy policy of the data in the hard disk pool; if any OSD instance corresponding to a hard disk is abnormal, it means that the hard disk is abnormal.

[0091] As an example, if any rack has an abnormal storage node, it indicates that the rack is abnormal; if any storage node has an abnormal hard drive, it indicates that the storage node is abnormal; if any OSD instance corresponding to any hard drive is abnormal, it indicates that the hard drive is abnormal. An OSD instance that meets any of the following conditions is an abnormal OSD instance: the OSD instance is in the first preset state, the OSD instance is in the second preset state, or the OSD instance is in the third preset state. The first preset state indicates that the OSD instance is offline and unreachable, but the OSD instance has the ability to participate in data processing tasks; the second preset state indicates that the OSD instance is offline and unreachable, but does not have the ability to participate in data processing tasks; the third preset state indicates that the OSD instance is performing local data reconstruction and recovery.

[0092] This concludes the process. Figure 4 Structural description of the device shown.

[0093] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0094] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided as an exemplary embodiment of this application. The electronic device includes a processor and a computer-readable storage medium; the computer-readable storage medium stores a plurality of computer program instructions, which, when executed by the processor, implement the method disclosed in the above example of this application. Depending on the actual function of the electronic device, other hardware may also be included, which will not be elaborated further.

[0095] Based on the same concept as the above method, this application also provides a computer-readable storage medium storing a plurality of computer program instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0096] For example, the aforementioned computer-readable storage medium can be any electronic, magnetic, optical, or other physical storage device, such as: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof.

[0097] The above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A hard disk isolation method, characterized in that, This method is applied to a storage node in a distributed storage system; the method includes: For the hard drive to be tested, if the pool status of the hard drive pool to which the hard drive belongs does not meet the set full state, check whether the hard drive pool is in a steady state; any hard drive pool is in a steady state when the placement group (PG) on each hard drive in the hard drive pool is in the same normal working state. If the disk pool is in a steady state, the disk is isolated; if the disk pool is not in a steady state, the disk is isolated according to the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool; the abnormal devices associated with the disk pool are determined based on the fault redundancy type configured for the disk pool.

2. The method according to claim 1, characterized in that, The isolation of the hard drive based on the fault redundancy type configured for the hard drive pool and the number of faulty devices associated with the hard drive pool includes: If the fault redundancy type configured for the hard disk pool is node redundancy, and the abnormal device associated with the hard disk pool under the node redundancy type is an abnormal storage node that has malfunctioned, then the hard disk is isolated according to the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type. If the fault redundancy type configured for the hard disk pool is rack redundancy, and the abnormal device associated with the hard disk pool under rack redundancy is an abnormal rack that has experienced an anomaly, then the hard disk is isolated based on the number of abnormal racks associated with the hard disk pool and the number of racks that are allowed to be abnormal under rack redundancy.

3. The method according to claim 2, characterized in that, The step of isolating the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type includes: Based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes where the hard disks belonging to the hard disk pool are located among all the hard disks to be detected, determine the total number of abnormal nodes corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the first preset policy, then the hard disk will be isolated when the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of storage nodes allowed to be abnormal under the node redundancy type; the first preset policy is the erasure coding EC policy which sets the number of storage nodes allowed to be abnormal. If the data redundancy policy in the hard disk pool is the first preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of allowed abnormal nodes corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of storage nodes allowed to be abnormal under this node redundancy type, the number of abnormal nodes allowed for any storage pool, and the number of abnormal hard disks allowed for any storage pool are all determined based on the data redundancy policy of the data in the hard disk pool; if the object storage device OSD instance corresponding to any hard disk is abnormal, it indicates that the hard disk is abnormal.

4. The method according to claim 2, characterized in that, The process of isolating the hard drive based on the number of faulty racks associated with the hard drive pool and the number of racks allowed to be faulty under the rack redundancy type includes: Based on the number of abnormal racks associated with the hard disk pool and the number of racks in which the hard disks belonging to the hard disk pool are located among all the hard disks currently being detected, determine the total number of abnormal racks corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the second preset policy, then the hard disk will be isolated when the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of racks allowed to be abnormal under the rack redundancy type; the second preset policy is the EC policy which sets the number of racks allowed to be abnormal. If the data redundancy policy in the hard disk pool is the second preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of allowed abnormal racks corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of racks allowed to be abnormal under this rack redundancy type, the number of racks allowed to be abnormal for any storage pool, and the number of hard disks allowed to be abnormal for any storage pool are all determined based on the data redundancy policy of the data in the hard disk pool; if any OSD instance corresponding to a hard disk is abnormal, it means that the hard disk is abnormal.

5. The method according to claim 2, characterized in that, If any rack contains an abnormal storage node, the rack is considered abnormal; if any storage node contains an abnormal hard drive, the storage node is considered abnormal; if any OSD instance corresponding to a hard drive is abnormal, the hard drive is considered abnormal. An OSD instance that meets any of the following conditions is an abnormal OSD instance: the OSD instance is in the first preset state, the OSD instance is in the second preset state, or the OSD instance is in the third preset state. The first preset state indicates that the OSD instance is offline and unreachable, but the OSD instance has the ability to participate in data processing tasks; the second preset state indicates that the OSD instance is offline and unreachable, but does not have the ability to participate in data processing tasks; the third preset state indicates that the OSD instance is performing local data reconstruction and recovery.

6. A hard disk isolation device, characterized in that, This device is used in a storage node of a distributed storage system; the device includes: The isolation module is used to detect whether the hard drive pool to which the hard drive belongs is in a steady state when the pool status does not meet the set full state. A hard drive pool is in a steady state when all the placement groups (PGs) on each hard drive in the hard drive pool are in the same normal working state. If the disk pool is in a steady state, the disk is isolated; if the disk pool is not in a steady state, the disk is isolated according to the fault redundancy type configured for the disk pool and the number of abnormal devices associated with the disk pool; the abnormal devices associated with the disk pool are determined based on the fault redundancy type configured for the disk pool.

7. The apparatus according to claim 6, characterized in that, The isolation of the hard drive based on the fault redundancy type configured for the hard drive pool and the number of faulty devices associated with the hard drive pool includes: If the fault redundancy type configured for the hard disk pool is node redundancy, and the abnormal device associated with the hard disk pool under the node redundancy type is an abnormal storage node that has malfunctioned, then the hard disk is isolated according to the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type. If the fault redundancy type configured for the hard disk pool is rack redundancy, and the abnormal device associated with the hard disk pool under rack redundancy is an abnormal rack that has experienced an anomaly, then the hard disk is isolated based on the number of abnormal racks associated with the hard disk pool and the number of racks that are allowed to be abnormal under rack redundancy.

8. The apparatus according to claim 7, characterized in that, The step of isolating the hard disk based on the number of abnormal storage nodes associated with the hard disk pool and the number of abnormal storage nodes allowed under the node redundancy type includes: Based on the number of abnormal storage nodes associated with the hard disk pool and the number of storage nodes where the hard disks belonging to the hard disk pool are located among all the hard disks to be detected, determine the total number of abnormal nodes corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the first preset policy, then the hard disk will be isolated when the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of storage nodes allowed to be abnormal under the node redundancy type; the first preset policy is the erasure coding EC policy which sets the number of storage nodes allowed to be abnormal. If the data redundancy policy in the hard disk pool is the first preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal nodes corresponding to the hard disk pool is less than or equal to the number of allowed abnormal nodes corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of storage nodes allowed to be abnormal under this node redundancy type, the number of abnormal nodes allowed for any storage pool, and the number of abnormal hard disks allowed for any storage pool are all determined based on the data redundancy strategy of the data in the hard disk pool; if the object storage device OSD instance corresponding to any hard disk is abnormal, it means that the hard disk is abnormal. And / or, The process of isolating the hard drive based on the number of faulty racks associated with the hard drive pool and the number of racks allowed to be faulty under the rack redundancy type includes: Based on the number of abnormal racks associated with the hard disk pool and the number of racks in which the hard disks belonging to the hard disk pool are located among all the hard disks currently being detected, determine the total number of abnormal racks corresponding to the hard disk pool; If the data redundancy policy of the data in the hard disk pool is not the second preset policy, then the hard disk will be isolated when the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of racks allowed to be abnormal under the rack redundancy type; the second preset policy is the EC policy which sets the number of racks allowed to be abnormal. If the data redundancy policy in the hard disk pool is the second preset policy, then for each storage pool into which the hard disk pool is divided, check whether the total number of abnormal racks corresponding to the hard disk pool is less than or equal to the number of allowed abnormal racks corresponding to the storage pool. If yes, then determine that the isolation result for the storage pool is to isolate the hard disk. If no, then when the number of abnormal hard disks in the hard disk pool is less than or equal to the number of allowed abnormal hard disks corresponding to the storage pool, determine that the isolation result for the storage pool is to isolate the hard disk. And when the isolation results for each storage pool into which the hard disk pool is divided all indicate that the hard disk should be isolated, then the hard disk is isolated. The number of racks allowed to be abnormal under this rack redundancy type, the number of racks allowed to be abnormal for any storage pool, and the number of hard drives allowed to be abnormal for any storage pool are all determined based on the data redundancy policy of the hard drive pool; if any OSD instance corresponding to a hard drive is abnormal, it indicates that the hard drive is abnormal. And / or, If any rack contains an abnormal storage node, the rack is considered abnormal; if any storage node contains an abnormal hard drive, the storage node is considered abnormal; if any OSD instance corresponding to a hard drive is abnormal, the hard drive is considered abnormal. An OSD instance that meets any of the following conditions is an abnormal OSD instance: the OSD instance is in the first preset state, the OSD instance is in the second preset state, or the OSD instance is in the third preset state. The first preset state indicates that the OSD instance is offline and unreachable, but the OSD instance has the ability to participate in data processing tasks; the second preset state indicates that the OSD instance is offline and unreachable, but does not have the ability to participate in data processing tasks; the third preset state indicates that the OSD instance is performing local data reconstruction and recovery.

9. An electronic device comprising: processor; as well as A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 5.

Images