Method for handling failure of a magnetic disk and electronic device

Abstract

The application discloses a fault processing method of a magnetic disk and an electronic device, and relates to the technical field of magnetic disk management. Through array overall state analysis, temporary state fluctuations of the magnetic disk in the array caused by power-on and power-off of a cabinet and batch online and offline of the magnetic disk can be recognized. In combination with steady-state review on the magnetic disk, temporary offline of the magnetic disk is recognized, unnecessary storage system performance consumption caused by data reconstruction on the temporarily offline magnetic disk is avoided, and storage performance is improved.

Description

Technical Field

[0001] This application relates to the field of disk management technology, and in particular to disk fault handling methods and electronic devices. Background Technology

[0002] In cloud computing data centers and large-scale storage systems, disk failure is one of the core challenges faced by storage systems. Current technologies determine disk failure based on its operational status and then reconstruct data on the failed disk. However, this method is prone to misjudgment, leading to unintended data reconstruction and consequently, low storage performance. Summary of the Invention

[0003] This application provides a disk fault handling method and electronic device to at least solve the problem of low storage performance in storage systems in related technologies.

[0004] This application provides a disk failure handling method, including:

[0005] The array global state machine reads the operating status of multiple disks corresponding to multiple member disk state machines. The state of multiple member disk state machines is synchronized with the state of multiple disks connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks.

[0006] Based on the operating status of multiple disks, a global analysis is performed using the array global state machine to obtain the initial array operating status.

[0007] The initial array operating state is checked in steady state to obtain the target array operating state.

[0008] Identify the faulty disks based on the target array's operating status and perform data reconstruction on the faulty disks.

[0009] This application also provides a disk fault handling apparatus, comprising:

[0010] The read module is used to read the operating status of multiple disks corresponding to multiple member disk state machines through the array global state machine. The multiple member disk state machines synchronize their status with the multiple disk state machines connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks.

[0011] The analysis module is used to perform global analysis based on the operating status of multiple disks through the array global state machine to obtain the initial array operating status.

[0012] The verification module is used to perform steady-state verification of the initial array operating state to obtain the target array operating state.

[0013] The processing module is used to identify faulty disks based on the target array's operating status and to perform data reconstruction on the faulty disks.

[0014] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the disk fault handling methods described above when executing the computer program.

[0015] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the disk fault handling methods described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the disk fault handling methods described above.

[0017] This application utilizes overall array status analysis to identify temporary fluctuations in disk status caused by rack power-on / off and disk batch online / off. Combined with steady-state verification of disks, it can identify temporary disk offline, avoiding unnecessary storage system performance consumption caused by data reconstruction of temporarily offline disks, thereby improving storage performance. Attached Figure Description

[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an application scenario of a disk fault handling method provided in an embodiment of this application;

[0020] Figure 2 A schematic flowchart illustrating a disk fault handling method provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating another disk fault handling method provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram of the state machine architecture provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram illustrating the fault handling of the task suggestion state machine provided in this application embodiment;

[0024] Figure 6 A schematic diagram of a disk fault handling device provided in an embodiment of this application;

[0025] Figure 7A schematic diagram of another disk fault handling device provided in an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0028] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] For example, a Redundant Array of Independent Disks (RAID), hereinafter referred to as an array, is widely used to improve the reliability and performance of data storage. An array combines multiple physical disks into a logical disk and incorporates data striping and redundancy check techniques to achieve efficient data storage and fault recovery.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The specific application environment architecture upon which the disk failure handling method depends is described here. (References) Figure 1 It enables unified management of multiple physical disks, constructing an array, where the array is a logical storage unit for multiple disks.

[0032] For example, multiple disks on the underlying layer can be accessed by accessing the array to improve data storage reliability and I / O performance.

[0033] In practical applications, disks may experience brief data access failures and be marked as offline due to temporary software malfunctions, excessive input / output pressure, poor cable connections, or power fluctuations. This temporary failure is called a disk outage. If the system immediately initiates data reconstruction for every offline event, it will lead to unnecessary performance overhead and reduced data redundancy.

[0034] In related technologies, storage systems typically employ a simple fault diagnosis mechanism: when a disk reports an OFFLINE status, the system immediately uses a hot spare disk as a replacement and initiates a data reconstruction task for data recovery. This approach has significant drawbacks: firstly, temporary disk failures such as intermittent disconnections can be mistakenly identified as permanent failures, triggering reconstruction and leading to a significant decrease in system performance and increased input / output response latency; secondly, in scenarios where all disks simultaneously become OFFLINE (e.g., due to rack power failure), the varying OFFLINE times of each disk may cause the system to misjudge the failure as affecting only a portion of the disks, initiating unnecessary reconstruction tasks and further compromising data security.

[0035] In complex scenarios such as temporary disk failures and power outages / updates of the server rack, the solutions of related technologies cannot distinguish between permanent failures and transient anomalies, resulting in a large number of unnecessary reconstruction operations being frequently initiated, which seriously affects the performance and reliability of the storage system.

[0036] Figure 2 A flowchart illustrating the disk fault handling method provided in this application embodiment is shown below. Figure 2 As shown, embodiments of this application provide a disk fault handling method, which is described in detail below:

[0037] S201. Read the operating status of multiple disks corresponding to multiple member disk state machines through the array global state machine. The state of multiple member disk state machines is synchronized with the state of multiple disk state machines connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks.

[0038] For example, the array global state machine is a logical component that manages the overall operating state of the array, and each array corresponds to one array global state machine.

[0039] The member disk state machine is a state recording component associated with each member disk in the array, and its state is synchronized with the corresponding disk state machine. Each physical disk operates independently, responsible for recording and updating its real-time operating state, such as online or offline status. When the physical state of a disk changes (e.g., the disk becomes disconnected due to hardware failure), the disk state machine marks its state as offline, and the associated member disk state machines are simultaneously updated to offline status.

[0040] For example, during the array initialization phase, the system creates a corresponding disk state machine and member disk state machine for each member disk connected to the array. The initial state of the disk state machine is ONLINE_UNUSE (online and unused). When a disk is selected as a RAID member disk, the state changes to ONLINE_MEMBER (online member). When a disk fails, the disk state machine changes to OFFLINE_MEMBER (offline member), and the corresponding member disk state machine also synchronously changes to the OFFLINE state.

[0041] Optionally, the array global state machine periodically reads the current state of all member disk state machines via polling or event-driven methods. Each member disk state machine stores a real-time operating status identifier for that disk, such as a binary flag indicating "ONLINE" or "OFFLINE". The array global state machine aggregates this identifier information to form a global view of the operating status of all member disks within the array.

[0042] S202. Based on the operating status of multiple disks, perform global analysis through the array global state machine to obtain the initial array operating status.

[0043] For example, after acquiring the operating states of multiple member disk state machines, the array global state machine evaluates the overall operating state of the array based on preset global analysis rules. The global analysis not only considers the online or offline status of each disk at the current moment, but also combines factors such as the duration and frequency of disk status changes, as well as the array's redundancy configuration, to comprehensively analyze whether the array is in a normal operating state, an abnormal state, or a critical state.

[0044] Optionally, the global analysis process may include: counting the number of member disks currently in OFFLINE state and calculating the comparison between this number and the number of redundant disks in the array. If the number of OFFLINE disks exceeds the number of redundant disks, the array global state machine determines the initial array operating state as offline (array unavailable); if the number of OFFLINE disks is within the allowable range of the number of redundant disks, it is determined to be online (array can still operate normally).

[0045] Optionally, global analysis can further evaluate the temporal characteristics of changes in the status of each disk, such as whether there is a collective anomaly of multiple disks simultaneously going offline, thereby identifying regular failure patterns in scenarios such as rack power outages.

[0046] S203. Perform a steady-state verification on the initial array operating state to obtain the target array operating state.

[0047] For example, the target array operating state is the operating state of the array after correction of the initial array operating state.

[0048] For example, steady-state verification is a step to further verify and confirm the initial array operating state after the global analysis generates it. The initial array operating state may be misjudged as abnormal due to transient anomalies such as disk flashovers. Steady-state verification introduces a time window mechanism to continuously observe the stability of the initial array operating state, thereby distinguishing between permanent faults and transient anomalies.

[0049] S204. Determine the faulty disk based on the target array's operating status and perform data reconstruction on the faulty disk.

[0050] For example, after obtaining the target array's operating status after steady-state verification, the system filters out faulty disks that require data reconstruction based on this status.

[0051] The exemplary process of identifying faulty disks does not simply treat all OFFLINE disks as faulty. Instead, it combines historical fault records, fault mode analysis, and redundancy status to make a comprehensive judgment, thereby reducing the possibility of misjudgment and improving the accuracy of data reconstruction.

[0052] The disk fault handling method provided in this application embodiment reads the operating states of multiple member disk state machines corresponding to multiple disks through the array global state machine. The state machines of multiple member disks are synchronized with the state machines of multiple disks connected to the array, and each disk state machine is used to mark the operating state of its corresponding disk. Based on the operating states of multiple disks, a global analysis is performed through the array global state machine to obtain the initial array operating state. A steady-state verification is performed on the initial array operating state to obtain the target array operating state. The faulty disk is identified based on the target array operating state, and data reconstruction processing is performed on the faulty disk. This solution, through overall array state analysis, can identify temporary state fluctuations of disks in the array caused by rack power-on / off and batch disk online / off. Combined with steady-state verification of the disks, it can identify temporary disk offline, avoiding unnecessary storage system performance consumption caused by data reconstruction of temporarily offline disks, thus improving storage performance.

[0053] Based on any of the above embodiments, the following, in conjunction with Figure 3 This section provides a detailed explanation of the disk failure handling process.

[0054] Figure 3 This is a flowchart illustrating another disk fault handling method provided in an embodiment of this application. Figure 3 As shown, the method includes:

[0055] S301. Read the operating status of multiple disks corresponding to multiple member disk state machines through the array global state machine. The state of multiple member disk state machines is synchronized with the state of multiple disk state machines connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks.

[0056] Below, in conjunction with Figure 4 The state machine architecture is explained.

[0057] Figure 4 This is a schematic diagram of the state machine architecture provided in an embodiment of this application. Figure 4 As shown, each disk corresponds to an independent disk state machine, which synchronously transmits the original operating state of the disk hardware to the bound member disk state machines. All member disk state machines within the array aggregate disk status data and input it into the array global state machine. After completing the array condition analysis and determination, the array global state machine outputs the array operating status, thereby driving the disk fault identification and data reconstruction process.

[0058] S302. Determine multiple durations corresponding to the initial array operating state.

[0059] For example, during the global analysis process, the array global state machine not only reads the current running state of each member disk state machine, but also records the duration of each state.

[0060] Duration refers to the cumulative time that a certain operating state remains unchanged since the last state transition. For example, if the member disk state machine of a certain disk is currently in the OFFLINE state, the system records the time when the OFFLINE state first appears using an internal timestamp and calculates the time difference from that time to the current time in real time, which is the duration of the OFFLINE state.

[0061] For example, each member disk state machine records a timestamp of its state change when it changes state (e.g., from ONLINE to OFFLINE, or from OFFLINE to ONLINE). When polling the state of each member disk, the array's global state machine reads the current state and calculates the difference between it and the timestamp of the state change, thus obtaining the duration of each running state. This duration data is stored in a state statistics table in memory for subsequent global analysis.

[0062] In this feasible implementation, by recording and analyzing the duration of each disk's operating status, time-dimensional data support is provided for judging the stability of the status, which can effectively identify the difference between transient anomalies and persistent failures.

[0063] S303. Based on the initial array operating status and multiple durations, perform a global analysis of the overall timing variation characteristics of the array to obtain the global operating condition of the array.

[0064] For example, overall temporal change characteristic analysis comprehensively evaluates the regularity of state changes of all member disks in the array over time. The system takes the state change event of each member disk as input and, combined with the duration of each state, analyzes whether there are collective state change patterns.

[0065] To illustrate with scenario examples, if multiple member disks switch from ONLINE to OFFLINE almost simultaneously within the same time frame, and the duration of each disk's OFFLINE is similar, it may be a group-wide false anomaly (such as a power outage scenario in the server rack); conversely, if only a few disks are in the OFFLINE state for a long time, it may be a partial disk anomaly (such as a single disk hardware failure).

[0066] Optionally, the time-series change characteristic analysis can employ a sliding window statistical method: the system sets a preset time window (e.g., 2 seconds) and counts the frequency, direction, and number of status change events for all member disks within this window. If the number of OFFLINE events within the window exceeds a preset threshold and the change time points are highly concentrated, it is determined to be a group-wide pseudo-abnormal condition; if the number of OFFLINE events is small and the time points are scattered, it is determined to be a partial disk abnormal condition.

[0067] In this feasible implementation, by analyzing the overall temporal variation characteristics, it is possible to identify abnormal patterns of different natures at the array level.

[0068] S304. Determine the initial array operating status based on the overall operating conditions.

[0069] For example, after obtaining the classification result of the global operating condition, the array global state machine generates the initial array operating state based on the result. If the global operating condition is a group-wide false anomaly, the initial array operating state is determined to be online (i.e., the array as a whole is still considered available) to avoid misjudging the entire array as offline due to a group-wide transient anomaly. If the global operating condition is a partial disk anomaly, the number of OFFLINE disks and the array redundancy capability are further considered to determine whether the initial array operating state is online or offline.

[0070] For example, when the global operating condition is that some disks are abnormal and the number of OFFLINE disks does not exceed the number of redundant disks, the system determines that the initial array operating status is online, and the array can still process input and output requests normally. When the number of OFFLINE disks exceeds the number of redundant disks, the system determines that the initial array operating status is offline, and the array can no longer meet the data redundancy protection requirements, and data reconstruction or entering the security protection mode needs to be performed immediately.

[0071] In this feasible implementation, the initial array operating state is determined based on the classification results of the global operating conditions, which makes the determination of the initial state more refined and avoids misjudgment caused by the time difference of state change, thereby improving storage performance.

[0072] One feasible approach is to perform a global analysis of the overall time-series change characteristics using the following method: determining the first disk to be synchronously offline and its corresponding first number based on the initial array operating state and multiple durations; determining the global operating condition based on the first number and a first number threshold, whereby the global operating condition includes group pseudo-abnormal operating conditions or partial disk abnormal operating conditions.

[0073] For example, synchronous offline refers to multiple member disks switching from ONLINE to OFFLINE at approximately the same time, with similar durations of their OFFLINE states. The system identifies the set of synchronously offline disks by analyzing the difference in the timestamps of the state change (e.g., a difference less than 10 milliseconds) and the variance of the duration of each disk's OFFLINE state (e.g., a variance less than 500 milliseconds). The first disk is one of these synchronously offline disks, and the first number is the total number of synchronously offline disks.

[0074] To illustrate with a scenario example, suppose an array contains 5 disks, of which 4 disks enter OFFLINE state within 50 milliseconds after the rack loses power, and the OFFLINE duration of each disk is more than 3 minutes (until the rack is restored to ONLINE). The system will identify these 4 disks as the first disk to go offline synchronously, and the number of the first disks is 4.

[0075] For example, the size of the first quantity threshold is related to the redundancy configuration of the array. For instance, if the number of redundant disks in the array is 1, then the first quantity threshold can be set to 1. If the first quantity (number of synchronous offline disks) is greater than the first quantity threshold, the overall operating condition is determined to be a group-wide pseudo-abnormal operating condition; if the first quantity is less than or equal to the first quantity threshold, it is determined to be a partial disk abnormal operating condition.

[0076] Optionally, the first quantity threshold can also be dynamically adjusted based on the array's historical operating data. If the array frequently experiences a cluster of false anomalies recently (e.g., more than 3 times within a statistical period), the system can appropriately increase the first quantity threshold to prevent the abnormal behavior of individual disks from being misjudged as a cluster anomaly.

[0077] In this feasible implementation, by identifying disks that are synchronously offline, the scope of clustered false anomalies can be accurately located, thereby reducing false identification of faulty disks and improving storage performance.

[0078] One feasible implementation method is to determine the initial array operating status by: determining the number of abnormal disks from the global operating conditions and determining the number of redundant disks in the array; when the number of abnormal disks is greater than the number of redundant disks, determining the initial array operating status as offline; when the number of abnormal disks is less than or equal to the number of redundant disks, determining the initial array operating status as online.

[0079] For example, the number of abnormal disks refers to the total number of member disks judged to be in an abnormal state in the global operating condition, that is, abnormal disks that are currently in an OFFLINE state and are identified as partially abnormal or in a group of false anomalies. The number of redundant disks is the number of disks in the array used to store verification data.

[0080] For example, when the number of faulty disks exceeds the number of redundant disks, the array is unable to recover data from all the faulty disks using redundancy check data. In this case, the initial array operating state is determined to be offline, indicating that the entire array is unavailable. For instance, if the array has one redundant disk and the number of faulty disks reaches two, the array is determined to be offline, normal input / output operations are stopped, and data reconstruction or data protection processes are triggered.

[0081] For example, when the number of abnormal disks does not exceed the number of redundant disks, the array still has sufficient data redundancy to handle input / output requests normally. For instance, if the array has one redundant disk and the number of abnormal disks is one, the array is determined to be online, and the system can continue to read and write data normally. At the same time, it will decide whether to initiate data reconstruction based on the subsequent steady-state verification results.

[0082] In this feasible implementation, the array remains online when the number of abnormal disks is controllable, avoiding interruption of normal business due to temporary anomalies and ensuring the performance of the storage system.

[0083] S305. If the initial array operating state is abnormal, collect the stable duration of the initial array operating state and start the first timer.

[0084] For example, the first step in the steady-state verification phase is to monitor the stability of the initial array operating state. When the initial array operating state is determined to be abnormal (e.g., offline), the system records the time point when the abnormal state occurs and starts a first timer.

[0085] The stability duration refers to the cumulative time from the first occurrence of the abnormal state to the current moment. The system updates the stability duration every fixed sampling period (e.g., 100 milliseconds).

[0086] Optionally, the first timer is implemented using a software timer, with the timing period configured according to the array's fault detection response time requirements. For example, for a storage system, the first timer period is 2 seconds. After the first timer starts, the system continuously monitors whether the initial array operating state changes, and records the duration of stability.

[0087] Based on the above implementation methods, by introducing a first timer mechanism to monitor the stability of abnormal states, a time buffer window is provided to distinguish between transient false anomalies and permanent faults.

[0088] S306. Based on the stable duration and the first timer, perform a steady-state review of the initial array operating state to determine whether the initial array operating state includes transient pseudo-abnormal conditions.

[0089] For example, the core logic of steady-state verification is to continuously track the dynamic changes of the initial array operating state within the timing period. The initial array operating state value in the array global state machine is read at a fixed frequency (e.g., once every 200 milliseconds), and the changes in state are recorded each time it is read.

[0090] If the initial array operating state does not change within the timing period (i.e., it remains in an abnormal state), the abnormal state is considered stable and is a permanent fault; if the initial array operating state changes within the timing period (e.g., it recovers from an abnormal state to a normal state), the initial abnormal state is considered a transient pseudo-abnormal condition.

[0091] To illustrate with a scenario example, suppose a disk in the array briefly goes offline due to excessive input / output pressure. The array's global state machine initially determines the array's operating state as abnormal (offline). A first timer of 2 seconds is started. After 1.5 seconds of the first timer running, the disk resumes normal access, and the initial array operating state changes back to normal. At this point, the system determines that there was a transient false abnormal condition in the initial array operating state and removes the initial abnormal state.

[0092] In this feasible implementation, by identifying transient false abnormal conditions through steady-state verification, temporary faults such as disk flashovers can be effectively filtered out, avoiding unnecessary reconstruction operations triggered by misjudgment, thereby improving storage performance.

[0093] One feasible implementation method is to determine the transient pseudo-abnormal operating condition by means of the following: during the timing period of the first timer, continuously monitor the initial array operating state and obtain the monitoring result, which is either a change in state or no change in state; if the monitoring result is a change in state, then it is determined that the initial array operating state includes the transient pseudo-abnormal operating condition; if the monitoring result is no change in state, then it is determined that the initial array operating state does not include the transient pseudo-abnormal operating condition.

[0094] For example, the monitoring process adopts a polling or event-driven approach. After the first timer starts, a state change notification callback function is registered. When the initial array running state in the array global state machine changes, the callback function is triggered, and the system records the monitoring result as a state change. If no state change notification is received before the end of the timer cycle, the system records the monitoring result as no state change.

[0095] For example, when monitoring results indicate that the initial array operating state has changed within the timing period (e.g., from an abnormal state to a normal state), the system immediately determines that the initial abnormal state is a transient false abnormal condition and removes it. For instance, if a disk recovers to ONLINE within 100 milliseconds after being in an OFFLINE state, the monitoring result is "state change," and the system determines that the OFFLINE event is a transient false abnormality.

[0096] For example, if the monitoring result remains unchanged after the timing period ends, meaning the initial array operating state remains abnormal, the system determines that this abnormal state does not include transient false abnormal conditions and belongs to a stable permanent fault state. At this time, the system outputs the initial array operating state as the target array operating state and initiates the fault disk identification and data reconstruction process accordingly.

[0097] In this feasible implementation, the stability of the abnormal state is confirmed by monitoring the unchanged state in the results, ensuring that data reconstruction is only performed after a real disk failure is confirmed, thus avoiding erroneous operations and improving storage performance.

[0098] S307. If so, then eliminate the transient false abnormal conditions to obtain the target array operating status.

[0099] For example, if the steady-state check determines that there is a transient false abnormality in the initial array operating state, the system will correct the initial array operating state from the abnormal state to the normal state. The target array operating state is the stable state after removing the transient abnormality. For example, if the initial array operating state is "offline", and the steady-state check finds that the state recovers to "online" within the timing period, then the target array operating state is "online".

[0100] In this feasible implementation, by eliminating transient false abnormal conditions, the final target array operating state truly reflects the actual operating status of the array, thereby improving storage performance.

[0101] S308. Determine the faulty disk based on the target array's operating status and perform data reconstruction on the faulty disk.

[0102] One feasible approach is to identify faulty disks by: determining multiple initial abnormal disks based on the target array's operating status; determining fault judgment criteria for each of the multiple initial abnormal disks; and filtering the multiple initial abnormal disks according to the fault judgment criteria to obtain the faulty disks.

[0103] For example, after the target array's operating status is determined, the system filters out member disks that are currently in an abnormal state based on that status. Initial abnormal disks refer to member disks in the target array's operating status that are marked as OFFLINE or abnormal. For instance, if the target array is operating online, but two of its member disks are still in an OFFLINE state, then these two member disks are identified as initial abnormal disks.

[0104] For example, fault judgment criteria are used to assess whether each initially anomalous disk is a truly faulty disk that requires data reconstruction. The fault judgment criteria for each initially anomalous disk may differ because the historical operating conditions and failure modes of different disks vary. The system performs statistical analysis based on the historical anomaly data of each disk (including anomaly duration, anomaly recovery interval, and other indicators) to determine the fault judgment threshold for each disk.

[0105] For example, the screening process involves comparing the current status data of each initially faulty disk with its corresponding fault judgment criteria.

[0106] With the example of a scenario, if the current state characteristics (such as the duration of OFFLINE) of an initially abnormal disk exceed the threshold in its fault judgment criteria, the disk is judged as a faulty disk; if it does not exceed the threshold, it is judged as a non-faulty disk (which may belong to an abnormal disk that still needs further observation).

[0107] In this feasible implementation, a screening mechanism based on personalized fault judgment criteria can accurately identify faulty disks that truly require data reconstruction, avoiding the misclassification of all abnormal disks as faulty disks, thereby improving storage performance.

[0108] One feasible approach is to determine the fault judgment criteria by: identifying multiple historical anomaly data points for multiple initially abnormal disks, including the duration of the anomaly and the recovery interval; and performing statistical analysis on the multiple historical anomaly data points to obtain the fault judgment criteria corresponding to each of the multiple initially abnormal disks.

[0109] For example, historical anomaly data reflects the regularity of abnormal disk behavior over a period of time. Anomaly duration refers to the duration of each time the disk enters an OFFLINE state; anomaly recovery interval refers to the time interval between when the disk recovers from an OFFLINE state to an ONLINE state and when it enters an OFFLINE state again. The system can collect historical anomaly records from the disk over the past 24 hours or 7 days and construct an anomaly data statistics table.

[0110] Optionally, statistical analysis can use statistical indicators such as mean, standard deviation, and percentiles. For example, for a certain disk, if the average duration of historical anomalies is 1.2 seconds and the standard deviation is 0.5 seconds, the fault judgment threshold can be set to "the current duration of anomalies exceeds the average plus twice the standard deviation," which is 2.2 seconds. If the current duration of anomalies exceeds 2.2 seconds, the disk is determined to be a faulty disk.

[0111] In this feasible implementation, personalized fault judgment criteria are obtained by statistically analyzing historical abnormal data, making fault judgment more accurate and avoiding the limitations of fixed thresholds, thereby improving the accuracy of disk screening and thus improving storage performance.

[0112] One feasible implementation method involves data reconstruction processing as follows: disconnecting the data interaction link between the faulty disk and the array; using the verification data and stripe data stored on the redundant disk, performing step-by-step verification and restoration of the missing data on the faulty disk to obtain restored data; migrating the restored data back to the array's hot spare disk line by line, while simultaneously collecting real-time load data, including processor utilization and input / output request queue length; and adjusting the data reconstruction scheduling sequence based on the real-time load data until the restored data migration is complete.

[0113] For example, all input / output operations on the failed disk can be stopped via software commands, and the disk can be removed from the array's data interaction link to prevent the failed disk from continuing to participate in data read / write operations and causing data inconsistency.

[0114] The system reads parity data and stripe data stored on other normal disks in the array, and uses the XOR parity algorithm to recover the data on the failed disk block by block. For example, if a physical disk in the array fails, the system reads data from other data disks and parity disks in the same stripe, and recovers the data blocks on the failed disk through XOR calculation.

[0115] For example, when writing the restored data to the hot spare disk, the system adopts a line-by-line migration method: after each stripe of data is restored, it is immediately written to the hot spare disk, instead of waiting for all data to be restored before writing it all at once.

[0116] During the data migration process, the system collects load data in real time, such as CPU utilization and I / O request queue length. CPU utilization reflects the current computing resource usage of the system, while I / O request queue length reflects the I / O pressure on the storage system.

[0117] For example, scheduling timing adjustment refers to dynamically adjusting the execution strategy of data reconstruction tasks based on real-time load data. If CPU utilization exceeds a preset threshold (e.g., 80%) or the input / output queue length exceeds a preset threshold (e.g., 100 requests), the system can reduce the priority of reconstruction tasks to reduce the impact of reconstruction operations on normal input / output performance; when the system load decreases, the execution frequency of reconstruction tasks can be increased again.

[0118] Optionally, adjustments may include: modifying the thread priority of the reconstruction task, adjusting the amount of data recovered each time, and setting the execution interval of the reconstruction operation.

[0119] In this feasible implementation, by dynamically adjusting the reconstruction sequence according to the real-time load, the performance impact of reconstruction operations on normal system operations is minimized while ensuring data security, thereby improving storage performance.

[0120] One feasible implementation method for disk fault handling may further include: receiving state change information reported by the disk state machine; when a disk switches to an offline state, reading the array global state and the number of redundant disks; if the array global state is offline or the number of redundant disks is less than a second threshold, terminating the reconstruction task scheduling operation; if the array global state is online and the number of redundant disks is greater than or equal to the second threshold, starting a second timer; after the second timer completes, verifying that the corresponding disk is still offline and there is no matching reconstruction task for the disk, and issuing the corresponding reconstruction task; when a disk switches to an online state, reading the number of redundant disks and the array global state change record; if the number of redundant disks is sufficient, the array global state switches from offline to online, and there are offline disks in the array, then starting a third timer; after the third timer completes, verifying that the corresponding disk is still offline and there is no matching reconstruction task for the disk, and issuing the corresponding reconstruction task; within the timer period of the third timer, if each disk switches to an online state, or an offline disk has been bound to a corresponding reconstruction task, canceling the current timer.

[0121] For example, when the member disk state machine changes to the OFFLINE state, the event is simultaneously reported to the task suggestion state machine.

[0122] Below, in conjunction with Figure 5 Explain the handling of faults in the task suggestion state machine.

[0123] Figure 5 This is a schematic diagram illustrating the fault handling of the task suggestion state machine provided in an embodiment of this application. Figure 5 As shown, the task suggestion state machine starts a timer (e.g., 1 second) when certain conditions are met, based on the current array global state machine and the number of redundant disks. If the member disk is still in the OFFLINE state and there is no corresponding reconstruction task after the timer expires, a reconstruction task is issued. Similarly, when the member disk state machine changes to the ONLINE state, another timer (e.g., 5 seconds) is started for delay judgment based on whether the array global state machine has changed from OFFLINE to ONLINE and whether there are still OFFLINE member disks.

[0124] For example, the core of this timer mechanism is that the timer of the array global state machine is shorter than the timer of the task suggestion state machine (e.g., 2 seconds for RAID state verification and 1 second for RAID task verification), ensuring that RAID state changes can be verified before the task is issued, thereby avoiding the mistaken issuance of reconstruction tasks when the RAID state is not yet stable.

[0125] Based on the above implementation methods, by introducing a timer mechanism through a task suggestion state machine to perform secondary review of the refactoring task issuance, it is possible to effectively suppress the erroneous start of the refactoring task caused by transient changes in disk status, thereby improving storage performance.

[0126] Figure 6 This is a schematic diagram of a disk fault handling device provided in an embodiment of this application. Figure 6 As shown, the disk failure handling device 60 may include: a reading module 61, an analysis module 62, a verification module 63, and a processing module 64.

[0127] The reading module 61 is used to read the operating status of multiple disks corresponding to multiple member disk state machines through the array global state machine. The multiple member disk state machines synchronize their status with the multiple disk state machines connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks.

[0128] Analysis module 62 is used to perform global analysis based on the operating status of multiple disks through the array global state machine to obtain the initial array operating status.

[0129] The verification module 63 is used to perform steady-state verification on the initial array operating state to obtain the target array operating state.

[0130] The processing module 64 is used to determine the faulty disk based on the operating status of the target array and to perform data reconstruction processing on the faulty disk.

[0131] Optionally, the reading module 61 can execute... Figure 2 S201 in the embodiment.

[0132] Optionally, analysis module 62 can execute... Figure 2 S202 in the embodiment.

[0133] Optionally, the review module 63 can execute... Figure 2 S203 in the embodiment.

[0134] Optionally, processing module 64 can execute Figure 2 S204 in the embodiment.

[0135] In one possible implementation, the analysis module 62 is specifically used for:

[0136] Determine multiple durations corresponding to the initial array operating state.

[0137] Based on the initial array operating state and multiple durations, a global analysis of the overall timing variation characteristics of the array is performed to obtain the global operating condition of the array.

[0138] Determine the initial array operating status based on the overall operating conditions.

[0139] In one possible implementation, the analysis module 62 is specifically used for:

[0140] Based on the initial array operating status and multiple durations, determine the first disk to be synchronized offline and its corresponding first number.

[0141] Based on the first quantity and the first quantity threshold, the global operating condition is determined. The global operating condition includes group pseudo-abnormal operating conditions or partial disk abnormal operating conditions.

[0142] In one possible implementation, the analysis module 62 is specifically used for:

[0143] Determine the number of abnormal disks from the overall operating conditions, and determine the number of redundant disks in the array.

[0144] When the number of abnormal disks exceeds the number of redundant disks, the initial array operating status will be determined as offline.

[0145] When the number of abnormal disks is less than or equal to the number of redundant disks, the initial array operating status is determined to be online.

[0146] In one possible implementation, the review module 63 is specifically used for:

[0147] If the initial array is in an abnormal state, collect the stable duration of the initial array operation and start a timer.

[0148] Based on the stable duration and timer, a steady-state review of the initial array operating state is performed to determine whether the initial array operating state includes transient pseudo-abnormal conditions.

[0149] If so, then the transient false abnormal conditions are eliminated to obtain the target array's operating status.

[0150] In one possible implementation, the review module 63 is specifically used for:

[0151] During the timing period of the first timer, the initial array operating status is continuously monitored to obtain the monitoring results, which are either a change in status or no change in status.

[0152] If the monitoring result shows a change in state, it is determined that the initial array operating state includes transient pseudo-abnormal conditions.

[0153] If the monitoring result shows no change in status, it is determined that the initial array operating status does not include transient pseudo-abnormal conditions.

[0154] In one possible implementation, the processing module 64 is specifically used for:

[0155] Based on the target array's operating status, multiple initial abnormal disks were identified.

[0156] Determine the fault judgment criteria corresponding to each of the multiple initially abnormal disks.

[0157] Based on the fault judgment criteria corresponding to the multiple initial abnormal disks, the multiple initial abnormal disks are filtered to obtain the faulty disks.

[0158] Identify multiple historical anomaly data points for multiple initially abnormal disks. These historical anomaly data points include the duration of the anomaly and the interval between anomalies.

[0159] In one possible implementation, the processing module 64 is specifically used for:

[0160] The data interaction link between the faulty disk and the array is disconnected. The missing data on the faulty disk is then verified and restored step by step using the verification data and stripe data stored on the redundant disks to obtain the restored data.

[0161] The restored data is migrated back to the array hot spare disk one by one, while real-time load data is collected at the same time, including processor utilization and input / output request queue length.

[0162] Adjust the scheduling sequence of data reconstruction based on real-time load data until the data migration is completed.

[0163] It should be noted that the disk failure handling device shown in the embodiments of this application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so they will not be described again here.

[0164] Figure 7 This is a schematic diagram of another disk fault handling device provided in an embodiment of this application. Figure 6 Based on the illustrated embodiments, as Figure 7 As shown, the disk failure handling device 60 also includes an execution module 65.

[0165] Execution module 65 is used for:

[0166] Receive status change information reported by the disk state machine.

[0167] When a disk is switched to offline status, read the global status of the array and the number of redundant disks.

[0168] If the array is in an offline state or the number of redundant disks is less than the second threshold, the reconfiguration task scheduling operation will be terminated.

[0169] If the array is in an online state and the number of redundant disks is greater than or equal to the second quantity threshold, start the second timer.

[0170] After the second timer completes, if the corresponding disk is still offline and there is no matching disk reconstruction task, the corresponding reconstruction task will be issued.

[0171] When a disk is switched to online status, read the number of redundant disks and the global status change record of the array.

[0172] If there are enough redundant disks, the array's global status switches from offline to online, and there are offline disks in the array, then the third timer is started.

[0173] After the third timer completes, if the corresponding disk is still offline and there is no matching reconstruction task for that disk, the corresponding reconstruction task will be issued.

[0174] If any disk switches to online status or an offline disk is already bound to a corresponding reconstruction task during the third timer's timing period, the current timer is canceled.

[0175] For a description of the features in the embodiment of the disk fault handling device, please refer to the relevant description of the embodiment of the disk fault handling method, which will not be repeated here.

[0176] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the electronic device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus.

[0177] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to execute the above-described disk fault handling method embodiment.

[0178] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0179] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0180] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0181] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0182] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the disk fault handling method embodiments described above when running.

[0183] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0184] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the disk fault handling method embodiments described above.

[0185] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the disk fault handling method embodiments described above.

[0186] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0187] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0188] The above provides a detailed description of a disk fault handling method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for handling disk failures, characterized in that, include: The array global state machine reads the operating status of multiple disks corresponding to multiple member disk state machines. The state of the multiple member disk state machines is synchronized with the state of multiple disk state machines connected to the array. The multiple disk state machines are used to mark the operating status of the corresponding disks. Based on the operating states of the multiple disks, a global analysis is performed using the array global state machine to obtain the initial array operating state; The initial array operating state is checked for steady-state performance to obtain the target array operating state. The faulty disk is identified based on the target array's operating status, and data reconstruction is performed on the faulty disk. The step of obtaining the initial array operating state by performing global analysis through the array global state machine based on the operating states of the multiple disks includes: determining the duration of each of the multiple disk operating states; determining the first disk to be synchronously offline and its corresponding first number based on the duration of each of the multiple disk operating states; determining the global operating condition of the array based on the first number and a first number threshold, wherein the global operating condition includes a group of false abnormal operating conditions or a partial disk abnormal operating condition; determining the number of abnormal disks from the global operating condition, and determining the number of redundant disks of the array; determining the initial array operating state as offline when the number of abnormal disks is greater than the number of redundant disks; and determining the initial array operating state as online when the number of abnormal disks is less than or equal to the number of redundant disks.

2. The disk fault handling method according to claim 1, characterized in that, The initial array operating state is subjected to steady-state verification to obtain the target array operating state, including: If the initial array operating state is abnormal, the stable duration of the initial array operating state is collected and the first timer is started; Based on the stable duration and the first timer, a steady-state review is performed on the initial array operating state to determine whether the initial array operating state includes transient pseudo-abnormal conditions. If so, the transient false abnormal conditions are eliminated to obtain the target array operating state.

3. The disk fault handling method according to claim 2, characterized in that, Based on the stable duration and the first timer, a steady-state review is performed on the initial array operating state to determine whether the initial array operating state includes transient pseudo-abnormal conditions, including: During the timing period of the first timer, the operating status of the initial array is continuously monitored to obtain a monitoring result, which is either a change in status or no change in status. If the monitoring result is a state change, then the initial array operating state is determined to include the instantaneous pseudo-abnormal condition; If the monitoring result is that the state has not changed, then it is determined that the initial array operating state does not include the instantaneous false abnormal condition.

4. The disk fault handling method according to claim 1, characterized in that, Determining the faulty disk based on the target array's operating status includes: Based on the target array's operating status, multiple initial abnormal disks were identified; Determine multiple historical anomaly data of the multiple initially abnormal disks, the multiple historical anomaly data including the duration of the anomaly and the interval of the anomaly recovery; Statistical analysis is performed on the multiple historical abnormal data to obtain the fault judgment criteria corresponding to the multiple initial abnormal disks; Based on the fault judgment criteria corresponding to the multiple initial abnormal disks, the multiple initial abnormal disks are filtered to obtain the faulty disks.

5. The disk fault handling method according to claim 1, characterized in that, Data reconstruction processing of the faulty disk includes: The data interaction link between the faulty disk and the array is disconnected. The missing data on the faulty disk is then verified and restored step by step using the verification data and stripe data stored on the redundant disk to obtain the restored data. The restored data is migrated back to the array hot spare disk one by one, and real-time load data is collected at the same time. The real-time load data includes processor utilization and input / output request queue length. The scheduling sequence of data reconstruction is adjusted according to the real-time load data until the data migration is completed.

6. The disk fault handling method according to any one of claims 1-5, characterized in that, The method further includes: Receive status change information reported by the disk status machine; When a disk is switched to offline status, read the global status of the array and the number of redundant disks. If the array global status is offline or the number of redundant disks is less than the second threshold, terminate the reconstruction task scheduling operation. If the array is in an online state and the number of redundant disks is greater than or equal to the second threshold, start the second timer; After the second timer completes, if the corresponding disk is still offline and there is no matching reconstruction task for the disk, the corresponding reconstruction task is issued. When the disk is switched to online status, read the number of redundant disks and the global status change record of the array. If the number of redundant disks is sufficient, the global state of the array switches from offline to online, and there are offline disks in the array, then the third timer is started; After the third timer completes, if the corresponding disk is still offline and there is no matching reconstruction task for that disk, the corresponding reconstruction task will be issued. If any disk switches to online status or an offline disk is already bound to a corresponding reconstruction task during the third timer's timing period, the current timer is canceled.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the disk fault handling method as described in any one of claims 1 to 6 when executing the computer program.

Images