Fault processing method, controller, device, medium and product of storage device

Abstract

The application discloses a fault processing method, a controller, equipment, a medium and a product of a storage device, relates to the technical field of a storage system, and comprises the following steps: detecting, by a first baseboard management controller of a first controller, whether an alarm signal is sent by a second controller, the alarm signal corresponding to a second processor of the second controller being down, so that accurate identification of a fault is realized; after receiving the alarm signal, the first controller stops communicating with the second controller through a non-transparent bridge link, so that the fault is prevented from spreading from a single controller to the whole machine; the first baseboard management controller of the first controller simultaneously detects whether a first processor of the first controller is down, so that self-fault detection is realized; when it is detected that the first processor is down, the first baseboard management controller resets the first controller itself instead of resetting the whole machine, so that this repairing mode for only the fault controller does not affect the normal operation of other services, the service interruption range is reduced, and accurate isolation and control of the fault are realized.

Description

Technical Field

[0001] This application relates to the field of storage system technology, and in particular to a fault handling method, controller, device, medium and product for a storage device. Background Technology

[0002] To meet the high reliability and continuity requirements of storage products, dual-controller redundant systems use non-transparent bridge (NTB) links as the core to achieve functions such as data synchronization and fault switching. They are widely used in distributed storage and other scenarios, and the reliability of the NTB link determines the availability of the system.

[0003] The relevant system adopts an architecture of NTB main link and serial redundant link. Under normal circumstances, the NTB link is responsible for core interaction, and in the event of a failure, the serial port takes over heartbeat detection and triggers a restart of the peer. However, this solution has significant shortcomings: First, after the NTB link fails, manual or whole-system reset is required for recovery; second, the risk of fault propagation is high, and the failure of a single controller may cause the central processing unit (CPU) of the peer controller to fail as well, resulting in a complete interruption of services and affecting the service continuity of the storage system. Summary of the Invention

[0004] This application provides a fault handling method, controller, device, medium, and product for storage devices, to at least solve the problem in related technologies where the risk of fault propagation is high and the entire device must be reset for recovery when the NTB link fails.

[0005] This application provides a fault handling method for a storage device, comprising: a first controller applied to the storage device, the storage device including a first controller and a second controller, the first controller and the second controller communicating via a non-transparent bridge link; the first controller including a first processor and a first substrate management controller; the method comprising: detecting whether an alarm signal sent by the second controller is received by the first substrate management controller, the alarm signal indicating that the second processor of the second controller has crashed; stopping communication with the second controller via the non-transparent bridge link when the alarm signal is received; detecting whether the first processor has crashed by the first substrate management controller; and resetting the first controller by the first substrate management controller when the first processor has crashed.

[0006] This application provides another fault handling method for a storage device, comprising: a second controller applied to the storage device, the storage device including a first controller and a second controller, the first controller and the second controller communicating via a non-transparent bridge link; the second controller including a second processor and a second baseboard management controller; the method comprising: detecting whether the second processor has crashed via the second baseboard management controller; in the event of a crash of the second processor, sending an alarm signal to the first controller via the second baseboard management controller to cause the first controller to stop communicating with the second controller via the non-transparent bridge link; and resetting the second controller via the second baseboard management controller.

[0007] This application also provides a controller, including: a first processor and a first baseboard management controller; wherein, the first baseboard management controller is configured to detect whether an alarm signal is received from a second controller, the alarm signal indicating that the second processor of the second controller has crashed; upon receiving an alarm signal from the second controller, stop communicating with the second controller via a non-transparent bridge link; detect whether the first processor has crashed; and reset the controller if the first processor has crashed.

[0008] This application also provides another controller, including: a second processor and a second baseboard management controller; wherein the second baseboard management controller is used to detect whether the second processor has crashed; in the event of a crash in the second processor, it sends an alarm signal to the first controller so that the first controller stops communicating with the controller through the non-transparent bridge link; and resets the controller.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the fault handling method of the memory device when executing the computer program.

[0010] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault handling method of the above-described storage device.

[0011] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described fault handling method for the storage device.

[0012] This application uses a first baseboard management controller of the first controller to detect whether the second controller sends an alarm signal. This alarm signal corresponds to the second processor of the second controller crashing, thus achieving accurate fault identification. Upon receiving the alarm signal, the first controller stops communicating with the second controller through the non-transparent bridge link, preventing the fault from spreading from a single controller to the entire machine. The first baseboard management controller of the first controller also detects whether its own first processor has crashed, thus achieving self-fault detection. When the first processor crash is detected, the first baseboard management controller resets the first controller itself, rather than resetting the entire machine. This method of repairing only the faulty controller does not affect the normal operation of other services, reduces the scope of service interruption, and achieves accurate fault isolation and control. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A schematic diagram of the specific hardware architecture on which the execution of a fault handling method for a storage device provided in this application depends;

[0015] Figure 2 A flowchart illustrating a fault handling method for a storage device provided in an embodiment of this application;

[0016] Figure 3 A flowchart illustrating another fault handling method for a storage device provided in an embodiment of this application;

[0017] Figure 4 A flowchart illustrating another method for handling faults in a storage device provided in this application embodiment;

[0018] Figure 5A An interactive schematic diagram illustrating a fault handling method for a storage device provided in an embodiment of this application;

[0019] Figure 5B An interactive schematic diagram illustrating another fault handling method for a storage device provided in an embodiment of this application;

[0020] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To more clearly illustrate the embodiments of this application, the technical terms used in the embodiments will be briefly introduced below:

[0024] The Central Processing Unit (CPU) is the core of a computer system for computation and control.

[0025] A Baseboard Management Controller (BMC) is an embedded dedicated processor primarily used for hardware status management and remote control of servers or high-end computers, and it operates independently of the host CPU and operating system.

[0026] The catastrophic error (CATERR) signal is used to indicate that a catastrophic error has occurred in the system. The processor activates the CATERR signal when it detects an unrecoverable machine check error or other unrecoverable internal error.

[0027] A non-transparent bridge (NTB) is a hardware bridging technology used to connect two or more independent Peripheral Component Interconnect Express (PCIe) domains. Its core feature is that it hides the address space and configuration information of the connected PCIe domains from each other, so as to achieve physically isolated but logically communicable device interconnection.

[0028] To meet the high reliability and continuity requirements of storage products, dual-controller redundant systems have become the mainstream architecture. The core of this architecture relies on the NTB link between the two controllers to realize key functions, including data synchronization of controller devices, heartbeat signal detection, metadata interaction, and fault switching. This architecture is widely used in distributed storage, high-performance computing, disaster recovery and backup scenarios. Therefore, the reliability and fault recovery capability of the NTB link directly determine the overall availability of the system.

[0029] In related dual-controller redundant systems, the controllers typically employ a design with an NTB primary link and a serial port redundant link. Under normal operating conditions, the NTB link handles core data exchange and status detection. When the NTB link fails, the heartbeat detection function switches to the serial port link, which continuously monitors the working status of the peer motherboard. If the peer motherboard is found to be unresponsive for an extended period, it is determined to be a peer failure, and a peer restart is triggered via the serial port link to achieve system self-repair.

[0030] However, in NTB link failure scenarios, traditional handling methods require manual intervention or a complete system reset to restore the link. This not only leads to excessively long recovery times but also interrupts other normal services of the storage device. Furthermore, it cannot perform individual resets of the CPU or NTB portion of the link, lacking refined local fault repair capabilities. When a single controller's CPU fails, the peer controller continues to send NTB messages to the failed controller via the NTB link. Because the failed controller is down and cannot return response messages, the peer controller cannot receive error information, thus triggering the watchdog mechanism of its own CPU's NTB module. Ultimately, this causes the peer controller's CPU to also fail, and the fault spreads from a single controller to the entire system, causing a complete service interruption and severely impacting the service continuity of the storage system. When a single controller experiences a CPU failure, the backup controller cannot effectively distinguish the fault type. That is, it cannot determine whether the communication interruption is caused by a fault in the NTB link itself or by a failure of the peer CPU, easily triggering unnecessary peer reset operations. The root cause lies in the lack of a refined detection mechanism and tiered recovery strategy for the NTB link status. When a single controller CPU fails, the peer controller will continue to send NTB messages to the failed controller via the NTB link. Since the failed controller has failed and cannot return response messages, the peer controller cannot receive error information, which triggers the watchdog mechanism of the NTB module in its own CPU. Ultimately, this causes the peer controller CPU to also fail, and the fault spreads from a single controller to the entire machine, causing a complete interruption of the entire machine's services and seriously affecting the service continuity of the storage system.

[0031] To address all or part of the aforementioned technical problems, this application provides a fault handling method for storage devices. This method utilizes a first baseboard management controller to detect processor failure alarm signals from a second controller. Upon confirming a processor failure on the other end, the first controller proactively stops communication via the non-transparent bridge link, preventing a single controller failure from propagating to the entire system and ensuring the continuity of core storage system services. By leveraging the baseboard management controller's detection of local processor failure status, the first controller is only reset upon confirming its own processor failure, avoiding the interference of traditional system-wide resets on normal services, reducing the impact of fault repair, and improving system availability. Through hardware-level alarm signal transmission and independent detection logic of the baseboard management controller, the accuracy and timeliness of fault diagnosis are enhanced, reducing invalid operations caused by misjudgments of fault types. This provides a more reliable fault isolation and self-healing mechanism for dual-controller systems, comprehensively improving the operational stability and service quality of the storage device.

[0032] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 The diagram shows a schematic of the specific hardware architecture upon which the fault handling method for the storage device depends.

[0034] The first controller includes a first processor and a first substrate management controller; the first processor is connected to a second processor of the second controller via a non-transparent bridge (NTB) link, and the first and second controllers perform data synchronization and backup via the NTB link. The second controller includes a second processor and a second substrate management controller.

[0035] The first substrate management controller receives the catastrophic error CATERR signal transmitted by the first processor through a general-purpose input / output interface (GPIO), as shown in the figure. It transmits the interrupt INT signal to the first processor through GPIO1. It is connected to the second substrate management controller of the second controller through GPIO3, and to the second substrate management controller of the second controller through GPIO4.

[0036] The second substrate management controller receives the catastrophic error CATERR signal transmitted by the second processor via GPIO1; transmits the interrupt INT signal to the second processor via GPIO2; connects to the GPIO4 of the first substrate management controller of the second controller via GPIO3, and connects to the GPIO3 of the first substrate management controller of the first controller via GPIO4.

[0037] Taking the first controller as an example, if the first processor fails, it transmits a catastrophic error CATERR signal to the first baseboard management controller. The first baseboard management controller receives the CATERR signal transmitted by the first processor through GPIO1, triggers an interrupt INT signal in response to the CATERR signal, and transmits it to the first processor through GPIO2. In response to the interrupt signal, the first processor stops communicating with the second processor through the NTB link.

[0038] Simultaneously, in response to the CATERR signal, the first substrate management processor sends an alarm signal to the second substrate management controller via GPIO4. In response to the alarm signal, the second substrate management controller triggers an interrupt INT signal, which is transmitted to the second processor. In response to the interrupt signal, the second processor stops communicating with the first processor via the NTB link.

[0039] Embodiments of this application also provide a fault handling method for a storage device. This method is applied to a first controller of the storage device, the first controller including a first processor and a first baseboard management controller (BMC). The first controller and a second controller of the storage device communicate via a non-transparent bridge (NTB) link.

[0040] like Figure 2 As shown, the method includes the following steps:

[0041] S200: The first board management controller detects whether it receives an alarm signal sent by the second controller.

[0042] The first BMC is connected to the second controller via GPIO, such as Figure 1 As shown, GPIO3 of the first BMC is connected to GPIO4 of the second BMC. The first BMC detects whether it receives an alarm signal sent by the second BMC through GPIO4.

[0043] On the second controller side, the second BMC detects whether it receives a catastrophic error (CATERR) signal transmitted by the second processor. If the second BMC receives a CATERR signal, it indicates that the second processor has suffered a serious failure. The second BMC then sends an alarm signal to the first BMC of the first controller to inform the first BMC that the second processor has crashed.

[0044] S201. Upon receiving an alarm signal from the second controller, stop communicating with the second controller via the non-transparent bridge link.

[0045] In the storage device, when the first controller receives a catastrophic error CATERR signal from the second controller, it indicates that the second processor of the second controller has suffered a serious failure, and the first controller stops communicating with the second controller through the NTB link.

[0046] In some embodiments, stopping the first controller from communicating with the second controller via the NTB link includes: the first controller sending an interrupt INT signal to the first processor via the first BMC; and the first processor responding to the interrupt INT signal by stopping the sending of non-transparent bridge NTB messages to the second controller via the NTB link. This avoids the first controller being affected by the lack of response from the second controller, which could lead to a system crash of the local processor.

[0047] When the first BMC triggers an interrupt, it sends an interrupt signal to the first processor. The interrupt controller inside the first processor monitors this interrupt signal in real time. It first obtains the current operating status of the NTB controller to confirm whether the message transmission queue is active. Then, it suspends the transmission of new messages by writing a disable instruction to the transmit enable register of the NTB controller. At the same time, it iterates through and clears the message data in the NTB transmit buffer that has not yet been sent to avoid residual messages causing misjudgment by the second controller during the subsequent recovery process. Finally, it returns a feedback signal indicating that the interrupt handling is complete to the first BMC, ensuring that the interrupt process is closed.

[0048] When the second BMC detects that the second processor has crashed (catastrophic error) and sends an alarm signal to the first BMC, the first BMC immediately sends an interrupt signal to the first processor, instructing it to stop sending NTB messages to the first processor through the NTB link, thus cutting off the behavior of the other end continuously sending invalid messages from the source. Since the first processor actively stops sending messages, it no longer triggers the watchdog timeout mechanism of its own CPU's NTB module due to no response to messages, completely blocking the fault propagation chain of the single controller CPU crash leading to the other end CPU crash, preventing the fault from spreading from the single controller to the whole machine, and ensuring the overall business continuity of the storage system.

[0049] S202, The first baseboard management controller detects whether the first processor has crashed.

[0050] The first controller monitors the first processor via the first BMC to detect whether the first processor has crashed. If the first BMC receives a CATERR signal from the first processor, it indicates that the first processor has crashed.

[0051] S203. In the event of a first processor failure, the first controller is reset via the first baseboard management controller.

[0052] In the event of a first processor crash, the first controller performs a power-down and power-on operation via the first BMC to reset the first controller and correct the first processor crash error.

[0053] In summary, the fault handling method for a storage device provided in this application, through the detection of alarm signals from the second controller by the first baseboard management controller, can accurately identify the downtime state of the peer processor and promptly stop communication via the NTB link. This solves the problem of faults spreading from a single controller to the entire device in the traditional mode, ensuring the continuity of core services of the storage system. By detecting whether the local processor is down using the first baseboard management controller, and triggering a reset of the first controller when the local processor fails, the interruption of normal services caused by traditional whole-device reset is avoided, improving the targeted nature of fault repair and reducing the impact on the normal operation of the storage device.

[0054] like Figure 3 As shown, the fault handling method for a storage device provided in this application further includes, after performing step S200:

[0055] S301. If no alarm signal is received from the second controller, the first baseboard management controller detects whether the first processor has crashed.

[0056] If the first BMC of the first controller does not receive an alarm signal sent by the second BMC of the second controller via GPIO, it indicates that the second processor has not crashed and is operating normally, and then directly executes step S202 (detecting whether the first processor has crashed through the first BMC).

[0057] S302. If the first processor does not crash, obtain the link health status of the non-transparent bridge link through the first processor.

[0058] The first controller did not receive an alarm signal from the second controller, indicating that the second controller was operating normally. At this time, the first controller also detected that the first processor had not crashed, indicating that both controllers of the storage device were operating normally. The first processor, which was operating normally, was then used to detect whether the NTB was faulty.

[0059] In some embodiments, obtaining the link health status of the NTB link through the first processor includes: reading the high-speed peripheral component interconnect (PCIe) information of the NTB link through the first processor, and determining the link health status based on the PCIe information.

[0060] Specifically, the first processor accesses the PCIe configuration space mapped inside the NTB chip to read key register information related to the link status, including but not limited to the Link Status Register, Link Control Register, and Error Status Register. The Link Status Register is used to obtain the current link's operating status (e.g., whether it is in normal L0 operation, or in non-L0 states such as Detect or Recovery), the actual negotiated transmission rate (e.g., PCIe 3.0 / 4.0), and the link width (e.g., x4 / x8 channel count). The Link Control Register provides configuration information such as link training trigger signals and bandwidth adjustment status. The Error Status Register records whether abnormal events such as Cyclic Redundancy Check (CRC) errors, parity errors, and data link layer protocol errors have occurred on the link.

[0061] The first processor periodically (e.g., every 10ms) collects the aforementioned register data and performs logical analysis based on preset health status judgment rules: if the link status register shows L0 status, the transmission rate and bandwidth meet the preset configuration, and the error status register shows no continuously increasing error count, then the NTB link is determined to be in a healthy state; if the link is detected to frequently switch between L0 and Recovery states, the actual rate is lower than the negotiated value, the error count exceeds the threshold (e.g., CRC errors ≥ 5 times within 1 second), or the link status remains in an inactive state such as Detect / Disable for a long time, then it is determined to be in a sub-healthy state or a faulty state. Simultaneously, the processor compares the real-time collected PCIe information with historical status data to identify sudden changes in the link status (e.g., a sudden switch from L0 to Hot Reset), further accurately locating abnormal trends in the link health status and providing a basis for subsequent recovery strategies.

[0062] The above embodiments achieve real-time and accurate monitoring of the health status of the NTB link by directly reading the hardware status information of the PCIe protocol layer. Compared with the traditional indirect status judgment relying on serial ports, this improves the timeliness and accuracy of fault detection.

[0063] S303. Determine whether there is a fault in the non-transparent bridge link based on the link health status.

[0064] If the link health status is not in a normal operating state, it can be determined that there is a fault in the NTB link. Among them, non-L0 states include: Detect, Polling, Configuration, Recovery, Low Power Active (L0s), Deep Low Power (L1), System-Level Low Power (L2), Mechanical Shutdown (L3), Hot Reset, Loopback, and Disable.

[0065] For example, when the link is in Recovery state, retraining is underway to change the original rate or fix some problems in the link. If it is in L0s state, it is entered to reduce power consumption when the bus is idle, and it can be quickly switched back to L0 state from this state.

[0066] In some embodiments, if the link health status is normal operating state L0, the bandwidth and / or rate of the NTB link are obtained through the first processor; if the bandwidth and / or rate do not meet preset conditions, it is determined that the NTB link is faulty. The preset conditions can be set according to the design specifications of the NTB link.

[0067] During the process of obtaining the bandwidth and / or rate of the NTB link through the first processor, the current link rate field in the link status register is read, which identifies the actual operating rate level of the link; at the same time, the performance monitoring register is read, and the real-time bandwidth is calculated by calculating the total number of bytes of transaction layer data packets transmitted bidirectionally by the NTB link per unit time.

[0068] For example, the preset conditions of this application may include: if the nominal link rate is PCIe 4.0 (16GT / s), then the preset rate threshold is not less than 16GT / s; if the theoretical maximum bandwidth is 32GB / s (x8 channel), then the preset bandwidth threshold is not less than 90% of the theoretical value (i.e. 28.8GB / s).

[0069] In the process of determining whether the bandwidth and / or rate meet the preset conditions, the first processor compares the real-time collected rate value with the rate threshold. If the actual rate is lower than the threshold, it is determined that the rate does not meet the preset conditions. At the same time, the real-time bandwidth is compared with the bandwidth threshold. If it is lower than the threshold for n consecutive collection cycles, it is determined that the bandwidth does not meet the preset conditions.

[0070] The above embodiments can promptly identify hidden faults when the link is in L0 state but its bandwidth and rate do not meet preset conditions. This prevents issues such as data interaction delays and efficiency reductions caused by link performance degradation from being overlooked, ensuring that the NTB link maintains reliable performance support during core data interactions and improving the stability of data transmission in the storage system. Real-time monitoring of key performance indicators such as bandwidth and rate can detect performance anomalies before the link completely fails, providing a time window for early intervention and repair, and reducing the risk of service interruption due to sudden link failures.

[0071] In some embodiments, after obtaining the bandwidth and / or rate of the NTB link, the first processor first calculates the first data transmission volume based on the bandwidth and / or rate, and then compares whether the first data transmission volume is greater than or equal to the historical data transmission volume. If yes, it indicates that the bandwidth and / or rate of the NTB link meets a preset condition; if no, it indicates that the bandwidth and / or rate of the NTB link does not meet the preset condition, and thus it can be determined that the NTB link has a fault. The historical data transmission volume is the amount of data transmitted between the first controller and the second controller through the NTB link during a historical period. If either the rate or bandwidth parameter does not meet the preset condition, the first processor determines that the NTB link has a latent fault.

[0072] The above quantitative monitoring of rate and bandwidth can accurately identify performance anomalies in the L0 state, avoid complete link interruption due to the accumulation of hidden faults, and further improve the comprehensiveness of NTB link health detection.

[0073] In some embodiments, after obtaining the bandwidth and / or rate of the NTB link, the first processor first calculates the first data transmission amount based on the bandwidth and / or rate, and then determines whether the bandwidth and / or rate of the NTB link meet the preset conditions based on the first data transmission amount and the pre-trained link prediction model.

[0074] The link prediction model is trained based on historical sample data to predict data transmission volume during normal service operation. The historical sample data includes historical link status information and historical service status information. During model training, the historical sample data covers transmission volume records corresponding to different service loads (such as peak / off-peak periods) and different link states (such as normal / attenuated performance in L0 state). The parameters are iterated by minimizing the error between the predicted value and the actual historical transmission volume to ensure adaptability to service fluctuations.

[0075] In the process of calculating the first data transmission volume by the first processor based on bandwidth and / or rate, the real-time bandwidth or rate is integrated with the window duration in units of fixed time windows to obtain the data transmission volume of that time window.

[0076] In the process of determining whether the bandwidth and / or rate of the NTB link meets the preset conditions based on the first data transmission volume and the pre-trained link prediction model, the first processor first determines the second data transmission volume corresponding to the current service status information based on the current service status information and the pre-trained link prediction model; then, the performance degradation is calculated based on the first data transmission volume and the second data transmission volume; if the performance degradation exceeds the preset threshold, it is determined that the bandwidth and / or rate of the NTB link does not meet the preset conditions.

[0077] The current business status information includes, but is not limited to: the number of input / output operations per second of the storage system, business type tags (such as database transaction processing, video stream storage, backup archiving, etc.), the number of concurrent accessing clients, and the average read / write size of data blocks.

[0078] Specifically, the current business status information is first standardized into input features that the link prediction model can recognize. This input is then fed into the model, which outputs the normal data transmission volume that the link should achieve in this scenario, i.e., the second data transmission volume. The second data volume is generated by the link prediction model by learning the mapping relationship between business status and corresponding normal transmission volume in historical sample data. Then, the performance degradation rate can be obtained using the formula "(second data transmission volume - first data transmission volume) / second data transmission volume × 100%".

[0079] The above method converts the difference between the first data transmission volume (actual value) and the second data transmission volume (benchmark value) into a specific performance degradation rate and compares it with a preset threshold. This avoids overreacting to minor performance fluctuations and accurately captures significant performance degradation that is sufficient to affect business operations.

[0080] For example, a performance degradation of less than 20% indicates a minor increase in the bit error rate, which does not affect current business performance. A performance degradation of 20% or more but less than 50% indicates increased data transmission latency, affecting current business performance. A performance degradation of 50% or more indicates an extremely high bit error rate, severely impairing business performance.

[0081] The above embodiments calculate the first data transmission volume based on real-time bandwidth and / or rate using a first processor and combine it with a pre-trained model for prediction. This allows the preset conditions to dynamically match the current business operation requirements, solving the problem of misjudgment or omission by traditional static thresholds during business fluctuations and improving the accuracy of fault diagnosis. The link prediction model is trained based on historical samples and can learn the normal fluctuation patterns of link performance under different business states, thereby predicting whether the current data transmission volume deviates from the reasonable range required for normal business operation in real-time monitoring. This intelligent judgment based on historical experience and business correlation can not only identify performance anomalies that have occurred, but also provide early warnings for potential link problems that gradually emerge with changes in business, enabling the system to shift from passive detection to proactive perception and improving its adaptability to complex business scenarios.

[0082] If the link health status is normal and the bandwidth and / or rate meet the preset conditions, it is determined that there is no fault in the NTB link, no repair is required, and the process ends.

[0083] S304. If a non-transparent bridge link is faulty, repair the non-transparent bridge link.

[0084] In some embodiments, if the link health status of the NTB link is not in the L0 state, it indicates that the NTB link is faulty. In this case, the first processor first switches the link health status to the Recovery state to train and repair the NTB link. Then, the entire storage device is restarted, and the first BMC performs a power-down and power-on operation to reset the first BMC.

[0085] In some embodiments, if the link health status is normal but the bandwidth and / or rate does not meet the preset conditions, it is determined that the NTB link is faulty. Then, the first processor switches the link health status to Recovery status to train and repair the non-transparent bridge link. Then, the entire storage device is restarted, and the first BMC performs a power-down and power-on operation to reset the first BMC.

[0086] During the repair of the non-transparent bridge link, the first processor switches the link health status from a non-L0 state to the Recovery state by writing to the PCIe controller's registers. In this state, the processor drives the link to execute a complete retraining process: the transmitter continuously sends TS1 / TS2 training sequences, the receiver re-completes bit locking, symbol synchronization, and link parameter negotiation (including rate, width, equalization coefficient, etc.), and monitors the training progress in real time by reading the link status register. If the training is successful (the link state switches back to L0), the repair log is recorded and the process ends; if the training fails (it remains in Recovery or switches to another non-L0 state), the next step, device restart, is initiated. Then, the first processor sends a restart command to the main control module of the storage device through the system management bus, triggering the orderly exit of the operating system and related business processes, and executing a hardware reset process. During the restart process, the hardware initialization capability of device-level restart is used to repair link failures that may be caused by abnormal software configuration.

[0087] After the storage device restarts, a power-down and power-on operation is performed through the first BMC to reset the first BMC, clearing any software anomalies that may exist in the first BMC and ensuring that it can properly monitor the processor status and coordinate with the peer BMC.

[0088] If a link anomaly is detected, the first processor switches the link status to "recovery state" and attempts to repair the link through retraining. It restarts the storage device only when necessary, rather than performing a complete system reset. Subsequently, the first baseboard management controller (BMC) performs a power-down and power-on operation, focusing on resetting the controller's local hardware to avoid affecting other service modules of the storage device, shortening recovery time and reducing the scope of service interruption. This progressive repair process—first repairing the local link, then restarting the system, and finally resetting the software—covers various fault causes in non-L0 states, improving the success rate of fault repair.

[0089] For example, NTB faults can be repaired in stages. Based on the performance degradation calculated in the aforementioned embodiments, when the performance degradation is less than 20%, the NTB link balancing parameters can be adjusted by the first processor, and the NTB link transmission queue can be restarted to attempt recovery without interrupting services.

[0090] When the performance degradation is greater than or equal to 20% but less than 50%, switch to recovery state first, perform T1 / T2 sequence negotiation, and if the training is successful, restore normal communication; if the training fails, perform NTB controller soft reset, clear the NTB link error state and then retrain, and generate an alarm to notify the operation and maintenance personnel.

[0091] When performance degradation exceeds 50%, the first processor shuts down and reinitializes the NTB port, simultaneously notifying the second controller to rebuild the link. This enables accurate identification and efficient repair of NTB link failures, minimizing the impact on business continuity.

[0092] In summary, the fault handling method for a storage device provided in this application, when no alarm signal is received from the second controller, first confirms the local processor is normal through the first baseboard management controller, and then the processor actively obtains the health status of the NTB link. This effectively distinguishes between a fault in the link itself and a communication interruption caused by a crash in the peer processor, solving the problem of ambiguous fault type judgment in the traditional mode. After clearly determining that there is a fault in the NTB link, the link is repaired directly, rather than using a coarse method of resetting the entire device, avoiding unnecessary interruption of normal business of the storage device. This local repair strategy targeting only the faulty link significantly shortens the fault recovery time, reduces system disturbance, and at the same time makes up for the shortcomings of the traditional mode, which lacks the ability to reset the CPU and NTB link separately, enhancing the system's refined fault repair capability.

[0093] like Figure 4 As shown in the embodiment of this application, another fault handling method for a storage device is provided. The method is applied to a second controller of the storage device. The storage device includes a first controller and a second controller. The second controller communicates with the first controller through a non-transparent bridge link. The second controller includes a second processor and a second substrate management controller.

[0094] The method includes the following steps:

[0095] S400: The second baseboard management controller detects whether the second processor has crashed.

[0096] The second controller detects whether the second processor has experienced a catastrophic error (CATERR) and subsequently crashes via the second BMC. If the second BMC detects a CATERR signal transmitted by the second processor, it indicates that the second processor has crashed.

[0097] S401. In the event of a second processor failure, an alarm signal is sent from the second baseboard management controller to the first controller, causing the first controller to stop communicating with the second controller via the non-transparent bridge link.

[0098] The second BMC sends an alarm signal to the first BMC of the first controller, causing the first BMC to send an interrupt signal to the first processor. In response to the interrupt signal, the first processor stops sending NTB messages to the second processor of the second controller via the NTB link.

[0099] A second BMC is introduced to specifically detect the status of the second processor, especially to directly determine whether the CPU has crashed through the catastrophic error signal (CATERR signal). A CPU health detection channel is established in parallel with the NTB link detection to avoid confusion between the two fault signals. Only when the second BMC detects the catastrophic error signal of the second processor will it send an alarm signal to the first BMC, thereby triggering the first processor to stop sending NTB messages. This process clearly corresponds to the CPU crash scenario.

[0100] S402, Reset the second controller via the second baseboard management controller.

[0101] By performing a power-down and power-on operation through the second BMC, the second controller is reset, and the crash error of the local second processor is repaired.

[0102] The aforementioned fault handling method achieves rapid and accurate fault identification by independently detecting the second processor's downtime status through the second baseboard management controller. Upon detecting its own processor downtime, the second controller immediately sends an alarm message to the first controller, enabling the first controller to quickly stop NTB link communication. This blocks the fault propagation path at its source and prevents the first controller from triggering its watchdog mechanism due to continuous message sending without receiving responses. This effectively prevents a single controller failure from spreading to the entire system, ensuring the continuity of core system services. Only the downtime second controller needs to be reset, without affecting the normal operation of the first controller, avoiding the complete service interruption caused by traditional system-wide resets and improving the efficiency and accuracy of fault repair.

[0103] like Figure 5A As shown, Figure 5A This application provides an interactive schematic diagram of a fault handling method for a storage device, which includes the following steps:

[0104] S501, the second controller detects whether the second processor has crashed through the second baseboard management controller.

[0105] S502, In the event of a second processor failure, an alarm signal is sent to the first controller via the second baseboard management controller.

[0106] S503, the first controller responds to the alarm signal sent by the second board management controller and stops communicating with the second controller through the non-transparent bridge link.

[0107] S504, the first controller detects whether the first processor has crashed through the first baseboard management controller.

[0108] S505, In the event of a first processor failure, the first controller is reset via the first baseboard management controller.

[0109] like Figure 5B As shown, Figure 5B This application provides an interactive schematic diagram of a fault handling method for a storage device, which includes the following steps:

[0110] S601, the second controller detects whether the second processor has crashed through the second baseboard management controller.

[0111] If the second processor does not crash, the first controller executes step S602, which detects whether the first processor has crashed through the first baseboard management controller.

[0112] S603. If the first processor does not crash, obtain the link health status of the non-transparent bridge link through the first processor.

[0113] S604. The first processor determines whether there is a fault in the non-transparent bridge link based on the link health status.

[0114] S605. If a non-transparent bridge link is faulty, repair the non-transparent bridge link.

[0115] Specifically, when the link health status of the non-transparent bridge link is abnormal, the first processor first switches from the abnormal state to the recovery state, retrains and repairs the non-transparent bridge link, then restarts the storage device, and performs a power-down and power-on operation through the first baseboard management controller to reset the first controller.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0117] Embodiments of this application also provide a controller, which includes: a first processor and a first baseboard management controller;

[0118] The first baseboard management controller is used to detect whether it receives an alarm signal sent by the second controller, the alarm signal indicating that the second processor of the second controller has crashed; if it receives an alarm signal sent by the second controller, it stops communicating with the second controller through the non-transparent bridge link; detect whether the first processor has crashed; if the first processor crashes, it resets the first controller.

[0119] As an optional implementation provided in this application, the first baseboard management controller is specifically configured to: send an interrupt signal to the first processor upon receiving an alarm signal from the second controller; the first processor is configured to stop sending non-transparent bridge messages to the second controller via the non-transparent bridge link in response to the interrupt signal.

[0120] As an optional implementation provided in this application, the first baseboard management controller, after detecting whether it has received an alarm signal sent by the second controller, is further configured to: detect whether the first processor has crashed if no alarm signal has been received from the second controller; the first processor is configured to obtain the link health status of the non-transparent bridge link if no crash has occurred; determine whether the non-transparent bridge link has a fault based on the link health status; and repair the non-transparent bridge link if a fault exists in the non-transparent bridge link.

[0121] As an optional implementation provided in this application, the first processor is specifically used to switch the link health status to a recovery state and train to repair the non-transparent bridge link if the link health status of the non-transparent bridge link is in an abnormal working state and a fault is determined; and to restart the storage device; the first baseboard management controller is used to perform a power-down and power-on operation to reset the first controller.

[0122] As an optional implementation provided in this application, after determining whether a non-transparent bridge link is faulty based on the link health status, the first processor is further configured to: if the link health status of the non-transparent bridge link is in normal working condition, obtain the bandwidth and / or rate of the non-transparent bridge link; if the bandwidth and / or rate do not meet preset conditions, determine that the non-transparent bridge link is faulty.

[0123] As an optional implementation provided in this application, when the link health status is not in a normal working state, after obtaining the bandwidth and / or rate of the non-transparent bridge link through the first processor, the first processor is further configured to: calculate the first data transmission amount based on the bandwidth and / or rate; compare whether the first data transmission amount is greater than or equal to the historical data transmission amount, wherein the historical data transmission amount is the amount of data transmitted between the first controller and the second controller through the non-transparent bridge link during a historical period; if the first data transmission amount is less than the historical data transmission amount, then determine that the bandwidth and / or rate do not meet the preset conditions.

[0124] As an optional implementation provided in this application, after calculating the first data transmission volume based on bandwidth and / or rate, the first processor is further configured to: determine whether the bandwidth and / or rate meet preset conditions based on the first data transmission volume and a pre-trained link prediction model; wherein, the link prediction model is trained based on historical sample data and is used to predict the data transmission volume when the service is operating normally; the historical sample data includes historical link status information and historical service status information.

[0125] As an optional implementation method provided in this application, the first processor is specifically used for: determining the second data transmission volume corresponding to the service status information based on the current service status information and the link prediction model; calculating the performance degradation rate based on the first data transmission volume and the second data transmission volume; and determining that the bandwidth and / or rate do not meet the preset conditions when the performance degradation rate is greater than a preset threshold.

[0126] As an optional implementation provided in this application, the first processor is specifically used to: read the high-speed peripheral component interconnection information of the non-transparent bridge link through the first processor when the first processor does not crash; and determine the link health status based on the high-speed peripheral component interconnection information.

[0127] For a description of the features in the embodiment corresponding to the controller, please refer to the relevant description of the embodiment corresponding to the fault handling method for the storage device applied to the first controller, which will not be repeated here.

[0128] Embodiments of this application also provide a controller, which includes: a first processor and a first baseboard management controller; wherein, a second baseboard management controller is configured to detect whether the second processor has crashed; in the event that the second processor has crashed, send an alarm signal to the first controller so that the first controller stops communicating with the second controller through the non-transparent bridge link; and reset the second controller.

[0129] For a description of the features in the embodiment corresponding to the controller, please refer to the relevant description of the embodiment corresponding to the fault handling method for the storage device applied to the second controller, which will not be repeated here.

[0130] like Figure 6 As shown, embodiments of this application also provide an electronic device, including a memory 701 and a processor 702. The memory 701 stores a computer program, and the processor 702 is configured to run the computer program to perform the steps in any of the above-described embodiments of the fault handling method for a storage device.

[0131] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the fault handling method for storage devices when it is run.

[0132] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0133] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the fault handling method for storage devices.

[0134] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the fault handling method for storage devices.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The foregoing has provided a detailed description of a fault handling method, controller, device, medium, and product for a storage device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault handling method for a storage device, characterized in that, The method is applied to a first controller of the storage device, the storage device including a first controller and a second controller, the first controller and the second controller communicating via a non-transparent bridge link; The first controller includes a first processor and a first baseboard management controller; the method includes: The first baseboard management controller detects whether it receives an alarm signal sent by the second controller, the alarm signal indicating that the second processor of the second controller has crashed; the alarm signal detection is based on receiving a catastrophic error signal transmitted by the first processor through the general-purpose input / output interface; Upon receiving an alarm signal from the second controller, communication with the second controller via the non-transparent bridge link is stopped; The first baseboard management controller detects whether the first processor has crashed. In the event of a first processor failure, the first controller is reset by performing a power-down and power-on operation through the first baseboard management controller in order to repair the failed first processor. The step of stopping communication with the second controller via the non-transparent bridge link upon receiving an alarm signal from the second controller includes: upon receiving an alarm signal from the second controller, sending an interrupt signal to the first processor through the first baseboard management controller; the first processor responding to the interrupt signal to obtain the current working state of the non-transparent bridge controller and confirm whether the message transmission queue is active; if so, writing a disable instruction to the transmit enable register of the non-transparent bridge controller to suspend the transmission of non-transparent bridge messages; and simultaneously traversing and clearing the message data that has not yet been transmitted in the non-transparent bridge transmission buffer.

2. The method according to claim 1, characterized in that, After the first baseboard management controller detects whether an alarm signal sent by the second controller is received, the method further includes: If no alarm signal is received from the second controller, the first baseboard management controller detects whether the first processor has crashed. If the first processor does not crash, the link health status of the non-transparent bridge link is obtained through the first processor; The first processor determines whether the non-transparent bridge link is faulty based on the link health status. If the non-transparent bridge link is faulty, then repair the non-transparent bridge link.

3. The method according to claim 2, characterized in that, If the non-transparent bridge link is faulty, repairing the non-transparent bridge link includes: If the link health status of the non-transparent bridge link is in an abnormal working state, and a fault is determined, the first processor will switch the link health status to a recovery state to train and repair the non-transparent bridge link. Reboot the storage device; The first baseboard management controller performs a power-down and power-on operation to reset the first controller.

4. The method according to claim 2, characterized in that, The step of determining whether the non-transparent bridge link is faulty based on the link health status using the first processor includes: If the link health status of the non-transparent bridge link is normal working status, then the bandwidth and / or rate of the non-transparent bridge link are obtained through the first processor. If the bandwidth and / or the rate do not meet the preset conditions, it is determined that the non-transparent bridge link is faulty.

5. The method according to claim 4, characterized in that, If the link health status of the non-transparent bridge link is in normal working condition, then after obtaining the bandwidth and / or rate of the non-transparent bridge link through the first processor, and before determining that the non-transparent bridge link has a fault if the bandwidth and / or the rate does not meet a preset condition, the method includes: The first processor calculates the first data transmission amount based on the bandwidth and / or the rate; The first processor compares whether the first data transmission volume is greater than or equal to the historical data transmission volume, where the historical data transmission volume is the amount of data transmitted between the first controller and the second controller through a non-transparent bridge link during a historical period. If the first data transmission volume is less than the historical data transmission volume, then it is determined that the bandwidth and / or the rate do not meet the preset conditions.

6. The method according to claim 5, characterized in that, After the first processor calculates the first data transmission amount based on the bandwidth and / or the rate, the method includes: The first processor determines whether the bandwidth and / or the rate meet the preset conditions based on the first data transmission volume and the pre-trained link prediction model. The link prediction model is trained based on historical sample data and is used to predict the data transmission volume when the service is operating normally; the historical sample data includes historical link status information and historical service status information.

7. The method according to claim 6, characterized in that, The step of determining whether the bandwidth and / or the rate meet the preset conditions by the first processor based on the data transmission volume and the pre-trained link prediction model includes: The first processor determines the second data transmission volume corresponding to the current service status information based on the current service status information and the link prediction model. Calculate the performance degradation based on the first data transmission volume and the second data transmission volume; If the performance degradation exceeds a preset threshold, it is determined that the bandwidth and / or the rate do not meet the preset conditions.

8. A fault handling method for a storage device, characterized in that, A second controller applied to a storage device, the storage device including a first controller and a second controller, the first controller and the second controller communicating via a non-transparent bridge link; the second controller including a second processor and a second substrate management controller; the method includes: The second baseboard management controller detects whether the second processor has crashed; the alarm signal detection is based on receiving catastrophic error signals transmitted by the second processor through the general-purpose input / output interface. In the event of a second processor failure, the second baseboard management controller sends an alarm signal to the first controller. Upon receiving the alarm signal, the first controller sends an interrupt signal to the first processor via the first baseboard management controller. The first processor, responding to the interrupt signal, obtains the current operating state of the non-transparent bridge controller and confirms whether the message transmission queue is active. If so, it writes a disable instruction to the transmit enable register of the non-transparent bridge controller, pausing the transmission of non-transparent bridge messages. Simultaneously, it iterates through and clears any unsent message data in the non-transparent bridge transmit buffer. The second baseboard management controller performs a power-down and power-on operation to reset the second controller, thereby repairing the failed second processor.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault handling method for the storage device as claimed in any one of claims 1 to 7, or the steps of the fault handling method for the storage device as claimed in claim 8, when executing the computer program.

Images