Method for handling faults and electronic device

Abstract

The application provides a fault processing method and electronic equipment, and relates to the technical field of computers.When a backboard fault is monitored, the hardware fault and the firmware fault are accurately distinguished based on a state parameter, for the firmware fault, different detection strategies are adopted according to different fault hardware to further judge whether it is repairable, and different processing modes are executed for different fault types and fault states, including physical isolation, different firmware recovery strategies are adopted according to different fault hardware, multi-dimensional diagnosis and fine processing of the fault are realized, so that the cause of the backboard fault can be accurately identified, and effective automatic repair or isolation measures are executed.The technical effects of improving the accuracy, automation degree and overall reliability of the hard disk backboard fault processing are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a fault processing method and electronic equipment. BACKGROUND

[0002] In modern large data centers and high-performance computing environments, the server hard disk backboard, as the key hub connecting a large number of storage devices and system mainboards, undertakes important functions such as hard disk power supply, data exchange and state monitoring, and its running stability directly determines the reliability of the entire storage system.

[0003] The fault monitoring scheme of the server hard disk backboard in the related art usually only relies on threshold judgment of physical parameters such as voltage and temperature, and the monitoring dimension is single, which cannot accurately detect faults and lacks fault handling means. Therefore, how to accurately detect backboard faults and handle faults is a problem to be solved at present. SUMMARY

[0004] The present application provides a fault processing method and electronic equipment to at least solve the problem of how to accurately detect backboard faults and handle faults in the related art.

[0005] The present application provides a fault processing method, comprising:

[0006] In response to a fault occurring in a server hard disk backboard, determining a fault type of the hard disk backboard according to a monitored state parameter of the hard disk backboard;

[0007] In response to the fault type being a hardware fault, performing physical isolation processing on a first hardware corresponding to the hardware fault, and recording fault information of the hardware fault;

[0008] In response to the fault type being a firmware fault, determining a fault hardware corresponding to the firmware fault according to the state parameter, and detecting a fault state of the firmware fault based on register parameters, check codes and fault detection strategies corresponding to different fault hardware, wherein the state parameter comprises the register parameter;

[0009] In response to the fault state being a repairable fault, extracting a backup firmware from a pre-device firmware library, and performing firmware recovery processing on firmware with the firmware fault through the backup firmware based on firmware programming strategies corresponding to different fault hardware, wherein the firmware programming strategies at least include a programming strategy based on a first programming mode and a preset test clock frequency, or a programming strategy based on a second programming mode in a safe mode of third hardware.

[0010] The present application also provides a fault processing apparatus, comprising:

[0011] The judging unit is configured to determine the fault type of the hard disk backboard according to the monitored state parameter of the hard disk backboard in response to a fault of the server hard disk backboard.

[0012] The processing unit is configured to perform physical isolation processing on the first hardware corresponding to the hardware fault and record the fault information of the hardware fault in response to the fault type being the hardware fault.

[0013] The determining unit is configured to determine the fault hardware corresponding to the firmware fault according to the state parameter in response to the fault type being the firmware fault.

[0014] The detecting unit is configured to detect the fault state of the firmware fault based on the register parameter, the check code and the fault detection strategy corresponding to different fault hardware, wherein the state parameter comprises the register parameter.

[0015] The extracting unit is configured to extract the backup firmware from a pre-device firmware library in response to the fault state being the repairable fault.

[0016] The restoring unit is configured to perform firmware recovery processing on the firmware with the firmware fault by using the backup firmware based on the firmware programming strategy corresponding to different fault hardware, wherein the firmware programming strategy at least comprises a programming strategy based on a first programming mode and a preset test clock frequency, or a programming strategy based on a second programming mode in a security mode of third hardware.

[0017] The application further provides an electronic device, comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of the above-mentioned fault processing method.

[0018] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned fault processing method.

[0019] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned fault processing method.

[0020] The application provides a fault processing method and an electronic device. When a backboard fault is monitored, the hardware fault and the firmware fault are accurately distinguished based on a state parameter. For the firmware fault, different detection strategies are adopted according to different fault hardware to further determine whether the fault hardware is repairable. Different processing modes are performed for different fault types and fault states, including physical isolation, different firmware recovery strategies are adopted according to different fault hardware, multi-dimensional diagnosis and fine processing of the fault are realized, so that the cause of the backboard fault can be accurately identified, and effective automatic repair or isolation measures are performed. Therefore, the technical problems of inaccurate fault detection due to single monitoring dimension and lack of effective processing means in the related art can be solved, and the technical effects of improving the accuracy, automation degree and overall reliability of the hard disk backboard fault processing are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a fault processing method provided by an embodiment of the present application is shown.

[0023] Figure 2 A fault detection and processing flowchart provided by an embodiment of the present application is shown.

[0024] Figure 3 A firmware fault detection flowchart of a second hardware provided by an embodiment of the present application is shown.

[0025] Figure 4 A firmware fault detection flowchart of a third hardware provided by an embodiment of the present application is shown.

[0026] Figure 5 A firmware recovery flowchart provided by an embodiment of the present application is shown.

[0027] Figure 6 Another fault detection and processing flowchart provided by an embodiment of the present application is shown.

[0028] Figure 7 A firmware recovery flowchart provided by an embodiment of the present application is shown.

[0029] Figure 8 A structure diagram of a fault processing device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] This application relates to the field of server hardware fault handling, and provides an efficient and automated fault handling method for various faults that may occur in server hard drive backplanes during operation. As a critical hub connecting storage devices and the system bus, the stability of the server hard drive backplane directly affects the reliability of the entire storage system. Fault handling methods in related technologies often rely on comparisons of single parameter thresholds, which cannot distinguish the root cause of the fault, resulting in limited recovery methods and the need for manual intervention. This application, through real-time monitoring and intelligent analysis, achieves accurate judgment and differentiated handling of fault types, thereby improving the system's self-healing capabilities and operational efficiency.

[0033] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 This document provides a flowchart illustrating a fault handling method as an embodiment of the present application. The method will be described in detail below, taking into account the execution flow of the fault handling method.

[0035] like Figure 1 As shown, the fault handling method includes:

[0036] Step 101: In response to a server hard disk backplane failure, determine the type of hard disk backplane failure based on the monitored status parameters.

[0037] In the embodiments of the present application, the server hard disk backplane is a circuit board in the server for connecting and managing multiple hard disks, which not only provides electrical connection between the hard disk and the mainboard, but also integrates various monitoring and management functions to ensure the stability of data reading and writing and power distribution. In actual operation, the hard disk backplane may fail due to voltage fluctuation, temperature anomaly or chip logic error, etc., affecting the normal operation of the storage system. The present application realizes continuous evaluation of the health status of the backplane through real-time collection and analysis of state parameters. The state parameters refer to various parameter information reflecting the running state of the server hard disk backplane, including but not limited to the voltage value, current value, temperature reading of the backplane, and the working state signal of the key chip (such as Complex Programmable Logic Device (CPLD) or hard disk expander chip), which are collected periodically or event-driven by built-in sensors and management controllers (such as Baseboard Management Controller (BMC)) to form a multi-dimensional description of the running status of the backplane.

[0038] The fault type refers to the classification of the cause of the fault, mainly including hardware failure and firmware failure. Hardware failure usually involves damage to physical components, such as power module burnout, circuit short circuit or chip physical failure; while firmware failure refers to errors or damage in program code stored in non-volatile memory, causing abnormal chip logic function.

[0039] The judgment process is based on a preset fault model pre-stored in the management controller, which distinguishes the root cause of the fault by analyzing the abnormal patterns of the state parameters (such as voltage drop, continuous temperature exceeding the standard or heartbeat signal interruption). For example, if the input voltage feedback by the voltage detection circuit is zero or exceeds the safe range, it is preliminarily determined as a hardware failure; if the chip state register returns an abnormal value but the power supply is normal, it may belong to firmware failure.

[0040] Step 102, in response to the fault type being hardware failure, the first hardware corresponding to the hardware failure is physically isolated and processed, and the fault information of the hardware failure is recorded and processed.

[0041] In the embodiments of the present application, the first hardware refers to the specific physical component identified as the cause of the failure, such as a failed power module, a damaged current sensor or a failed voltage regulation chip. Physical isolation is a hardware-level protection mechanism that cuts off the power supply or signal path of the faulty hardware to prevent it from affecting the normal operation of other parts, for example, physical isolation can be achieved through fuses, electronic switches or relays, etc. to ensure that the fault is limited to the minimum range.

[0042] The fault information includes, but is not limited to, details such as a fault occurrence time, a location identifier, a parameter abnormal value, and a taken isolation measure; the record processing refers to writing the fault information into a non-volatile log memory of the management controller, facilitating subsequent operation and maintenance personnel to view and analyze, and providing data support for preventive maintenance.

[0043] In step 103, in response to the fault type being a firmware fault, a fault hardware corresponding to the firmware fault is determined according to a state parameter, and a fault state of the firmware fault is detected based on respective register parameters, check codes, and a fault detection strategy of different fault hardware, wherein the state parameter includes the register parameter, and the fault hardware includes second hardware and third hardware.

[0044] In the embodiments of the present application, the fault hardware is determined according to the state parameter, that is, a specific chip or component affected by the firmware fault is identified, for example, a CPLD or a hard disk expander chip, and the fault detection strategy is a predefined fault diagnosis process designed for different chip types, for example, for the CPLD, the strategy includes, but is not limited to, checking the continuity of the heartbeat signal, reading the state register value, or performing a firmware cyclic redundancy check (CRC) check; for the hard disk expander chip, the strategy can further include a link layer physical layer (PHY) test and a bit error rate analysis.

[0045] The fault state is used to describe the repairability of the firmware fault, and is divided into a repairable fault and an irreparable fault. The repairable fault usually refers to a partial damage of a firmware image or an error of configuration data, but the chip hardware itself is intact, and can be restored through firmware update; the irreparable fault can involve physical damage of a non-volatile memory or complete collapse of a firmware structure.

[0046] The fault detection strategy at least includes a strategy of detecting the second hardware based on the register parameter and the check code, or a strategy of detecting the third hardware based on the register parameter, the check code, an event log, and a physical layer test.

[0047] When detecting the fault state of the firmware fault, the respective register parameters, check codes, and fault detection strategies of different fault hardware are performed. The register parameter is a real-time reading of a hardware internal state register, for the second hardware (CPLD), the first state register parameter can include a power ready flag and an error code bit field; for the third hardware (hard disk expander chip), the second state register parameter can include a link connection state and a bit error rate counter. The check code is data used to verify the integrity of the firmware, for example, a cyclic redundancy check (CRC) value, whether the firmware is damaged is judged by comparing a real-time generated check code with a historically stored reference value. The fault detection strategy is a predefined diagnosis process designed for different hardware types.

[0048] In step 104, in response to the fault state being a repairable fault, backup firmware is extracted from the pre-device backup firmware library, and based on different fault hardware corresponding to respective firmware programming strategies, firmware recovery processing is performed on the firmware with the fault by using the backup firmware, wherein the firmware programming strategies at least include a programming strategy based on a first programming mode and a preset test clock frequency, or a programming strategy based on a second programming mode in a safe mode of third hardware.

[0049] In the embodiments of the present application, the pre-device backup firmware library is a firmware collection stored in a server security area (such as a separate Flash chip or a BMC firmware partition), including but not limited to: known intact firmware versions of different board card types and chip models; and a backup firmware refers to a firmware image file matched with the fault hardware, for example: a configuration file of a CPLD or a firmware image of a hard disk expander chip.

[0050] The firmware programming strategy refers to a specific programming mode and process preset for different types of fault hardware for burning or flashing the backup firmware into the fault hardware. Different fault hardware has different corresponding firmware programming strategies due to different hardware interfaces and working mechanisms, that is, the firmware programming strategy defines the specific mode of firmware burning, for example: for a CPLD, a Joint Test Action Group (JTAG) programming mode (first programming mode) is adopted to perform erase and burn operations through a JTAG interface; for a Microcontroller Unit (MCU) chip (such as a hard disk expander), a Serial Peripheral Interface (SPI) programming mode (second programming mode) is adopted to read and write a Flash memory through an SPI bus.

[0051] The firmware recovery processing includes but is not limited to suspending Input / Output (I / O) operations of an affected service domain, loading backup firmware, executing a programming sequence, performing a function self-check after completion, and resetting the chip, and finally restoring the service operation. The entire recovery process is completed automatically without human intervention, significantly improving system availability.

[0052] Further, in order to facilitate understanding of the process of fault processing in the present application, the embodiments of the present application provide a fault detection and processing flow diagram, as shown in Figure 2As shown, if the hard disk backboard fails, the BMC determines whether the fault type is a hardware fault or a firmware fault according to the preset fault model and the extracted fault characteristic parameters. If it is a hardware fault, such as voltage anomaly or power supply chip failure, a fuse strategy is adopted to isolate the faulty power supply or chip. If it is a firmware fault, it is determined whether it is a repairable fault. If it is an unrepairable fault, it is recorded and waits for subsequent maintenance. If it is a repairable fault, a firmware repair process is started, the chip is stopped, firmware information is refreshed from the backup firmware, and after the update is completed, a function self-check is performed. After the self-check passes, the chip is restarted, and normal work is resumed. The recovery log is recorded. If the self-check fails, the update failure is recorded in the log and the chip is isolated.

[0053] The present application can accurately and efficiently handle different types of faults of the server hard disk backboard. For hardware faults, physical isolation processing is used to ensure continuous operation of the system and record fault information for subsequent maintenance. For repairable firmware faults, backup firmware is automatically extracted and a suitable firmware programming strategy is used for firmware recovery to quickly repair the fault, significantly improving the fault handling capability of the server hard disk backboard and the overall reliability of the system, meeting the high requirements of large data centers and big data computing scenarios for the stability of the storage system.

[0054] In one implementation manner of the embodiment of the present application, when detecting the fault state of the firmware fault, the following manner can be used, but is not limited to: if the fault hardware is the second hardware, the fault state of the firmware fault is detected according to the first state register parameter corresponding to the second hardware and the first check code, wherein the state parameter includes the first state register parameter; if the fault hardware is the third hardware, the fault state of the firmware fault is detected according to the second state register parameter corresponding to the third hardware, the event log and the second check code, wherein the state parameter includes the second state register parameter and the event log.

[0055] In the embodiment of the present application, the fault hardware refers to the chip or component affected by the firmware fault. According to its function and structural characteristics, it can be classified as second hardware or third hardware. The second hardware usually refers to a complex programmable logic device (CPLD), which is responsible for monitoring the basic state signals on the hard disk backboard (such as hard disk in-place detection, power failure indication), and performing simple logic control functions. The third hardware refers to a hard disk expander chip, which is an interface expansion unit connecting the mainboard hard disk controller and physical hard disk, and undertakes complex tasks such as data routing and protocol conversion. The distinction between the second and third hardware is based on the difference in internal architecture and firmware storage method. For example, the firmware of CPLD is usually stored in an on-chip or external configuration memory, while the firmware of the hard disk expander chip is usually resident in a dedicated Flash chip.

[0056] If the faulty hardware is identified as the second hardware, the detection process relies on the first status register parameter and the first check code. The first status register parameter is a real-time reading of the second hardware's internal status register, which records key operating indicators of the second hardware, such as power readiness flags, signal phase-locked states, or error code bit fields. By analyzing the first status register parameter, it can be preliminarily determined whether the hardware is in an abnormal working mode, such as the register return value exceeding the preset range or a specific error bit being set. The first check code usually refers to the cyclic redundancy check (CRC) value, which is used to verify the integrity of the second hardware firmware image. During detection, the management controller reads the CRC calculation value of the current firmware and compares it with the pre-stored reference check code. If the CRC matches, it indicates that the firmware is not damaged, and the fault may be caused by temporary interference, which can be recovered by resetting; if the CRC does not match, it is confirmed that the firmware image is damaged, and the fault state is marked as a repairable fault, triggering the firmware recovery process. The entire detection process emphasizes the cooperation of status parameters and check mechanisms to ensure reliable diagnosis of second hardware firmware faults.

[0057] If the faulty hardware is identified as the third hardware, the detection process involves the second status register parameter, the event log, and the second check code. The second status register parameter is the status information of the third hardware, including but not limited to link connection status, data throughput indicators, or error counter values, which reflects the health of the chip at the data exchange level. For example, if the bit error rate field in the status register is consistently high, it may indicate a physical layer signal quality problem; and if the connection state bit is abnormal, it implies a protocol stack logic error.

[0058] The event log is a collection of historical events recorded by the third hardware during operation, stored in non-volatile memory, and includes but is not limited to error event timestamps, interrupt trigger records, or internal queue state changes. Analyzing the event log helps to distinguish between transient errors and persistent faults, for example, repeated communication timeout events in the log may point to firmware logic defects. The second check code is also based on CRC or similar algorithms, but is optimized for the firmware structure of the third hardware to ensure coverage of all key code segments.

[0059] During the detection of the third hardware, the second status register parameter, event log entries, and check code results are integrated: if the status register is abnormal but the log has no major errors and the check code matches, it may be resolved by resetting the chip; if the check code does not match or there are firmware-related error records in the log, it is determined as a repairable fault, which requires starting firmware update.

[0060] The application can accurately judge the repairability of the firmware failure by identifying the category to which the faulty hardware belongs and calling the corresponding diagnostic process, thereby providing a reliable basis for subsequent recovery operations. For the second hardware, the focus is on the rapid verification of the state register and the check code to adapt to its simple control function; for the third hardware, event log analysis is introduced to cope with its complex data processing role. The comprehensiveness and accuracy of the fault state detection are ensured.

[0061] In an implementation manner of the embodiment of the present disclosure, when detecting the fault state of the firmware failure of the second hardware, the following manner can be adopted but is not limited to: performing parameter comparison processing on the first state register parameter and the first preset register parameter to obtain a first comparison result; if the first comparison result is parameter inconsistency, it is determined that the fault state is an irreparable failure, and the failure information of the second hardware is recorded; if the first comparison result is parameter consistency, a first check code is generated based on the firmware in which the firmware failure exists in the second hardware, and the first check code is compared with a historical first check code to obtain a first check result, wherein the historical first check code is a check code generated by the firmware in which the firmware failure exists before the failure occurs; if the first check result is check code inconsistency, it is determined that the fault state is a repairable failure.

[0062] In the embodiment of the present application, the parameter comparison processing is to compare the first state register parameter collected in real time with the first preset register parameter stored in advance item by item. The first state register parameter is the real-time reading of the internal state register of the second hardware, including power readiness state, signal latch flag, timing control state and other key operating indicators; the first preset register parameter is the standard range or expected mode of the register value of the second hardware in the normal working state, which is stored in the non-volatile memory of the management controller. The parameter comparison processing is implemented by the management controller (such as BMC) through a special bus to read the state register, and then a bit-level or field-level comparison algorithm is executed. It aims to identify whether the second hardware is in a basic operable state, for example: whether the register return value is within the valid value range, or whether the specific control bit is correctly set.

[0063] The first comparison result obtained after comparison is the primary basis for judging the fault state. If the first comparison result is parameter inconsistency, that is, there is a significant deviation between the first state register parameter and the first preset register parameter, it indicates that the second hardware may have suffered physical damage or core logic unit failure. Such abnormalities usually point to irreversible hardware damage, so the fault state is determined to be an irreparable failure. Subsequently, the failure information of the second hardware, including but not limited to abnormal register address, deviation value, detection timestamp, etc., is recorded, which is written into the log area of the management controller to provide a basis for subsequent hardware replacement.

[0064] If the first comparison result is consistent with the parameters, it indicates that the basic function unit of the second hardware responds normally, and the fault may be limited to the firmware level. At this time, the detection enters the deep verification stage: the first check code is generated in real time based on the firmware with firmware faults in the second hardware. The check code generation process reads the firmware image currently residing in the storage area of the second hardware through the management controller, and calculates the real-time check value by running the cyclic redundancy check (CRC) algorithm. At the same time, the historical first check code, i.e., the reference check code generated and stored in advance by the firmware before the fault occurs and in a perfect state, is called. Then, the check code comparison process is performed to accurately match and verify the real-time generated value and the historical value.

[0065] The first check result obtained after comparison is a key indicator for determining the integrity of the firmware. If the first check result is inconsistent with the check code, it indicates that there is a data difference between the current firmware image and the original perfect version, such as a damaged program code segment, a disordered configuration data, or a tampered parameter area. Since the physical function of the second hardware is still normal through the aforementioned register verification, such data errors at the firmware level can be repaired by re-burning, so the fault state is determined to be a repairable fault, triggering the subsequent firmware recovery process.

[0066] Further, in order to facilitate understanding of the embodiments of the present application, the present application provides a schematic diagram of a firmware fault detection process of the second hardware, as shown in Figure 3 The BMC monitors the heartbeat signal of the second hardware (such as CPLD) in real time, and if the heartbeat signal is abnormal, it performs deep diagnosis, reads the CPLD state register, and if the register value is not legal, it indicates that there is a fault in the chip hardware, then records the fault, if the register value is legal, it performs CRC check, if the check is successful, it resets the heartbeat counter, i.e., resets the CPLD, if the check fails, it indicates a firmware fault, and starts the firmware repair process.

[0067] The present application effectively avoids misjudgment caused by temporary signal interference through progressive diagnosis of verifying the hardware basic state first and then verifying the integrity of the firmware, and accurately distinguishes between physical damage and data errors. The reliability of the second hardware firmware fault state detection is significantly improved, which provides a guarantee for implementing the correct recovery strategy.

[0068] In an implementation manner of the embodiment of the present application, when the comparison between the first check code and the historical first check code is completed and the first check result is obtained, if the first check result indicates that the check codes are consistent, it means that the firmware image running in the current second hardware is completely matched with the original intact version in the data level, and there is no detectable code damage or configuration error, at this time, the fault state is determined as a reset pending fault. The reset pending fault is a specific fault classification, which means that the hardware function shows abnormal behavior, but the firmware integrity verification is passed, indicating that the problem may be caused by temporary state confusion, register lock or timing synchronization error, etc. reversible logic fault, rather than persistent damage.

[0069] Specifically, the following method can also be used, but is not limited to: if the first check result is that the check codes are consistent, the fault state is determined as a reset pending fault, and the second hardware is reset to recover from the firmware fault.

[0070] In the embodiment of the present application, for the case of being determined as a reset pending fault, the reset processing of the second hardware is started. The reset pending fault case can mean that the second hardware enters an unexpected state due to power disturbance or signal conflict in long-term operation. The reset processing is an operation of forcibly restoring the initial state of the chip through a hardware signal, specifically, the management controller sends a specific timing pulse signal to the second hardware through a special reset signal line, or executes a soft reset command on its internal control register through a write operation. This process will clear the runtime state data of the second hardware, reset all logic units and reload the firmware configuration, so as to eliminate the function abnormality caused by temporary factors.

[0071] The reset processing realizes the fault recovery of the firmware fault, which means that the second hardware is restored to normal function only through state reset without relying on firmware reprogramming. After the reset operation is executed, the heartbeat signal and the state register parameter of the second hardware are monitored again to verify whether it returns to the expected working mode. If the second hardware function returns to normal after reset, it is confirmed that the fault has been eliminated; if the abnormality continues, more in-depth diagnosis may be needed. The present application effectively distinguishes firmware data error and runtime state abnormality through the confirmation of the reset pending fault, avoids unnecessary firmware refreshing operation, and significantly improves the server self-healing efficiency and operation and maintenance intelligent level.

[0072] In an implementation manner of the embodiment of the present application, when the fault state of the firmware fault of the third hardware is detected, the following manner can be adopted, but is not limited to: parameter comparison processing is performed on the second state register parameter and the second preset register parameter, to obtain a second comparison result; if the second comparison result is parameter consistent, physical layer test processing is performed on the third hardware, to obtain a test result, and the hardware state of the third hardware is determined according to the test result, wherein the hardware state is existing irreparable fault or no fault; if the second comparison result is parameter inconsistent, log analysis processing is performed on the event log, to obtain an analysis result, and in response to the analysis result being that there is abnormal data in the event log, it is determined that the fault state is an irreparable fault, wherein the abnormal data is data in the event log that does not meet a preset log condition; if the analysis result is that there is no abnormal data in the event log, a second check code is generated based on the firmware in the third hardware existing the firmware fault in real time, and the second check code is compared with a historical second check code, to obtain a second check result; if the second check result is check code inconsistent, it is determined that the firmware fault is a repairable fault.

[0073] In the embodiment of the present application, the parameter comparison processing is that the second state register parameter collected in real time is compared with the second preset register parameter stored in advance. The second state register parameter is a real-time reading of the internal state register of the third hardware, including but not limited to link connection state, data packet error count, buffer usage rate and other core running indicators; the second preset register parameter is the expected value range of each state bit of the third hardware in the standard working mode, which is stored in the configuration area of the management controller. The parameter comparison processing is completed by the management controller through field matching algorithm after reading the state register through the system bus, aiming to confirm whether the basic control logic of the hardware responds normally.

[0074] If the second comparison result is parameter consistent, it indicates that the basic state register of the third hardware responds normally, at this time, the physical layer test processing needs to be further performed. The physical layer test processing is a signal integrity detection performed on the physical channel connected to the third hardware, which evaluates the link state by sending a specific test mode and analyzing the quality indicators of the returned signal. The test process includes but is not limited to error rate measurement, signal eye diagram analysis and timing tolerance verification, etc., and finally generates a test result containing quantitative indicators.

[0075] According to the test result, the hardware state of the third hardware is determined, that is, if the test result shows that the error rate exceeds the tolerance threshold or the signal synchronization fails, it is determined that the hardware state is existing to be updated fault, which usually indicates that the firmware version is too low; if all test indicators are within the normal range, it is determined to be no fault, which indicates that the previous abnormality may be caused by temporary interference.

[0076] If the second comparison result is parameter inconsistency, proceed to deep analysis of the event log. The event log is a collection of operation traces and abnormal events recorded by the third hardware during operation and stored in the non-volatile memory. Perform log analysis processing on the event log, i.e., scan the log entries according to the predefined parsing rules to identify whether there is abnormal data. Abnormal data refers to records that do not meet the preset log conditions, such as continuous communication timeout events, illegal instruction exceptions, or buffer overflow warnings, which often indicate deep logical errors or hardware compatibility problems.

[0077] If the analysis result indicates that there is abnormal data in the event log, determine that the fault state is an irreparable fault, as such errors usually point to hardware design defects or physical damage; if the analysis result indicates that there is no abnormal data in the event log, proceed to the firmware integrity verification phase. At this time, generate a second check code based on the firmware in the third hardware that exists in real time, obtain the real-time check value by reading the current firmware image and running the check algorithm, and at the same time, call the historical second check code (i.e., the baseline check value when the firmware is intact) for comparison processing to obtain the second check result.

[0078] If the second check result is check code inconsistency, it indicates that there is data difference between the firmware image and the original version, such as program code segment damage or configuration parameter disorder, but since the hardware basic function is still normal after the foregoing verification, it is determined that the firmware fault is a repairable fault, triggering the subsequent firmware recovery process.

[0079] Further, in order to facilitate understanding of the embodiments of the present application, the present application provides a schematic diagram of a firmware fault detection process of the third hardware, as shown in Figure 4 If the heartbeat signal is abnormal, further check whether the state register is normal, if normal, perform link layer PHY testing, if the error rate is less than the threshold, reset the heartbeat counter (reset the third hardware) and return to normal monitoring, if the error rate is greater than the threshold, start firmware update. If the state register is abnormal, further diagnose, read the log, if the log is abnormal, record the communication fault, if the log is normal, perform firmware CRC check, if the CRC matches, reset the chip, if the CRC does not match, mark the firmware as damaged, and start the firmware repair process.

[0080] The present application first screens the basic state through state register comparison, then performs physical layer testing (checks link fault) or log analysis (checks communication fault) according to the comparison result, and finally determines the firmware problem through check code comparison, which not only avoids misjudging hardware faults as repairable firmware faults, but also prevents over-treatment of temporary firmware interference as hardware replacement, significantly improving the accuracy and efficiency of fault detection, and ensuring the stability of server hard disk backplane data transmission.

[0081] In an implementation manner of the embodiment of the present application, when the hardware state of the third hardware is determined, the following manner can be adopted, but is not limited to: if the test result is that the error code rate is less than the preset error code rate threshold, it is determined that the hardware state of the third hardware is fault-free, wherein the error code rate is obtained by performing physical layer test processing on the third hardware; if the test result is that the error code rate is not less than the preset error code rate threshold, it is determined that the hardware state of the third hardware is that there is a fault to be updated, and the firmware in the third hardware with a firmware fault is subjected to firmware update processing.

[0082] In the embodiment of the present application, the test result is quantitative data obtained after performing physical layer test processing on the third hardware, which includes the error code rate. The error code rate refers to the ratio of the number of error bits to the total number of transmission bits in the data transmission process, which is obtained by applying a specific test mode sequence to the sending and receiving channels of the third hardware and comparing the difference between the sending and receiving data. The error code rate directly reflects the pros and cons of the signal transmission quality.

[0083] When the test result shows that the error code rate is less than the preset error code rate threshold, it indicates that the physical layer signal quality is within an acceptable range. The preset error code rate threshold is a threshold value preset according to the communication protocol specification and the system reliability requirement, and is stored in the configuration file of the management controller. At this time, it is determined that the hardware state of the third hardware is fault-free. This means that although the state parameter may have been detected to be abnormal before, the physical channel itself is intact, and the fault may be caused by temporary interference or software state abnormality, so hardware isolation or firmware update is not required, and other recovery means (such as link reset) can be used to continue normal operation.

[0084] On the contrary, when the test result shows that the error code rate is not less than the preset error code rate threshold, it indicates that there is a persistent signal integrity problem in the physical layer. At this time, it is determined that the hardware state of the third hardware is that there is a fault to be updated. The fault to be updated is a specific fault classification, which means that the hardware physical channel is basically intact, but its driving firmware or configuration parameter may not match the current physical environment, resulting in a decrease in signal quality, and the link performance needs to be recovered by updating the firmware.

[0085] For the case of being determined as the fault to be updated, firmware update processing is started on the firmware with a firmware fault in the third hardware. The processing process includes but is not limited to: suspending the service data flow through the third hardware, selecting the firmware image matching the current hardware model and version from the firmware library, writing the new firmware into the storage area of the third hardware through the SPI bus or other programming interface, performing verification and reset operation after completion, and finally re-performing physical layer test to verify the update effect. By dynamically adjusting the firmware to adapt to the change of the physical layer characteristics, an effective way is provided to solve the performance degradation problem caused by the mismatch between the firmware configuration and the physical environment.

[0086] The application avoids unnecessary hardware replacement (greatly reduces operation and maintenance cost and hardware waste) caused by misjudging firmware logic defects as hardware physical damage through hardware state judgment based on bit error rate and targeted processing strategy, accurately locates scenarios that need to be repaired through firmware update, solves the problem of high link transmission bit error rate through automatic firmware update, guarantees low error rate and high stability of server hard disk backplane data transmission, improves fault handling efficiency, and reduces system downtime caused by link problems.

[0087] In an implementable manner of the embodiment of the application, if the second check result indicates that the check codes are consistent, it means that the firmware image running in the current third hardware and the original intact version are completely matched at the data level, and there is no detectable code damage or configuration error, at this time, the fault state is determined as a reset-to-be-performed fault.

[0088] Specifically, the following method can also be used but is not limited to: if the first check result is that the check codes are consistent, the fault state is determined as a reset-to-be-performed fault, and the third hardware is reset for fault recovery of the firmware fault.

[0089] In the embodiment of the disclosure, for the third hardware determined as a reset-to-be-performed fault, reset processing is started. The third hardware is a hard disk expander chip, which is a key component that undertakes data routing and protocol conversion. It may appear functional abnormalities due to signal conflict, timing deviation or state machine anomaly during operation. Reset processing is an operation of forcibly restoring the chip to the initial state through hardware signals or software commands, which is specifically implemented by the management controller sending a reset pulse through a dedicated reset pin or writing a specific sequence to a reset control register through a configuration interface. This process will clear all temporary state data of the third hardware, reset the internal logic unit and reinitialize each functional module, thereby eliminating functional abnormalities caused by temporary factors.

[0090] The application realizes fault recovery of the firmware fault through reset processing, restores the hardware to normal operation through state reset without relying on firmware reprogramming, effectively solves the problem of functional abnormalities caused by temporary factors. After the reset operation is executed, the state register parameters and link connection state of the third hardware are monitored again to verify whether its function is restored to normal. This significantly improves the self-healing efficiency of the system, avoids unnecessary firmware refreshing operation caused by temporary faults, and at the same time guarantees the requirement of business continuity of the storage system.

[0091] In an implementable manner of the embodiment of the application, when extracting the backup firmware from the pre-device firmware library, the following method can be used but is not limited to: obtaining the hardware identifier corresponding to the fault hardware, performing comparison processing from the pre-device firmware library according to the hardware identifier, and obtaining the backup firmware.

[0092] In the embodiments of the present application, the pre-device firmware library is a firmware storage warehouse pre-established in the server, usually located in the protected storage area of the baseboard management controller (BMC) or independent non-volatile memory. The pre-device firmware library systematically stores various firmware images suitable for different hardware models and versions, and establishes a perfect index management mechanism to ensure that the required firmware can be quickly and accurately located when needed.

[0093] The hardware identifier is coded information for uniquely identifying a specific hardware component, usually including board card type identifier (ID), chip model code, hardware version number and other metadata. The identifier information is stored in the read-only memory or field replaceable unit of the hardware and can be read through the management bus. For example: for a complex programmable logic device (CPLD), its identifier may contain vendor code and logic version; for a hard disk expander chip, it may contain device ID and interface protocol version.

[0094] After obtaining the hardware identifier, comparison processing is performed in the pre-device firmware library according to the identifier, that is, the obtained hardware identifier is matched with the index table pre-established in the firmware library. Each record in the index table is associated with a specific hardware identifier and the corresponding firmware storage address, checksum and compatibility information. The comparison algorithm will traverse the index table to find a record that completely matches the current hardware identifier or meets the backward compatibility condition, and finally determine the most suitable firmware image as the backup firmware.

[0095] Through the intelligent matching mechanism based on the hardware identifier, the present application can accurately select the correct firmware version that completely corresponds to the faulty hardware from the firmware library that may contain multiple versions and multiple models, effectively avoiding recovery failure or compatibility problems caused by firmware version mismatch, laying a foundation for firmware programming recovery, significantly improving the reliability and automation degree of the firmware recovery process, and reducing errors that may be introduced by manual intervention.

[0096] In one implementation manner of the embodiments of the present application, when the firmware recovery processing is performed on the firmware with firmware failure by using the backup firmware, the following manner can be adopted, but is not limited to: suspending the input and output operation of the service domain affected by the firmware failure, wherein the service domain is the service function corresponding to the faulty hardware; if the faulty hardware is the second hardware, performing the firmware recovery processing on the firmware with firmware failure by using the backup firmware based on the first programming mode and the preset test clock frequency; if the faulty hardware is the third hardware, performing the firmware recovery processing on the firmware with firmware failure by using the backup firmware in the safe mode of the third hardware based on the second programming mode.

[0097] In the embodiments of the present application, performing specific firmware programming operations requires differentiated programming strategies according to the type of the faulty hardware, and requires suspending the input and output operations of the service domain affected by the faulty hardware. The service domain refers to a specific functional range supported by the faulty hardware, for example, if the faulty hardware is a CPLD responsible for the in-place detection of a hard disk, its service domain may cover the hard disk state monitoring and indicator light control; if it is a hard disk expander chip responsible for data forwarding, its service domain involves all hard disk read-write channels passing through the chip. Suspending the input and output operations is to temporarily suspend all data transmission requests passing through the service domain through system management interrupts or bus isolation technology, to ensure that data corruption or program conflicts caused by concurrent access do not occur during firmware recovery, and to create a stable environment for safe firmware update.

[0098] If the faulty hardware is identified as the second hardware, i.e., a complex programmable logic device (CPLD), the recovery will be implemented based on the first programming mode. The first programming mode is, for example, an online programming mode through a JTAG interface. In this mode, a preset test clock frequency is first set, which is a JTAG interface communication rate determined according to the CPLD model characteristics and stored in the configuration file of the management controller. Using an appropriate clock frequency is crucial for successful programming, and a frequency that is too high may cause signal timing violations, and a frequency that is too low will prolong the recovery time. During the recovery process, the programmer establishes communication with the CPLD through the JTAG interface according to the preset clock frequency, and sequentially performs standard operation processes such as erasing the configuration memory, writing the backup firmware data, and verifying the programming results, to finally complete the firmware recovery.

[0099] If the faulty hardware is identified as the third hardware, i.e., a hard disk expander chip, the system will implement recovery based on the second programming mode. The second programming mode is, for example, a programming mode through an SPI bus to program the Flash memory connected to the chip. The recovery process needs to be performed in the safe mode of the third hardware, which is a special working state of the hard disk expander chip, in which state the chip suspends all normal business functions and only retains the minimum firmware update interface, thereby avoiding interference from business traffic during programming. Entering the safe mode usually requires a specific pin combination or writing to a special control register. In the safe mode, the Flash chip storing the firmware is directly accessed through the SPI bus to perform sector erasing, block writing of backup firmware, reading verification, and other operations, to ensure complete firmware recovery.

[0100] Further, in order to facilitate understanding of the embodiments of the present application, the present application provides a flowchart of firmware recovery, as shown in Figure 5As shown in the figure, the firmware repair is divided into two programming modes, and the CPLD usually adopts the JTAG programming mode, that is, the firmware information of the CPLD is burned through the JTAG interface, and the MCU chip such as the hard disk controller chip adopts the SPI programming mode, that is, the Flash is communicated through the SPI bus to realize the read and write operation. The firmware repair process is as follows: first, suspend the service domain I / O operation affected by the fault, load the backup firmware, if it is the JTAG programming mode, set the test clock frequency, erase the configuration area, burn the firmware, if it is the SPI flash programming mode, set the Flash chip to the programming mode, erase the sector, write in blocks, and after completion, perform chip function self-checking, if the self-checking fails, take hardware isolation, and if it passes, reset the chip and restore the service.

[0101] The application suspends the service domain input and output operation, eliminates the conflict between the firmware recovery and the service data transmission, guarantees the integrity of the storage data, adapts the programmable logic architecture of the CPLD to the JTAG programming mode of the second hardware and the preset test clock frequency, ensures the stability and compatibility of the firmware burning, matches the communication logic of the hard disk expander chip and the Flash chip through the SPI programming mode and the security mode of the third hardware, isolates the service interference through the security mode, improves the security of the recovery process, ensures that different types of faulty hardware can obtain an adaptive repair path through the differentiated and step-by-step firmware recovery scheme, guarantees the data security and service continuity in the repair process, and effectively improves the overall reliability of the server hard disk backplane firmware fault repair.

[0102] In an implementable manner of the embodiment of the application, after the backup firmware completes the firmware recovery processing on the faulty firmware, a function verification process of the repaired hardware needs to be started to verify the firmware recovery effect through function detection, so as to ensure that the repaired hardware can stably undertake the original service function and avoid secondary faults or service abnormalities caused by incomplete recovery; at the same time, the hardware that fails to pass the detection is timely physically isolated to prevent the fault from spreading and affecting the overall operation of the server hard disk backplane.

[0103] Specifically, the following methods can also be used, but are not limited to: performing function detection processing on the repaired hardware to obtain a first function detection result, wherein the repaired hardware is the faulty hardware after the firmware recovery processing; if the first function detection result meets the preset function condition, starting the service function corresponding to the repaired hardware; if the first function detection result does not meet the preset function condition, physically isolating the repaired hardware.

[0104] In the embodiment of the application, the repaired hardware refers to the faulty hardware that has just completed the firmware recovery processing, and the firmware image thereof has been updated but still needs to confirm the function integrity.

[0105] The function detection processing performed on the repaired hardware is an automatic test for the core capability of the hardware. The detection content includes, but is not limited to, register read-write verification, basic instruction execution test, internal state machine logic check, and communication handshake test with associated components. For example, for a complex programmable logic device, the logic unit response and signal driving capability are detected; for a hard disk expander chip, the link establishment process and data forwarding function are verified. This test item is pre-set based on the hardware design specification to form a complete detection kit.

[0106] The first function detection result obtained after the function detection processing is the basis for judging the recovery effect. The first function detection result is generated by comparing the actual test output with the expected behavior mode, and is specifically manifested as the pass / fail state summary of each test item. The result is compared with the preset function condition, which is a set of technical specifications that the hardware should meet when working normally, including response time threshold, signal quality standard, and protocol compliance requirement, and is stored in the policy library of the management controller.

[0107] If the first function detection result meets the preset function condition, it indicates that all function indicators of the repaired hardware have been restored to normal, and the firmware recovery processing has achieved the expected effect. At this time, the system will start the service function corresponding to the repaired hardware, that is, gradually restore the business responsibilities undertaken by the hardware. The starting process includes, but is not limited to, reactivating the working mode of the hardware, restoring its connection with the service bus, re-registering the interrupt service routine, and finally releasing the suspension of input and output operations of the related service domain, so that the hardware re-integrates into the system and restores the complete service capability.

[0108] If the first function detection result does not meet the preset function condition, it indicates that although the firmware has been updated, the hardware still has functional defects. This anomaly may be caused by firmware version mismatch, un-eliminated hardware physical damage, or new problems introduced in the programming process. In this case, the repaired hardware will be subjected to physical isolation processing, that is, through the fuse protection circuit, turning off the power supply, or switching to the standby hardware, etc., to completely isolate the hardware from the system running environment, so as to prevent its functional anomaly from affecting the overall system stability. At the same time, the system will record the recovery failure details in the management log to provide diagnostic basis for subsequent manual intervention.

[0109] The present application forms a complete fault handling closed loop through the function detection after firmware recovery processing, service starting / physical isolation process. The function detection ensures that only the hardware that has been truly restored to normal will be put into use again, avoiding secondary faults. The service starting process guarantees business continuity, reduces the impact of the fault on the storage system for a long time. The physical isolation quickly blocks the spread of the fault, protects the safety of other components of the system, and strengthens the stability and risk resistance of the storage system.

[0110] In an implementation manner of the embodiment of the present application, potential abnormalities are identified in advance and the occurrence of faults is accurately determined by real-time monitoring and standard comparison of the state parameters of the hard disk backplane, so that the problem caused by the failure to be discovered in time is avoided, and the foundation for subsequent fault type judgment and targeted processing is laid.

[0111] Specifically, the following methods can be used, but are not limited to: monitoring the state parameters of the hard disk backplane; if the state parameters do not conform to the preset parameter specification, it is determined that the hard disk backplane has a fault.

[0112] In the embodiment of the present application, the state parameters of the hard disk backplane are continuously monitored by the various sensors and monitoring chips deployed on the hard disk backplane. The state parameters constitute a multi-dimensional index system for evaluating the health of the backplane, including but not limited to: voltage and current values of power input points, junction temperature of key chips (such as CPLD, hard disk expander), signal quality of each interface connector, and chip internal state register values read by the management controller through the Inter-Integrated Circuit (I2C) bus. The monitoring process is periodically polled by the Baseboard Management Controller (BMC) or responds to sudden parameter changes through hardware interrupts, ensuring the real-time and comprehensiveness of data collection.

[0113] The collected state parameters are compared with the preset parameter specification in real time. The preset parameter specification is a set of technical standards stored in the non-volatile memory of the management controller, which defines the numerical range, trend threshold and duration limit that each state parameter should meet in the normal working state. For example: the voltage specification may stipulate that the normal fluctuation range of 12V power input is ±5%, the temperature specification may set the upper limit of the working temperature of the chip to 85℃, and the state register specification clearly defines the valid value combination of each function flag bit.

[0114] When the monitoring system finds that the state parameters do not conform to the preset parameter specification, it is determined that the parameters are abnormal. The non-conformance may be manifested as numerical over-limit (such as voltage dropping to below 10V), trend abnormality (such as temperature rising by 2℃ per minute continuously) or state contradiction (such as hard disk in-place signal valid but no read-write activity). When such abnormalities are detected, it is determined that the hard disk backplane has a fault, thereby triggering the subsequent complete fault handling process.

[0115] The present application realizes the transformation from passive response to active prevention through the early fault detection mechanism based on parameter specification comparison, and provides time for the system to take measures before the fault influence expands.

[0116] In an implementable manner of the embodiment of the present application, when judging the fault type of the hard disk backplane, the following manner can be adopted, but is not limited to: inputting the state parameter into a preset fault model for parameter analysis processing to obtain a parameter analysis result; if the parameter analysis result is power parameter abnormality, it is determined that the fault type is hardware failure; if the parameter analysis result is signal parameter abnormality, it is determined that the fault type is firmware failure.

[0117] In the embodiment of the present application, the judgment process first inputs the acquired state parameter into a preset fault model for parameter analysis processing. The preset fault model is an intelligent diagnosis algorithm pre-established in the management system, which is generated based on a large number of historical fault data and can identify the parameter characteristic mode corresponding to different fault types. The preset fault model usually includes decision tree, neural network or pattern matching rule and other analysis logics, and is stored in a special storage area of the management controller. The parameter analysis processing is a deep analysis process of the preset fault model on the input parameter, including data normalization, feature extraction, pattern matching and probability calculation and other steps, and finally generates a structured parameter analysis result, which clearly indicates the fault category corresponding to the current parameter abnormality.

[0118] Power parameter abnormality refers to that an index related to power supply deviates from a normal range, for example: the main power input voltage continuously below the threshold value, the output current of the adjustment module seriously exceeding the standard or the temperature of the power supply chip sharply increasing and other phenomena. Such abnormality is usually directly related to the failure of physical components, such as: power management integrated circuit burning, filter capacitor damage or circuit board line fracture, and is therefore classified as hardware failure, which needs to take hardware-level processing measures such as physical isolation.

[0119] Signal parameter abnormality mainly refers to that an index related to data communication and logic control appears abnormal, for example: the signal integrity index on the inter-chip communication bus deteriorates, the number of protocol layer handshake timeouts increases, the state machine jump sequence is disordered or the configuration register read-write verification fails and other phenomena. Such abnormality is usually caused by firmware code execution error, configuration data damage or logic timing conflict, and the hardware physical connection can still remain intact, and is therefore classified as firmware failure, which is suitable for recovery through software means such as firmware update or reset.

[0120] The present application can more accurately identify the nature of the fault by analyzing the parameter characteristic mode rather than simple threshold comparison, thereby providing a reliable basis for subsequent differentiated processing strategies, and significantly improving the accuracy of fault diagnosis and the intelligent level of system maintenance.

[0121] In an implementable manner of the embodiment of the present application, in order to facilitate understanding of the process of the fault processing method in the present application, the embodiment of the present application further provides an example for illustration: as shown in Figure 6 Figure 6 ​Another fault detection and processing flow diagram provided by the embodiment of the application is shown in the figure. First, common faults are classified and preset in the BMC. The BMC monitors the hard disk backplane in real time, monitors the voltage, current and temperature of the hard disk backplane, and the state of the main chip CPLD and the hard disk expander chip. If a fault occurs in the backplane, the fault type is determined. If it is a power supply fault, it is usually not repairable, so the power supply fault position is located, the fault power supply is fused and powered off, and the record is recorded in the BMC log for subsequent maintenance and replacement. If it is not a power supply fault, it is determined whether it is a CPLD fault and a hard disk expander chip fault. If it is, it is determined whether it is repairable. If it is a physical chip damage, it is not repairable. If it is a non-volatile area damage, it can be repaired by re-flashing the firmware. If it is determined to be an unrepairable fault, the fault is recorded in the log for subsequent repair. If it is a firmware fault and a repairable fault, the firmware repair process is started. After repair, a function self-check is performed. If the self-check fails, hardware isolation is adopted. If it passes, the chip is reset and the service is restored.

[0122] Meanwhile, the fault state can be indicated by a light emitting diode (LED) indicator light on the board card. The correspondence between the fault state and the LED display mode is shown in Table 1.

[0123] Table 1

[0124]

[0125] In one implementation manner of the embodiment of the application, in order to facilitate understanding of the process of firmware recovery in the application, the embodiment of the application also provides an example for illustration: as shown in Figure 7 , Figure 7 A multi-backplane firmware recovery flow diagram provided by the embodiment of the application is shown in the figure. Usually, a server can be equipped with multiple types and quantities of hard disk backplanes. The CPLD on the backplane can use the detection method and fault processing method of the CPLD. The MCU chip on the backplane can use the fault detection method and fault processing method of the hard disk expander chip. The board card type ID and chip ID can be identified to determine the firmware information that needs to be reprogrammed. The firmware recovery process is as follows: suspend the service domain I / O operation affected by the fault, read the board card type ID and chip type ID, select the backup firmware, load the backup firmware, and select the programming mode according to the chip ID for programming.

[0126] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiment can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better implementation manner.

[0127] The embodiment of the application also provides a fault processing device,Figure 8 A structural schematic diagram of an apparatus for fault processing provided in the present application is shown in FIG. 1, which comprises: Figure 8

[0128] A judging unit 81 is configured to determine a fault type of the hard disk backboard according to the monitored state parameters of the hard disk backboard in response to a fault of the server hard disk backboard.

[0129] A processing unit 82 is configured to perform physical isolation processing on first hardware corresponding to the hardware fault and record fault information of the hardware fault in response to the fault type being the hardware fault.

[0130] A determining unit 83 is configured to determine a fault hardware corresponding to the firmware fault according to the state parameters in response to the fault type being the firmware fault.

[0131] A detecting unit 84 is configured to detect a fault state of the firmware fault based on register parameters, check codes and fault detection strategies of different fault hardware respectively, wherein the state parameters comprise the register parameters, and the fault hardware comprises second hardware and third hardware.

[0132] An extracting unit 85 is configured to extract backup firmware from a pre-device firmware library in response to the fault state being a repairable fault.

[0133] A restoring unit 86 is configured to perform firmware recovery processing on firmware with firmware fault by the backup firmware based on firmware programming strategies of different fault hardware respectively, wherein the firmware programming strategies at least comprise a programming strategy based on a first programming mode and a preset test clock frequency, or a programming strategy based on a second programming mode in a security mode of the third hardware.

[0134] In an embodiment of the present application, the detecting unit 84 is specifically configured to:

[0135] If the fault hardware is the second hardware, the fault state of the firmware fault is detected according to first state register parameters and a first check code corresponding to the second hardware, wherein the register parameters comprise the first state register parameters and second register parameters, and the check code comprises the first check code and a second check code.

[0136] If the fault hardware is the third hardware, the fault state of the firmware fault is detected according to second state register parameters, an event log and a second check code corresponding to the third hardware, wherein the state parameters comprise the second state register parameters and the event log.

[0137] In an embodiment of the present application, the detecting unit 84 is specifically configured to:

[0138] The first state register parameters are compared with first preset register parameters to obtain a first comparison result.​

[0139] If the first comparison result is that the parameters are inconsistent, it is determined that the fault state is an irreparable fault, and fault information of the second hardware is recorded;

[0140] If the first comparison result is that the parameters are consistent, a first check code is generated in real time based on firmware with a firmware fault in the second hardware, and the first check code is compared with a historical first check code to obtain a first check result, wherein the historical first check code is a check code generated by the firmware with the firmware fault before the fault occurs;

[0141] If the first check result is that the check codes are inconsistent, it is determined that the fault state is a repairable fault;

[0142] If the first check result is that the check codes are consistent, it is determined that the fault state is a reset-to-be-performed fault, and the second hardware is reset to recover from the firmware fault.

[0143] In an embodiment of the present application, the detection unit 84 is specifically configured to:

[0144] perform parameter comparison processing on the second state register parameter and a second preset register parameter to obtain a second comparison result;

[0145] If the second comparison result is that the parameters are consistent, physical layer test processing is performed on the third hardware to obtain a test result, and the hardware state of the third hardware is determined according to the test result, wherein the hardware state is that there is a fault to be updated or no fault;

[0146] If the second comparison result is that the parameters are inconsistent, log analysis processing is performed on the event log to obtain an analysis result, and in response to the analysis result being that there is abnormal data in the event log, it is determined that the fault state is an irreparable fault, wherein the abnormal data is data in the event log that does not meet a preset log condition;

[0147] If the analysis result is that there is no abnormal data in the event log, a second check code is generated in real time based on firmware with a firmware fault in the third hardware, and the second check code is compared with a historical second check code to obtain a second check result;

[0148] If the second check result is that the check codes are inconsistent, it is determined that the firmware fault is a repairable fault;

[0149] If the first check result is that the check codes are consistent, it is determined that the fault state is a reset-to-be-performed fault, and the third hardware is reset to recover from the firmware fault.

[0150] In an embodiment of the present application, the detection unit 84 is specifically configured to:

[0151] If the test result is that the error code rate is less than the preset error code rate threshold, it is determined that the hardware state of the third hardware is fault-free, wherein the error code rate is obtained by performing physical layer test processing on the third hardware.

[0152] If the test result is that the error code rate is not less than the preset error code rate threshold, it is determined that the hardware state of the third hardware is that there is a fault to be updated, and the firmware update processing is performed on the firmware with a firmware fault in the third hardware.

[0153] In an embodiment of the present application, the extraction unit 85 is specifically configured to:

[0154] Obtain the hardware identifier corresponding to the faulty hardware, and perform comparison processing on the backup firmware from the pre-device firmware library according to the hardware identifier to obtain the backup firmware.

[0155] In an embodiment of the present application, the recovery unit 86 is specifically configured to:

[0156] Suspend the input and output operation of the service domain affected by the firmware fault, wherein the service domain is a service function corresponding to the faulty hardware;

[0157] If the faulty hardware is the second hardware, the firmware recovery processing is performed on the firmware with a firmware fault by the backup firmware based on the first programming mode and the preset test clock frequency;

[0158] If the faulty hardware is the third hardware, the firmware recovery processing is performed on the firmware with a firmware fault by the backup firmware based on the second programming mode in the safe mode of the third hardware.

[0159] In an embodiment of the present application, the recovery unit 86 is further configured to:

[0160] Perform function detection processing on the repaired hardware to obtain a first function detection result, wherein the repaired hardware is the faulty hardware after the firmware recovery processing;

[0161] If the first function detection result meets the preset function condition, the service function corresponding to the repaired hardware is started;

[0162] If the first function detection result does not meet the preset function condition, the repaired hardware is subjected to physical isolation processing.

[0163] In an embodiment of the present application, the judgment unit 81 is further configured to:

[0164] Monitor the state parameters of the hard disk backplane;

[0165] If the state parameters do not meet the preset parameter specification, it is determined that the hard disk backplane has a fault.

[0166] In an embodiment of the present application, the processing unit 82 is specifically configured to:

[0167] inputting the state parameter into a preset fault model to perform parameter analysis processing, to obtain a parameter analysis result;

[0168] if the parameter analysis result is a power parameter abnormality, determining that the fault type is a hardware fault;

[0169] if the parameter analysis result is a signal parameter abnormality, determining that the fault type is a firmware fault.

[0170] The description of the features in the embodiments of the device for fault processing can be referred to the related description of the embodiments of the method for fault processing, which will not be repeated here.

[0171] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above-mentioned embodiments of the method for fault processing.

[0172] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned embodiments of the method for fault processing when running.

[0173] In an example embodiment, the above-mentioned computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0174] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for fault processing.

[0175] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for fault processing.

[0176] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the claimed application. Accordingly, modifications and / or additions, other than those explicitly described herein, can be obvious to those skilled in the art in the light of this disclosure. The claimed application is intended to embrace all such modifications and / or additions.

[0177] The method for processing faults and the electronic device provided by the application are described in detail above. The principles and implementation manners of the application are described by applying specific examples in the present disclosure. The above description of the examples is only used to help understand the method of the application and its core idea. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A method for troubleshooting, characterized in that, include: In response to a server hard disk backplane failure, the type of failure of the hard disk backplane is determined based on the monitored status parameters of the hard disk backplane. In response to the fact that the fault type is a hardware fault, the first hardware corresponding to the hardware fault is physically isolated, and the fault information of the hardware fault is recorded. In response to the fault type being a firmware fault, the faulty hardware corresponding to the firmware fault is determined based on the status parameters, and the faulty hardware includes a second piece of hardware and a third piece of hardware. If the faulty hardware is the second hardware, then the fault status of the firmware fault is detected according to the first status register parameters corresponding to the second hardware and the first check code, wherein the status parameters include the first status register parameters; If the faulty hardware is the third hardware, the fault status of the firmware fault is detected according to the second status register parameters, event log, second check code and physical layer test results corresponding to the third hardware, wherein the status parameters include the second status register parameters and the event log; In response to the fault state being a repairable fault, backup firmware is extracted from a preset backup firmware library, and firmware recovery processing is performed on the firmware with the firmware fault based on the firmware programming strategy corresponding to each of the different faulty hardware. The firmware programming strategy includes at least a programming strategy based on a first programming mode and a preset test clock frequency, or a programming strategy based on a second programming mode in the security mode of the third hardware.

2. The fault handling method according to claim 1, characterized in that, The step of detecting the firmware fault state based on the first status register parameters corresponding to the second hardware and the first checksum includes: The parameters of the first status register are compared with the parameters of the first preset register to obtain the first comparison result. If the first comparison result is that the parameters are inconsistent, then the fault status is determined to be an unrepairable fault, and the fault information of the second hardware is recorded; If the first comparison result is that the parameters are consistent, the first verification code is generated in real time based on the firmware with the firmware fault in the second hardware, and the first verification code is compared with the historical first verification code to obtain the first verification result. The historical first verification code is the verification code generated by the firmware with the firmware fault before the fault occurred. If the first verification result is that the verification code is inconsistent, then the fault status is determined to be a repairable fault. If the first verification result is that the verification code matches, then the fault state is determined to be a fault to be reset, and the second hardware is reset to recover the firmware fault.

3. The fault handling method according to claim 1, characterized in that, The step of detecting the firmware fault state based on the second status register parameters, event log, second checksum, and physical layer test results corresponding to the third hardware includes: The parameters of the second status register are compared with the parameters of the second preset register to obtain the second comparison result; If the second comparison result is that the parameters are consistent, then the third hardware is subjected to physical layer testing to obtain the test result, and the hardware status of the third hardware is determined according to the test result, wherein the hardware status is either a fault that needs to be updated or no fault. If the second comparison result is that the parameters are inconsistent, then the event log will be analyzed to obtain the analysis result. In response to the analysis result that there is abnormal data in the event log, the fault status will be determined to be an unrepairable fault, wherein the abnormal data is data in the event log that does not meet the preset log conditions. If the analysis result indicates that the abnormal data is not found in the event log, then the second verification code is generated in real time based on the firmware with the firmware fault in the third hardware, and the second verification code is compared with the historical second verification code to obtain the second verification result. If the second verification result is that the verification code is inconsistent, then the firmware fault is determined to be a repairable fault. If the second verification result is that the verification code matches, then the fault state is determined to be a fault to be reset, and the third hardware is reset to recover the firmware fault.

4. The fault handling method according to claim 3, characterized in that, Determining the hardware status of the third hardware based on the test results includes: If the test result is that the bit error rate is less than the preset bit error rate threshold, then the hardware status of the third hardware is determined to be fault-free, wherein the bit error rate is obtained by performing physical layer testing on the third hardware. If the test result indicates that the bit error rate is not less than the preset bit error rate threshold, then the hardware status of the third hardware is determined to be a fault that needs to be updated, and the firmware with the firmware fault in the third hardware is updated.

5. The fault handling method according to claim 1, characterized in that, The step of extracting backup firmware from the preset backup firmware library includes: Obtain the hardware identifier corresponding to the faulty hardware, and perform a comparison process from the preset backup firmware library based on the hardware identifier to obtain the backup firmware.

6. The fault handling method according to claim 1, characterized in that, The firmware recovery process for firmware with firmware faults, based on the firmware programming strategies corresponding to the different faulty hardware, and using the backup firmware, includes: Suspend input / output operations of the service domain affected by the firmware failure, wherein the service domain is the service function corresponding to the faulty hardware; If the faulty hardware is the second hardware, then based on the first programming mode and the preset test clock frequency, the firmware with the firmware fault is restored using the backup firmware. If the faulty hardware is the third hardware, then based on the second programming mode in the security mode of the third hardware, the firmware with the firmware fault is restored using the backup firmware.

7. The fault handling method according to claim 6, characterized in that, After performing firmware recovery processing on the firmware with the firmware fault using the backup firmware based on the firmware programming strategies corresponding to the different faulty hardware, the method further includes: The repaired hardware is subjected to functional testing to obtain a first functional testing result, wherein the repaired hardware is the faulty hardware after firmware recovery. If the first function detection result meets the preset function conditions, then the service function corresponding to the repaired hardware is activated; If the first function detection result does not meet the preset function conditions, then the repaired hardware will be physically isolated.

8. The fault handling method according to claim 1, characterized in that, The step of determining the fault type of the hard drive backplane based on the monitored status parameters of the hard drive backplane includes: The state parameters are input into a preset fault model for parameter analysis and processing to obtain the parameter analysis results; If the parameter analysis result indicates abnormal power parameters, then the fault type is determined to be a hardware fault. If the parameter analysis result indicates abnormal signal parameters, then the fault type is determined to be a firmware fault.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault handling method as described in any one of claims 1 to 8 when executing the computer program.

Images