Fault processing method, electronic device, storage medium, and program product

Abstract

This application discloses a fault handling method, electronic device, storage medium, and program product, relating to the field of fault handling technology. The method includes: when the real-time status of the server meets detection conditions, identifying the actual response status of the target processor using its first communication address; if the actual response status is non-responsive, determining that the target processor is in a fault state and generating a recovery instruction for controlling the target processor's self-loading; sending the recovery instruction to the target processor using its second communication address; and determining that the target processor exits the fault state upon responding to the recovery instruction, thus completing the fault handling. This solves the problems of reliance on dedicated hardware, susceptibility to accidental triggering, lack of security mutual exclusion, low success rate, and lack of traceability in related technologies, achieving the technical effect of detecting and recovering from CPLD self-loading failures without modifying the hardware structure.

Description

Technical Field

[0001] This application relates to the field of fault handling technology, and in particular to a fault handling method, electronic device, storage medium and program product. Background Technology

[0002] In related technologies, CPLDs (Complex Programmable Logic Devices) exhibit a typical fault: after AC power-on, the CPLD fails to complete the self-loading process, resulting in the loss of regular communication addresses. There are several solutions for recovering from CPLD startup failures. For example, repairing the CPLD via GPIO pins or online upgrade packages; monitoring the system's operational status at each startup stage using a watchdog mechanism, triggering a system reset or switching to an alternative startup path upon detecting a timeout; and simulating hot-plug events to perform power-on and power-off operations on the faulty device to achieve fault recovery, etc.

[0003] However, in related technologies, the solutions for recovery processing after CPLD boot failure usually rely on hardware resources, making the solutions difficult to reuse and lacking universality; moreover, their triggering conditions are relatively simple, and the risk of false triggering is extremely high; they lack safe judgment on the timing of the repair operation, and performing reloading at an inappropriate time may damage the running logic and cause secondary failures. The logic of a single repair is also greatly affected by instantaneous bus interference, resulting in a low repair success rate, which urgently needs to be solved. Summary of the Invention

[0004] This application provides a fault handling method, electronic device, storage medium, and program product to at least solve the problems in the related art, such as the triggering mechanism of the recovery handling scheme after CPLD startup failure relying on dedicated hardware and having a single triggering condition, poor universality and easy to be falsely triggered, lacking a safety judgment mechanism, which may cause secondary failures when performing repair at an inappropriate time, and the low success rate of the single-attempt repair logic.

[0005] This application provides a fault handling method, comprising the following steps: determining the real-time status of the server, and when the real-time status meets the detection conditions, identifying the actual response status of the target processor using the first communication address of the target processor; when the actual response status is a non-response status, determining that the target processor is in a fault state, and generating a recovery instruction for controlling the self-loading of the target processor; sending the recovery instruction to the target processor using the second communication address of the target processor, and when the target processor responds to the recovery instruction, determining that the target processor exits the fault state, thus completing the fault handling.

[0006] This application also provides a fault handling device, comprising: an identification module, configured to determine the real-time status of a server, and, when the real-time status meets detection conditions, identify the actual response status of the target processor using a first communication address of the target processor; a first generation module, configured to determine that the target processor is in a fault state when the actual response state is a non-response state, and generate a recovery instruction for controlling the target processor to self-load; and a processing module, configured to send the recovery instruction to the target processor using a second communication address of the target processor, and, when the target processor responds to the recovery instruction, determine that the target processor exits the fault state, thus completing the fault handling.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described fault handling methods when executing the computer program.

[0008] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0010] This application enables the detection of whether the CPLD is in a fault state when the real-time status of the server meets certain conditions, and generates a recovery command when it is determined that the CPLD is in a fault state, so as to force the CPLD to re-execute the self-loading process, and finally determines that the CPLD exits the fault state if the verification is successful. This solution addresses the shortcomings of existing CPLD boot failure recovery schemes, which rely on dedicated hardware and have limited triggering conditions, resulting in poor versatility and susceptibility to false triggering. Furthermore, these schemes lack a safety judgment mechanism, potentially leading to secondary faults when attempting repair at inappropriate times. The single-attempt repair logic also suffers from low success rates. This solution utilizes a dual I2C (Inter-Integrated Circuit) address architecture (regular address + UFM address) with a three-layer judgment mechanism, ensuring repair is triggered only in safe scenarios of AC power-on and power-off. A tiered retry and result verification mechanism forms a closed loop of "fault detection – fault repair – result verification," with records added at each stage to create a three-level fault log for traceability. The entire process is implemented purely in software, requiring no hardware modifications. Only the BMC initialization script needs modification, resulting in low cost, strong compatibility, and widespread applicability to multi-board hardware systems with CPLDs, such as servers, storage devices, and industrial control computers. This significantly improves the reliability, safety, and versatility of server CPLD boot failure and recovery. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a fault handling method provided according to an embodiment of this application; Figure 2 This is a flowchart illustrating the power-on process of a complete machine according to an embodiment of this application; Figure 3 This is a schematic diagram of a dual I2C address communication hardware architecture of BMC and CPLD according to an embodiment of this application; Figure 4 This is a schematic diagram of the hierarchical nodes and corresponding operations of fault log recording according to one embodiment of this application; Figure 5 This is an overall flowchart of the BMC performing fault detection and fault repair according to one embodiment of this application; Figure 6 This is a schematic diagram of a fault handling device provided according to an embodiment of this application.

[0013] Figure label: Among them, 10 is a fault handling device; 100 is an identification module; 200 is a first generation module; and 300 is a processing module. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Before explaining the fault handling methods in the embodiments of this application, the root causes of typical CPLD faults in actual systems will be explained first.

[0018] CPLD evolved from PAL and GAL devices and is a large-scale integrated circuit. Users can construct the logic function of digital integrated circuits according to their own needs, generate target files using integrated development software platforms, and transfer the code to the target chip through a download cable to realize the digital system design. It is a large-scale and complex device.

[0019] In the hardware system, the CPLD is the core logic device that controls peripherals such as backplane, fan board, power board, and switching board. Its responsibilities include, but are not limited to: hard disk power supply enable control; power-on sequence management of the whole machine; backplane presence detection; sensor data acquisition; and status interaction with BMC (Baseboard Management Controller).

[0020] For the CPLD to function properly, it must first successfully complete the SDM (self-download mode) self-loading process after the entire unit is powered on via AC. Only after successful loading will the CPLD open its regular communication address, allowing the BMC (Baseboard Management Controller) to perform device enumeration, status reading, and control distribution via I2C.

[0021] However, in actual systems, a typical fault is common: after AC power-on, the CPLD cannot complete the SDM process, resulting in the loss of the regular communication address, but the UFM (User flash Memory, the location reserved by the CPLD manufacturer for the storage of upgrade firmware for developers) address is still accessible.

[0022] The root cause of this fault is the inconsistent power supply timing between the peripheral board and the motherboard standby power supply (P3V3_STBY): the peripheral board powers on faster than the motherboard, which causes the CPLD's I2C signal to enter an intermediate level between high and low levels at the moment of startup.

[0023] The CPLD's internal state machine mistakenly interprets this intermediate state level as an I2C start signal or bus activity, thus prematurely terminating the SDM self-loading process and entering a waiting instruction state. Ultimately, this leads to the disappearance of the regular communication address, the hard drive not powering on, and the entire machine failing to boot.

[0024] In related technologies, solutions for recovering from CPLD boot failures also have significant shortcomings: The triggering mechanism relies on dedicated hardware and has poor universality: the technical solutions in related technologies mostly rely on specific registers of BMC, fixed GPIO (General Purpose Input / Output), and dedicated CPLD flag bits. When the hardware platform changes (such as changing the BMC platform, changing the motherboard, or changing the CPLD model), the solution immediately becomes invalid, making it difficult to reuse and resulting in poor universality. Without scenario differentiation, the risk of accidental triggering is high: most of the technical solutions in the related technologies are executed unconditionally or under single conditions. Such unconditional or single-condition triggering will cause non-target scenarios such as BMC restart, DC restart, restart during operation, and manual power-on to trigger fault repair, which is extremely risky and may further lead to problems such as hard drive power failure, business interruption, and board disconnection. Lack of secure mutual exclusion mechanism: The technical solutions in the related technologies do not have a secure mutual exclusion mechanism. Sending a reload command to the CPLD in the power-on state will destroy the running logic, leading to secondary failures such as bus deadlock, hard disk recognition failure, and abnormal system reset. No graded retry and fault verification: Most of the technical solutions in related technologies are single-attempt repairs, which are greatly affected by transient I2C interference, have a low success rate, and lack subsequent result verification, making it impossible to determine whether the CPLD fault has been truly recovered. Lack of a complete logging system: The technical solutions in related technologies do not record key aspects such as fault detection, repair execution, and result verification in a hierarchical manner, resulting in faults that are untraceable, root causes that cannot be located, and difficulty in supporting subsequent circuit optimization and firmware improvement.

[0025] The embodiments of this application provide a fault handling method, and the method is described in detail in conjunction with the fault handling execution flow.

[0026] Specifically, Figure 1 This is a flowchart of a fault handling method provided according to an embodiment of this application.

[0027] like Figure 1 As shown, the fault handling method includes the following steps: In step S101, the real-time status of the server is determined, and if the real-time status meets the detection conditions, the actual response status of the target processor is identified using the first communication address of the target processor.

[0028] In some embodiments, before detecting and repairing the CPLD for faults, this application may first detect the real-time status of the server to ensure that the application can repair faults in a safe environment.

[0029] Here, the real-time status of the server can be understood as a set of multi-dimensional operating status information obtained by the BMC from the server system (the whole machine) before executing the CPLD fault detection and repair process, used to determine whether the repair trigger conditions are met at the current moment.

[0030] Based on the real-time status of the server, embodiments of this application can detect whether the CPLD fault detection and repair process can be executed. If the real-time status of the server meets the detection conditions, embodiments of this application can first execute the CPLD fault detection process.

[0031] Here, the detection conditions can be understood as pre-defined rules / standards used to determine whether the server's real-time status at any given moment is suitable for CPLD fault detection and repair procedures. Only when all of the server's states meet certain requirements can the current moment be considered to meet the detection conditions, and only then can the BMC proceed with the subsequent CPLD fault detection and repair process.

[0032] It should be noted that the specific testing conditions can be set and adjusted by professionals in this field according to actual needs. The embodiments in this application are only illustrative and do not impose specific limitations.

[0033] When detecting CPLD faults, embodiments of this application can identify the actual response status of each target processor through the first communication address of the target processor.

[0034] Here, the target processor can be understood as the CPLD device mounted on each peripheral board in the server system, hereinafter referred to as CPLD.

[0035] Specifically, in this embodiment of the application, a certain instruction or signal can be sent from the BMC to the first communication address of the CPLD, and the real-time response status of the CPLD can be determined by the feedback from the CPLD.

[0036] The real-time response status here can be understood as the CPLD's reaction status (responding or not responding) to the instructions or signals sent by the BMC through the first communication address.

[0037] It should be noted that during the power-on process, the CPLD initialization is faster than the BMC. Therefore, by the time the BMC can execute the fault handling method in this embodiment, the CPLD has already been clearly loaded, and its success or fault state has been determined. At this point, the BMC can directly detect whether the CPLD loading was successful after startup.

[0038] Figure 2 This is a flowchart illustrating the power-on process of an embodiment of this application. Figure 2 As shown, in practical applications, after the server is connected to AC power, the BMC and CPLD will start and initialize simultaneously, but the CPLD will start and initialize earlier than the BMC.

[0039] When the server is connected to AC power, it enters standby mode. After the user powers on the server, the server will execute the power-on process. After the server completes the power-on process, the hard drive backplane CPLD will output power enable, which will be recognized by the BMC / BIOS (Basic Input Output System). After the BMC recognizes the CPLD power supply, it can execute the CPLD detection and repair process.

[0040] This application embodiment can determine the real-time status of the server in advance and only perform subsequent operations when the detection conditions are met. This can ensure that the fault detection and repair process of the CPLD is triggered only in safe scenarios, effectively preventing misoperation.

[0041] Optionally, in one embodiment of this application, the method further includes: obtaining a communication address in the target processor that meets preset communication conditions to determine a first communication address; and obtaining a flash memory address in the target processor that meets preset access conditions to determine a second communication address.

[0042] Understandably, the BMC, as the management core of the entire system, connects to the I2C multiplexer via the I2C bus to enable time-sharing access to the CPLDs of multiple peripheral boards.

[0043] Each CPLD has two independent I2C addresses: (1) Conventional communication address: used by BMC to read CPLD version, status, control signals, hard disk power, etc.; (2) UFM address (user flash address, also known as UFM upgrade address): dedicated to firmware upgrade and firmware reload command reception, unaffected by SDM failure, and always accessible.

[0044] Figure 3 This is a schematic diagram of a dual I2C address communication hardware architecture for a BMC and CPLD according to an embodiment of this application. Figure 3 As shown, the dual I2C address communication architecture between the BMC and CPLD can be described as follows: The BMC connects to the PCA9548I2C multiplexer via the I2C bus and communicates with the CPLDs of various boards such as the backplane, SW board, and SDB board through channel switching. Each CPLD is configured with a communication address (possibly 0x10 / 0x11 / 0x18) and a UFM address (0x40). The two addresses are independent of each other. The communication address can be used for routine control such as obtaining the version number, while the UFM address is used for firmware upgrades and receiving REFRESH_CMD and performing reload.

[0045] In some embodiments, in order to accurately identify whether the CPLD can communicate with the BMC, the embodiments of this application can obtain the communication address in the CPLD that meets the preset communication conditions as the first communication address, that is, the regular communication address of the CPLD. The accessibility of this communication address can effectively reflect whether the CPLD has completed the SDM self-loading process and is in normal working condition.

[0046] Here, the preset communication conditions can be understood as the BMC being able to communicate and interact with the CPLD through this communication address.

[0047] Similarly, in this embodiment of the application, the flash memory address in the CPLD that meets the preset access conditions can be obtained as the second communication address, i.e., the UFM address.

[0048] In this context, the preset access conditions can be understood as meaning that the address can always be accessed.

[0049] The UFM address is a user flash memory address pre-allocated by the CPLD's internal hardware. This address is fixed at the CPLD's UFM hardware level and is always available to respond to external access requests. Its accessibility does not depend on whether the CPLD has completed the SDM self-loading process. Even if the CPLD fails to boot and the regular communication address becomes unavailable, this address will still remain accessible.

[0050] This application embodiment can set the first communication address to an address that meets preset communication conditions, i.e., a conventional communication address. The accessibility of the CPLD's conventional communication address reflects whether the CPLD has completed SDM self-loading and is in normal working condition, providing a solid and reliable basis for CPLD fault diagnosis. Furthermore, this application can also set the second communication address to a flash memory address that meets preset access conditions, i.e., a UFM address. This effectively utilizes the characteristic that the accessibility of the CPLD's UFM address does not depend on whether the CPLD has completed SDM self-loading and can respond to external access requests at any time to complete subsequent instruction writing. Through the functional separation of the two addresses, this application embodiment can construct a dual-channel architecture where "the conventional communication address is used for fault detection, and the UFM address is used for instruction transmission," providing a communication foundation for the subsequent "detection-repair-verification" closed loop.

[0051] Optionally, in one embodiment of this application, before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: identifying the startup type of the server according to the target query interface; and determining that the real-time state meets the detection conditions if the startup type is AC power-on type.

[0052] In some embodiments, before the BMC identifies the real-time response status of the CPLD through the first communication address of the CPLD, the present application may, when detecting whether the real-time status of the server meets certain detection conditions, include, but is not limited to, three-layer judgment.

[0053] The first layer of judgment is the startup reason determination, which is mainly used to determine whether the server's startup type meets certain detection conditions.

[0054] For example, this application can be based on the startup type of the target query interface server during this startup.

[0055] Here, the target query interface can be understood as a query interface used to query the type of the server's current startup; in this embodiment, the standard startup reason query interface provided by the BMC's own firmware can be used as the target query interface, but is not limited to.

[0056] Therefore, during actual queries, BMC can directly obtain the server's current boot type through the standard boot reason query interface provided by its own firmware, including but not limited to: AC power-on boot, DC soft reboot boot, BMC software reboot, watchdog reboot, manual button reboot, firmware upgrade reboot, and other boot types.

[0057] Only when the server's startup reason is AC power-on startup, i.e., the server's startup type is AC power-on, can this application embodiment determine that the server's real-time status meets certain detection conditions at the first level, and then proceed to the next level, i.e., the second level of determination.

[0058] This is because the CPLD will only undergo the complete SDM self-loading process when powered on by AC. This is also why the I2C signal is in an intermediate state due to the inconsistency between the power timing of the peripheral cards and the motherboard, which leads to the premature termination of the SDM self-loading.

[0059] For example, if the server host is running normally, and the BMC performs a software restart in S0 power-on state for some reason, even if the restart reason is not "AC power-on," it should be shut down immediately. This is because the CPLD may be normally controlling devices such as hard drives, fans, and power supplies while the host is running. If a repair command is sent to the CPLD because of the BMC restart, it will severely disrupt the running business logic. Therefore, if the restart reason is not "AC power-on," the detection should be abandoned, and no repair should be performed.

[0060] For example, if the host performs a DC soft reboot, this reboot does not disconnect the AC power supply to the entire machine. The motherboard's standby power supply (P3V3_STBY) remains powered. The cause of the reboot can be determined as a DC reboot, and it is necessary to exit directly. Because the entire machine is not powered off at this time, the motherboard, CPU, memory, and other loads have not been completely powered down and then powered on again. In this scenario, if the CPLD is already working normally, there will be no problem of "SDM self-loading failure due to I2C level abnormality," and there is no need to perform repair. Therefore, it is possible to exit directly.

[0061] If the server's current startup type is any type other than AC power-on, it can be determined that the server's real-time status does not meet the detection conditions, and the system can exit directly.

[0062] This application embodiment can determine that the detection conditions are met only when the server's startup type is identified as AC power-on type. This ensures that the application can perform fault detection and repair in scenarios where the CPLD needs to go through a complete SDM self-loading process, effectively avoiding invalid work in non-target scenarios, accurately locking the actual occurrence scenario of CPLD startup failure, and eliminating the risk of false triggering from the source.

[0063] Optionally, in one embodiment of this application, before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: detecting the current power state of the server according to the target management interface; and determining that the real-time state meets the detection conditions if the power state is off.

[0064] Based on the descriptions of other embodiments, it is understood that before the present application uses BMC to identify the real-time response status of CPLD through the first communication address of CPLD, when detecting whether the real-time status of the server meets certain detection conditions, there are, but are not limited to, three layers of judgment.

[0065] The first layer of judgment is the startup reason determination, which is mainly used to determine whether the server's startup type meets certain detection conditions.

[0066] The second layer is for determining the overall power status, which is mainly used to determine whether the power status of the server at the current moment meets certain detection conditions.

[0067] Here, the target management interface can be understood as the management interface for obtaining the power status information of the server at the current moment; in this embodiment of the application, the standard power status management interface may be used as the target management interface, but is not limited to.

[0068] Therefore, during actual queries, BMC can obtain the current power mode of the entire server through the standard power status management interface, including but not limited to: S5 (power off / standby); S0 (power on); S3 / S4 (sleep) and other power states.

[0069] Only when the server's current power state is off, i.e., the server's current power mode is S5 off, can this application embodiment determine that the server's real-time state meets certain detection conditions at the second layer, and then proceed to the next layer, i.e., the third layer, for determination.

[0070] If the server's current power state is any power mode other than the S5 shutdown state (such as S0 (host normal working state, usually referring to the state of booting into OS (Operating System)); S3 / S4 (hibernation), etc.), it can be directly determined that the server's real-time state does not meet the detection conditions, and the process can be directly exited, disallowing the execution of CPLD detection and repair processes.

[0071] This application embodiment can determine that the detection conditions are met only when the power is off. At this time, the main loads such as the CPU, memory, and hard disk are not powered on, so the CPLD will not participate in the operation of any business logic. Performing CPLD fault detection and repair operations at this time will not have any impact on the business, nor will it damage the running logic. This ensures that the fault repair operation in this application can be performed only when the whole machine is in a safe window period, effectively avoiding the occurrence of secondary faults such as bus deadlock, hard disk recognition failure, or even abnormal system reset.

[0072] Optionally, in one embodiment of this application, before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: obtaining the main load power supply state and bus occupancy state of the server; and determining that the real-time state meets the detection conditions when the main load power supply state is not powered on, the bus occupancy state is not occupied, and the external device connected to the server is not running, and the target processor is not accessed or upgraded.

[0073] In some embodiments, when determining whether the real-time status of the server meets certain detection conditions through a three-layer state judgment, the first layer is the startup reason judgment, which is mainly used to determine whether the startup type of the server meets certain detection conditions; the second layer is the overall power status judgment, which is mainly used to determine whether the power status of the server at the current moment meets certain detection conditions; and the third layer is the security mutual exclusion condition judgment.

[0074] Specifically, in the third-level determination, this application mainly detects whether the main load power supply status, bus occupancy status, peripheral status, and the status of the implemented CPLD object of the server meet certain detection conditions.

[0075] For example, only when the server's main load power supply status is not powered on (e.g., CPU, memory, hard drive, etc. are not powered on), the server's bus occupancy status is not occupied (i.e., the I2C bus is idle and no other tasks are occupying it), the external devices connected to the server are not running (e.g., no fans, power supplies, or other peripherals are performing power-on or calibration processes and the BIOS is not running), the CPLD is not accessed and not being upgraded (e.g., no CPLD is undergoing firmware upgrade processes and no out-of-band tasks are accessing any CPLD), and the server as a whole has not sent a power-on command, can this application embodiment determine that the server's real-time status meets certain detection conditions at the third layer, and only then can the subsequent CPLD detection and repair process be performed.

[0076] If any requirement in the third-level judgment is not met, the server's real-time state is considered to fail the detection conditions, and the CPLD detection and repair processes are not allowed. In other words, the repair process is only allowed if all security mutual exclusion conditions are not triggered; if any security mutual exclusion condition is not met, the repair process is prohibited.

[0077] This application embodiment can determine that the detection conditions are met when the main load is not powered on, the bus is not occupied, the external device is not running, and the CPLD is not accessed and not being upgraded. Combined with the determination of the first two layers, it can form a triple protection mechanism of "startup reason + power status + mutual exclusion safety", which can effectively eliminate the possibility of accidental triggering of repair during operation.

[0078] Optionally, in one embodiment of this application, before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: determining that the real-time state meets the detection conditions when the server's startup type is AC power-on type, the server's current power state is power off, the server's main load power supply state is not powered on, the bus occupancy state is not occupied, the external device connected to the server is not running, and the target processor is in an unaccessed state and an unupgraded state.

[0079] Based on the descriptions of other embodiments, it can be understood that the real-time status determination of the server in this application has three layers: startup reason determination, overall power status determination, and security mutual exclusion condition determination. The second layer can only be entered when the first layer is satisfied, and the third layer can only be entered when the second layer is satisfied.

[0080] In actual implementation, this application can only enter the CPLD detection and repair process when all three conditions are met.

[0081] In other words, the real-time status of the server can only be determined to meet the detection conditions when the server's startup type is AC power-on, the server's current power status is power-off, the server's main load power supply status is not powered on, the bus occupancy status is not occupied, the external devices connected to the server are not running, and no CPLD is accessed or upgraded.

[0082] The embodiments of this application can complete the three-layer determination through the BMC standard software interface without using any specific registers, fixed GPIOs, or dedicated flags. While effectively ensuring that the fault repair in this application is performed in a safe environment, it greatly expands the applicable scenarios of this application and is suitable for BMCs and CPLDs from various manufacturers and with various architectures.

[0083] Optionally, in one embodiment of this application, identifying the actual response state of the target processor using the first communication address of the target processor includes: obtaining the real-time state of the board where the target processor is located; and identifying the actual response state of the target processor using the first communication address of the target processor when the real-time state of the board where the target processor is located is in the present state.

[0084] Those skilled in the art will understand that backplanes, fan boards, power boards, and switch boards in servers support hot-swapping. Sometimes, during BMC startup, some boards may not be physically inserted or have completed power-on initialization. Performing I2C communication operations on boards that are not in place may cause bus timeouts, thus affecting overall system startup efficiency. Furthermore, for hot-swappable boards, users may remove or replace them during system operation for fault detection, and the BMC may misinterpret the missing board as a CPLD failure.

[0085] Based on this, in some embodiments, in order to ensure that the implemented CPLD object is valid, the embodiments of this application can also detect the real-time status of the board where the target processor is located.

[0086] That is, after the BMC starts, it needs to check whether the board where the CPLD is located is in place. Only when the real-time status of the board where the CPLD is located is in place, that is, after the BMC starts and checks that the board where the CPLD is located is in place, can it detect whether the CPLD has been successfully loaded. In other words, the BMC uses the first communication address of the target processor to identify the actual response status of the target processor.

[0087] This application embodiment can obtain the real-time status of the board where the CPLD is located before performing CPLD fault detection. Subsequent detection is only performed when the board is in place. Through this prior detection based on in-place determination, this application can effectively distinguish between two different scenarios: "actual CPLD fault" and "board not in place", avoiding misjudgment and improving the accuracy of fault detection.

[0088] Optionally, in one embodiment of this application, identifying the actual response state of the target processor using the first communication address of the target processor includes: sending a first version number read instruction to the first communication address of the target processor through the target communication bus; after sending, obtaining the version number read state corresponding to the first version number read instruction; and identifying the actual response state of the target processor based on the version number read state.

[0089] In some embodiments, when using the BMC to identify the actual response status of the target processor through the first communication address of the target processor, this application may, but is not limited to, detect whether the version number of the target processor has been successfully read.

[0090] For example, this application can enable the BMC to send a first version number read instruction to the first communication address of the CPLD, which is mainly used to read the version number of the CPLD.

[0091] Since the version number read instruction reads the firmware version number or device identification information in CPLD register 0, it does not change any internal state of the CPLD, making it safe and reliable.

[0092] Furthermore, the transmission process in this application embodiment can be implemented through, but is not limited to, the target communication bus, which can be understood here as the communication bus between the BMC and the CPLD, and in this application embodiment refers to the I2C bus.

[0093] After the BMC sends the first version number read instruction to the first communication address of the CPLD, it can, but is not limited to, obtain the version number read status corresponding to the first version number read instruction to determine whether the version number read status corresponding to the first version number read instruction is a read success status (successfully read the CPLD version number) or a read failure status (unsuccessfully read the CPLD version number).

[0094] Based on the version number reading status corresponding to the first version number reading instruction, the real-time response status of the CPLD can be determined in this embodiment: a successful reading status means that the real-time response status of the CPLD is a successful response status, and a failed reading status means that the real-time response status of the CPLD is a non-response status.

[0095] In this embodiment, the first version number read command can be sent to the first communication address of the CPLD via the target communication bus (I2C bus) to obtain the corresponding version number read status and identify whether the CPLD responds. This is because the CPLD can only correctly respond to the version number read command after completing SDM self-loading and opening the regular communication address. This detection method does not rely on any dedicated hardware and can be completed solely through standard I2C bus operation, enabling pure software detection.

[0096] Optionally, in one embodiment of this application, after identifying the actual response status of the target processor using the first communication address of the target processor, the method includes: when the version number reading status corresponding to the first version number reading instruction is a reading failure status, generating a second version number reading instruction and sending it to the first communication address of the target processor, until the number of times the version number reading instruction is sent reaches a target number; when the version number reading status corresponding to the target number is a reading failure status, determining that the actual response status is a non-response status.

[0097] As one possible approach, in order to ensure that the real-time response status of the CPLD can be accurately identified, this application may determine the real-time response status of the CPLD by sending version number read commands multiple times.

[0098] Given that the I2C bus may be affected by transient factors such as electromagnetic interference or unstable voltage levels during the initial startup of the server, resulting in a single communication failure, this application can, when the version number read status corresponding to the first version number read instruction is a read failure status, that is, when the BMC fails to read the CPLD version number, cause the BMC to generate a version number read instruction again, referred to here as the second version number read instruction, and then send the second version number read instruction to the first communication address of the target processor.

[0099] If the version number read status corresponding to the second version number read instruction is still in the read failure state, the embodiments of this application can make the BMC generate the version number read instruction again until the number of version number read instructions sent reaches the target number.

[0100] Here, the target number can be understood as a pre-set limit on the number of times the version number read command can be sent, such as 3 times. The specific target number can be set and adjusted by those skilled in the art according to actual needs. This embodiment is only illustrative and does not impose any specific limitations.

[0101] When the version number read command corresponding to the sent number limit value is in the read failure state, the embodiment of this application can determine that the real-time response state of the CPLD is in the non-response state.

[0102] Specifically, if, before reaching the limit for the number of transmissions, any version number read instruction corresponds to a version number read success status, this embodiment of the application can determine that the real-time response status of the CPLD is a successful response status.

[0103] This application embodiment can continue to generate and resend version number reading instructions when the version number reading status corresponding to the first version number reading instruction is a reading failure, until the number of transmissions reaches the target number. Only when all reading statuses corresponding to the target number are reading failures is the actual response status of the CPLD determined to be no response. This hierarchical retry mechanism effectively distinguishes between communication failures caused by transient interference in I2C communication and communication failures caused by actual CPLD malfunctions, significantly reducing the false judgment rate, improving the accuracy of fault detection, and effectively avoiding unnecessary repair operations triggered by false judgments.

[0104] Optionally, in one embodiment of this application, determining that the target processor is in a fault state when the actual response state is a non-response state includes: determining that the first communication address is in a fault state when the actual response state is a non-response state; and determining that the target processor is in a fault state when the first communication address is in a fault state.

[0105] In some embodiments, when the reason why the actual response status of the CPLD is determined to be a non-response status is that the version number read status corresponding to at least one (one or more) version number read instructions is a read failure status, it proves that the BMC failed to successfully read the version number from the first communication address of the CPLD, which means that the first communication address of the CPLD is lost and cannot communicate successfully. That is, the first communication address of the CPLD has a communication failure. Therefore, this application can determine that the first communication address of the CPLD is in a fault state.

[0106] Therefore, when the first communication address of the CPLD is determined to be in a fault state, the embodiments of this application can determine that the CPLD is in a fault state.

[0107] This application embodiment can determine that the first communication address is in a fault state when the actual response status is no response, and then determine that the CPLD is in a fault state. It clarifies that the essence of the CPLD fault is that the first communication address is lost and inaccessible, because the first communication address will only be opened after the CPLD completes SDM self-loading, providing a clear triggering condition for the subsequent generation of recovery instructions.

[0108] Optionally, in one embodiment of this application, after determining that the actual response status is a non-response status when all version number read states corresponding to the target number are in the read failure state, the method further includes: generating a first fault log based on the timestamp, the first communication address, the response code corresponding to the read failure status, and the fault recovery process triggering marker information corresponding to all version number read states corresponding to the target number; and feeding the first fault log back to the server to store the fault status tracing data on the server.

[0109] In some embodiments, in order to trace the cause of CPLD failure, this application can record failures at different stages during the detection and repair process.

[0110] For example, Figure 4 This is a schematic diagram illustrating the hierarchical nodes and corresponding operations of fault log recording according to one embodiment of this application. Figure 4 As shown, when the version number read status corresponding to the version number read instruction for the target number is a read failure status, this application may, but is not limited to, generate a first fault log (level 1 fault log).

[0111] The content of the first fault log includes, but is not limited to, the timestamp corresponding to when the real-time response status of the CPLD is determined to be non-responsive, that is, the timestamp corresponding to the moment when the version number read status of the version number read instruction for the target number (e.g., three times) is all in the read failure state.

[0112] Furthermore, the version number read status corresponding to the version number read instruction is a read failure status, which proves that the first communication address of the CPLD has a communication failure. Therefore, the first communication address of the CPLD can still be recorded in the first fault log.

[0113] Furthermore, embodiments of this application may also record the response code corresponding to each version number read instruction in a read failure state, as well as information such as the fault recovery process trigger flag generated when a communication failure is found in the first communication address of the CPLD and fault repair needs to be performed.

[0114] For example, during the overall machine startup process, if it is a power-off startup, the BMC is a component that starts up as soon as it is powered on, and the CPLD also starts up as soon as it is powered on. However, the CPLD often starts up much faster, completing the startup in milliseconds. This ensures that if the CPLD's communication address is missing or unable to communicate when the BMC starts up and the check begins, then the CPLD must have malfunctioned during this power-on startup.

[0115] Therefore, in this embodiment, the BMC can send a version number read instruction (reading the version number data in register 0) to the first communication address of the CPLD (such as 0x10 / 0x11 / 0x18) via the I2C bus, and set a 3-level retry mechanism, with each retry requiring an interval of 1 second between the last version number read instruction and the transmission time. If any version number is successfully read (the version number read status corresponding to the version number read command is read successfully), it can be determined that the CPLD is powered on normally and the fault detection process can be exited. If multiple consecutive read failures (e.g., 3 times) occur (i.e., the version number read status corresponding to the version number read instruction for the target number of times is in the read failure status), it can be determined that the first communication address of the CPLD is lost. At this time, the embodiment of this application can record the fault log LOG1, i.e., the first-level fault log (indicating that the CPLD failed to power on and triggers the repair process), in order to proceed to the next step.

[0116] The number of retries and the time interval can be modified by those skilled in the art according to actual needs. The embodiments in this application are only illustrative and do not impose specific limitations.

[0117] After generating the first fault log, this embodiment of the application can promptly feed the first fault log back to the server so that the fault can be traced later.

[0118] This application embodiment can generate a first fault log when all version number readings fail during the detection process, and feed it back to the server for storage. This provides a data foundation for subsequent CPLD fault tracing, and can be used to determine whether the CPLD fault is a hardware timing problem, I2C bus interference, or CPLD itself is abnormal, thereby providing an analyzable data foundation for subsequent circuit design optimization and firmware improvement.

[0119] In step S102, if the actual response status is non-response, the target processor is determined to be in a fault state, so as to generate a recovery instruction for controlling the target processor to self-load.

[0120] In some embodiments, if the BMC sends a version number read instruction to the first communication address of the CPLD at least once (one or more times) and the version number read status is always a read failure status, this application can determine that the first communication address of the CPLD has a communication failure, that is, the CPLD is in a fault state.

[0121] After determining that the CPLD is in a faulty state, in order to ensure normal communication between the BMC and the CPLD, this application embodiment can cause the BMC to generate a recovery instruction for controlling the CPLD to perform a self-loading process, so that the CPLD can perform a self-loading process and thereby restore communication with the BMC.

[0122] The embodiments of this application can detect the actual response status of the CPLD through the first communication address to determine the fault. It does not rely on any special registers, GPIO or hardware flags, and can be universal across all platforms without being affected by hardware. When it is determined that the CPLD is in a faulty state, a recovery instruction for controlling the CPLD self-loading can be generated instead of a firmware upgrade instruction, so that the CPLD only needs to re-execute the SDM loading process, which can efficiently repair without modifying the firmware and at extremely low cost.

[0123] Optionally, in one embodiment of this application, when the actual response state is a non-response state, the target processor is determined to be in a fault state, so as to generate a recovery instruction for controlling the self-loading of the target processor, including: determining the self-loading recovery target of the target processor based on the fault state; and generating a recovery instruction according to the self-loading recovery target.

[0124] Based on the descriptions of other embodiments, it is understood that when it is determined that the real-time response status of the CPLD is in an unresponsive state because the first communication address of the CPLD has failed, causing the BMC to be unable to communicate successfully with the CPLD, this application can generate a recovery instruction for controlling the self-loading of the target processor.

[0125] Since there are many reasons for CPLD failures, if the BMC detects that the CPLD's first communication address cannot communicate after power-on, it can be determined that the CPLD's self-loading process has failed, such as being interrupted and not completing the loading process.

[0126] Therefore, when repairing a fault based on the CPLD's real-time response status being non-responsive, this application can first determine the repair target as causing the CPLD to re-execute the self-loading process. Thus, when generating recovery instructions, this application can first determine that the target of the recovery instructions is the CPLD's self-loading recovery target, and then generate the recovery instructions based on this.

[0127] After the recovery instruction is written to the CPLD via the UFM address, the CPLD's internal state machine will exit the instruction waiting state and restart the SDM self-loading, thereby fixing the fault caused by the misjudgment of the I2C level that led to the failure of SDM self-loading.

[0128] This application embodiment can generate a self-loading recovery target for the CPLD based on the fault state, and then generate a recovery instruction based on the target. The logic for generating the recovery instruction is to first determine the self-loading recovery target, and then generate the corresponding recovery instruction. The abstract fault state of the CPLD is transformed into a clear self-loading recovery target, and then into a specific recovery instruction, so as to force the CPLD to re-execute the SDM firmware loading process, rather than upgrade the firmware or perform other operations.

[0129] Step S103: Send a recovery command to the target processor using the second communication address of the target processor, and determine that the target processor has exited the fault state if the target processor responds to the recovery command, thus completing the fault handling.

[0130] In other embodiments, after the BMC determines that the first communication address of the CPLD has failed and generates a recovery instruction, this application allows the BMC to use the second communication address of the CPLD to write the recovery instruction into the CPLD.

[0131] Specifically, after the BMC generates the recovery instruction, it can write the recovery instruction to the second communication address of the CPLD, because the second communication address is a flash memory address that can always be accessed (regardless of whether the CPLD self-loads successfully or not).

[0132] Therefore, after the recovery instruction is written to the second communication address of the CPLD, if the CPLD responds effectively to the recovery instruction and reloads, the BMC can communicate with the CPLD through the first communication address, which means that the communication fault of the CPLD has disappeared. At this time, it can be determined that the CPLD has exited the fault state and the fault handling is completed.

[0133] This application embodiment can use the second communication address of the CPLD to send recovery commands, effectively utilizing the characteristic that the CPLD's UFM address is still accessible even when the CPLD is faulty, ensuring that the repair commands can be reliably delivered. After successful recovery, it can be promptly determined that the CPLD has exited the fault state, forming a complete closed loop. The entire fault detection and repair process is implemented purely in software, without involving hardware modifications, which can effectively guarantee the universality, security, and reliability of this application.

[0134] Optionally, in one embodiment of this application, after sending the recovery instruction to the target processor using the second communication address of the target processor, the method further includes: obtaining the write status of the recovery instruction, so as to generate a third version number read instruction if the write status is a write success status; sending the third version number read instruction to the target processor through the target communication bus based on the first communication address; obtaining the version number read status corresponding to the third version number read instruction, so as to identify whether the target processor responds to the recovery instruction based on the version number read status corresponding to the third version number read instruction.

[0135] In some embodiments, after the BMC generates a recovery command and sends it to the CPLD, in order to detect whether the CPLD has responded to the recovery command, this application can again read the version number through the first communication address of the CPLD.

[0136] For example, this application can first detect the write status of the recovery instruction. When it is determined that the write status of the recovery instruction is a successful write status, that is, the recovery instruction is successfully written to the second communication address of the CPLD, a version number read instruction (referred to here as the third version number read instruction) is generated again and sent to the first communication address of the CPLD through the target communication bus, i.e., the I2C bus.

[0137] Similarly, this application can still obtain the version number read status corresponding to the third version number read instruction: whether it is a read success status (successfully read the CPLD version number) or a read failure status (unsuccessfully read the CPLD version number), and then determine whether the target processor responds to the recovery instruction based on the version number read status corresponding to the third version number read instruction.

[0138] It should be noted that the first version number reading instruction, the second version number reading instruction, and the third version number reading instruction in the embodiments of this application are all essentially version number reading instructions. The different names are only used in the embodiments of this application to facilitate the distinction of the order of transmission, the number of transmissions, and the generation / usage period. They are essentially the same.

[0139] This application embodiment can generate a third version number read instruction and send it to the first communication address of the CPLD after the recovery instruction is successfully written. The CPLD can be judged by the corresponding version number read status to determine whether the CPLD has responded to the recovery instruction. Combined with the previous detection and repair process, a complete closed loop of "detection-repair-verification" is formed, which effectively solves the defect of sending instructions without verifying the results in related technologies and ensures the verifiability of the repair operation.

[0140] Optionally, in one embodiment of this application, identifying whether the target processor responds to the recovery command based on the version number reading status corresponding to the third version number reading command includes: determining that the target processor responds to the recovery command when the version number reading status corresponding to the third version number reading command is a successful reading status; and determining that the target processor does not respond to the recovery command when the version number reading status corresponding to the third version number reading command is a failed reading status.

[0141] In some embodiments, based on the version number reading status corresponding to the third version number reading instruction, this application embodiment can determine whether the CPLD responds to the recovery instruction: When the version number read status corresponding to the third version number read instruction is in the read success state, it means that the CPLD's first communication address has resumed communication, proving that the CPLD has effectively responded to the recovery instruction, reloaded and successfully completed the process.

[0142] When the version number read status corresponding to the third version number read instruction is in the read failure state, it means that the first communication address of the CPLD has failed to resume communication, and the CPLD has failed to respond effectively to the recovery instruction: it has not performed a reload or has executed an instruction but failed to successfully perform a reload.

[0143] Similarly, since read commands of the same version number can be sent multiple times, this application can also perform multiple recovery command writes. That is, when the write status of the first recovery command to the second communication address of the CPLD is in a write failure state, this application can generate a second recovery command and write it to the second communication address of the CPLD in the same way, until the number of recovery command writes reaches the second target number.

[0144] Here, the second target number can be understood as a pre-set limit value for the number of times the recovery command can be written, such as 3 times. The specific second target number can be set and adjusted by those skilled in the art according to actual needs. This embodiment is only illustrative and does not impose any specific limitations.

[0145] If the write status corresponding to the recovery command in the second target number is a write failure status, it can be finally determined that the write status of the recovery command is a write failure status, that is, the target processor did not respond to the recovery command.

[0146] In this embodiment, when the read status corresponding to the third version number read instruction is read successfully, it can be determined that the CPLD has responded to the recovery instruction: because the CPLD has completed reloading and reopened the normal communication address, it can normally respond to the BMC's access request, and then exit the fault state, thus ensuring the accuracy and reliability of the CPLD response determination.

[0147] Optionally, in one embodiment of this application, after sending a recovery instruction to the target processor using the second communication address of the target processor, the method further includes: generating a second fault log based on the recovery instruction, the timestamp and bus status corresponding to the write failure state, the second communication address and the fault recovery process termination marker information when the write state is a write failure state; and sending the second fault log to the server to store the fault state traceability data on the server.

[0148] In some embodiments, this application can also detect whether there is a fault during the sending of the recovery command. If there is a fault, the fault can be recorded for later fault tracing.

[0149] For example, such as Figure 4 As shown, this application can generate a second fault log (and related information) when a recovery instruction fails to be successfully written to the second communication address of the CPLD, based on the content of the recovery instruction, the timestamp corresponding to the failure to write to the second communication address of the CPLD, the second communication address, the bus state at each write failure, and the fault recovery process termination flag information generated when the recovery instruction cannot be successfully written. Figure 4 (The secondary fault log serves the same purpose).

[0150] Similarly, after generating the second fault log, this embodiment of the application can promptly feed the second fault log back to the server for later fault tracing.

[0151] For example, this application can cause the BMC to write a repair instruction to the UFM address of the CPLD, that is, cause the BMC to write a REFRESH_CMD repair instruction to the UFM address of the CPLD for which the first communication address is lost. The instruction data is 0x7900. This instruction is a CPLD firmware reload trigger instruction, which can force the CPLD to re-execute the SDM firmware loading process.

[0152] Meanwhile, this application embodiment can set up a multi-stage retry mechanism (the same number as the second target number, such as 3 times, 5 times, etc.), with a 1-second interval between each retry: If multiple consecutive recovery commands fail to write, such as 3 or 5 consecutive recovery command write failures, this embodiment of the application can record a fault log LOG2 (i.e., a secondary fault log, used to identify repair command write failures), end the repair operation of this device, form a secondary fault log, and feed it back to the server.

[0153] If any recovery command is successfully written, this embodiment of the application can proceed to the subsequent CPLD fault recovery and verification process.

[0154] Through this dual retry mechanism, the embodiments of this application can effectively avoid repair failures caused by transient I2C interference.

[0155] In this embodiment of the application, when the write status of the recovery instruction is write failed, a second fault log is generated and sent to the server for storage based on the recovery instruction, the timestamp and bus status corresponding to the write failure status, the second communication address and the fault recovery process termination flag information. This further divides the CPLD fault into two scenarios: the regular communication address is lost but the UFM is available and the UFM is also unavailable, providing a more refined basis for fault location.

[0156] Optionally, in one embodiment of this application, after determining that the target processor does not respond to the recovery instruction, the method further includes: in the case that the target processor does not respond to the recovery instruction, generating a third fault log based on the timestamp corresponding to the time when the target processor does not respond to the recovery instruction, the successful sending flag information of the third version number read instruction, the return value corresponding to the third version number read instruction, the first communication address, and the hardware abnormality warning flag information; and sending the third fault log to the server through the target communication bus to store the fault status traceability data on the server.

[0157] As one possible implementation, when the recovery instruction is successfully written to the second communication address of the CPLD, but the version number read status corresponding to the third version number read instruction is in a read failure state, this application can generate a third fault log (and...). Figure 4 The three levels of fault logs serve the same purpose.

[0158] For example, if the recovery instruction is successfully written to the second communication address of the CPLD, but the version number read status corresponding to the third version number read instruction is in the read failure status, this application can determine that the first communication address of the CPLD has failed to restore communication, that is, the CPLD has failed to respond effectively to the recovery instruction: it has not performed a reload or has performed an execution instruction but failed to successfully perform a reload.

[0159] At this time, this application can generate a record based on the timestamp corresponding to when the CPLD does not respond to the recovery instruction, the marker information of the successful transmission of the recovery instruction (successful transmission marker information), the target address for verification reading (the regular communication address of the CPLD that the BMC tries to read again after writing the recovery instruction (such as the version register), which is the first communication address in this application embodiment) and the return value (the result actually read from the regular communication address).

[0160] Furthermore, if the recovery instruction is successfully written to the second communication address of the CPLD, but the CPLD still fails to complete the self-loading process and communication at the first communication address is still not restored, it can be considered that there is a problem with the hardware in the server. Therefore, this embodiment of the application can also generate hardware abnormality warning flag information.

[0161] Based on this information, embodiments of this application can generate a third fault log and promptly feed the third fault log back to the server for later fault tracing.

[0162] This application embodiment can generate a third fault log and send it to the server for storage when the CPLD does not respond to the recovery command, based on the corresponding timestamp, the third version number, the successful sending flag information of the command, the return value, the first communication address, and the hardware anomaly warning flag information. This third fault log, together with the first and second fault logs, constitutes a complete three-level log system of "fault detection → repair execution → recovery verification," corresponding to fault detection failure (LOG1), command sending failure (LOG2), and recovery verification failure (LOG3, ...). Figure 4 The three levels of fault logs serve the same purpose, enabling maintenance personnel to accurately pinpoint the stage at which a fault occurred, significantly improving fault traceability.

[0163] Optionally, in one embodiment of this application, after determining that the target processor has exited the fault state, the method further includes: generating a fault recovery report based on the repair completion timestamp of the fault recovery process of the target processor, the read value of the first communication address, the total duration of fault processing, and the fault processing success marker information; and feeding the fault recovery report back to the server.

[0164] In some embodiments, in the CPLD fault detection and recovery process, in addition to generating fault logs, this application can also generate a fault recovery report after the CPLD fault is successfully repaired (and...). Figure 4 (The successful verification log serves the same purpose).

[0165] For example, this application may, but is not limited to, consist of: a repair completion timestamp (corresponding to the moment when the fault repair is completed) determining the CPLD's fault repair process; the read value of the first communication address obtained when the read status corresponding to the third version number read instruction is in the read success state (i.e., the information read by the BMC through the CPLD's first communication address); the total time consumed by the entire fault handling (repair process), which may, but is not limited to, start timing from the moment when the CPLD's first communication address is determined to be in a fault state and a trigger mark for the repair process is generated, and end timing from the moment when the BMC reads the feedback information through the CPLD's first communication address; and fault handling success (repair success) marker information, etc., forming a fault recovery report, which is also fed back to the server for content tracing in subsequent processes.

[0166] For example, after the repair command is successfully written, the BMC can delay for 1 second (to allow time for CPLD firmware reloading) and then send a version number read command to the CPLD's communication address again via the I2C bus: If the read is successful, that is, the read status of the version number read instruction is a read success status, this application embodiment can determine that the CPLD communication address has been restored to normal, record the verification success log, and the repair process of the board and device where the CPLD is located is completed and exited; If the read fails, that is, the read status of the version number read command is read failed, this application embodiment can record fault log LOG3 (indicating CPLD repair failure) and end the repair operation of the board and device where the CPLD is located.

[0167] It should be noted that a server may contain multiple boards, which means there are multiple CPLDs, and each CPLD may fail. Therefore, the target processor in this embodiment refers only to the CPLD object currently performing the fault detection (and repair) process in the BMC, and does not mean that there is only one CPLD object.

[0168] That is, after completing the fault detection (and repair) of a CPLD, this application embodiment can traverse all boards in the server to check for unprocessed (CPLDs that have not undergone CPLD fault detection (and recovery) processes) CPLDs, so as to detect whether there are unprocessed programmable logic devices in the server.

[0169] If there are unprocessed programmable logic devices on the server, this embodiment will then perform the same process on the unprocessed (fault detection (and repair) processes) CPLDs until there are no unprocessed CPLDs.

[0170] For example, BMC can sequentially detect and repair faults in all CPLD-configured boards in the whole machine, such as the backplane, SW board (Switch Board), SDB board (Service Data Board / System Development Board), PSUPDB (Power Supply Unit - Power Distribution Board), and FANBD (Fan Board), in order to achieve batch automated processing of multi-board CPLDs.

[0171] In practical applications, the CPLD detection and repair process can be embedded into the initialization process of BMC startup to form a script, so as to complete all detection, repair and verification actions before the whole machine is powered on.

[0172] In actual execution, the BMC can use the PCA9548I2C multiplexer (address 0x74) to switch communication with different CPLDs. Before each read / write command, the channel switching of the multiplexer is completed and the successful switching is verified before the subsequent I2C interaction operation is executed. This can effectively ensure the accuracy of the communication link and realize time-sharing access to the CPLDs of multiple peripheral boards.

[0173] This application embodiment can perform hierarchical nodes and corresponding operations for fault log recording, dividing the logs into three levels of fault logs (LOG1 / LOG2 / LOG3) and verification success logs, corresponding to four key nodes: "multiple failures to read the communication address", "multiple failures to write the repair command", "failure to read the recovery verification", and "successful read of the recovery verification". Each log node is marked with the triggering condition, log number and core record content, providing reliable traceability data for fault tracing in subsequent processes.

[0174] The following is a detailed explanation of the fault handling method in the embodiments of this application, using a specific example.

[0175] Figure 5 This is an overall flowchart illustrating the fault detection and fault repair process performed by the BMC in one embodiment of this application. Figure 5 As shown, BMC executes the pre-imported scripts directly after startup; (1) When the detection conditions are met through the three-layer judgment, the version number read instruction (up to 3 times) is sent to the normal communication address of the target CPLD to determine whether the normal communication address exists (whether normal communication is possible): The BMC sends the version number read instruction (reads the version number data of register 0) to the communication address (0x10 / 0x11 / 0x18) of the CPLD through the I2C bus, and sets a 3-level retry mechanism with a 1-second interval between each retry: If any read is successful, the CPLD is determined to be powered on normally, and this device exits the fault detection process. If three consecutive reads fail, the CPLD communication address is determined to be lost. The fault log LOG1 (i.e., the first-level fault log, used to indicate CPLD power-on failure and trigger the repair process) is recorded, and the process proceeds to step (2). (2) Write firmware reload instructions to the UFM upgrade address (send up to 3 times) to force CPLD to re-execute SDM self-load: For CPLDs that have lost their communication address, BMC writes a REFRESH_CMD instruction (repair instruction) to their UFM address. The instruction data is 0x7900. This instruction is a CPLD firmware reload trigger instruction, which can force the CPLD to re-execute the SDM firmware loading process. At the same time, a three-level retry mechanism is set for this step, with a 1-second interval between each retry. If three consecutive write attempts fail, record fault log LOG2 (indicating repair command write failure) and terminate the repair operation for this device. If any write operation is successful, proceed to the recovery verification process in step (3).

[0176] (3) After the repair command is successfully written, the BMC delays for 1 second (to reserve the execution time for CPLD firmware reloading) and sends a version number read command to the CPLD communication address again via the I2C bus. If the read is successful, the CPLD communication address is determined to be back to normal, the verification success log is recorded, the repair process of this device is completed and exits; if the read fails, the fault log LOG3 is recorded (indicating CPLD repair failure), and the repair operation of this device ends.

[0177] (4) For all CPLD-configured boards such as backplane, SW board, SDB board, PSUPDB, FANBD, etc. in the whole machine, BMC performs fault detection and repair in sequence according to the process of steps (1)-(3) to realize batch automated processing of CPLDs of multiple boards.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0179] Embodiments of this application also provide a fault handling device 10, including: an identification module 100, a generation module 200, and a processing module 300.

[0180] The identification module 100 is used to determine the real-time status of the server, and when the real-time status meets the detection conditions, to identify the actual response status of the target processor using the first communication address of the target processor. The first generation module 200 is used to determine that the target processor is in a fault state when the actual response state is a non-response state, so as to generate a recovery instruction for controlling the target processor to self-load. The processing module 300 is used to send a recovery command to the target processor using the second communication address of the target processor, and determine that the target processor has exited the fault state and completes the fault handling when the target processor responds to the recovery command.

[0181] Optionally, in one embodiment of this application, it further includes: a first acquisition module, configured to acquire a communication address in the target processor that meets preset communication conditions, so as to determine a first communication address; and a second acquisition module, configured to acquire a flash memory address in the target processor that meets preset access conditions, so as to determine a second communication address.

[0182] Optionally, in one embodiment of this application, it further includes: a first query module, configured to identify the server's startup type according to the target query interface before identifying the actual response status of the target processor using the first communication address of the target processor; and a first determination module, configured to determine that the real-time status meets the detection conditions when the startup type is AC power-on type.

[0183] Optionally, in one embodiment of this application, it further includes: a detection module, configured to detect the current power state of the server according to the target management interface before identifying the actual response state of the target processor using the first communication address of the target processor; and a second determination module, configured to determine that the real-time state meets the detection conditions when the power state is off.

[0184] Optionally, in one embodiment of this application, it further includes: a third acquisition module, used to acquire the main load power supply status and bus occupancy status of the server before identifying the actual response status of the target processor using the first communication address of the target processor; and a third determination module, used to determine that the real-time status meets the detection conditions when the main load power supply status is not powered on, the bus occupancy status is not occupied, and the external device connected to the server is not running, and the target processor is not accessed and not upgraded.

[0185] Optionally, in one embodiment of this application, it further includes: a fourth determining module, configured to determine that the real-time state meets the detection conditions before identifying the actual response state of the target processor using the first communication address of the target processor, when the server's startup type is AC power-on type, the server's current power state is power off, the server's main load power supply state is not powered on, the bus occupancy state is not occupied, the external device connected to the server is not running, and the target processor is in an unaccessed state and an unupgraded state.

[0186] Optionally, in one embodiment of this application, the identification module 100 includes: a first acquisition unit, configured to acquire the real-time status of the board where the target processor is located; and a first identification unit, configured to identify the actual response status of the target processor by using the first communication address of the target processor when the real-time status of the board where the target processor is located is in an in-place state.

[0187] Optionally, in one embodiment of this application, the identification module 100 includes: a sending unit, configured to send a first version number reading instruction to a first communication address of the target processor via a target communication bus; a second acquisition unit, configured to acquire the version number reading status corresponding to the first version number reading instruction after sending; and a second identification unit, configured to identify the actual response status of the target processor based on the version number reading status.

[0188] Optionally, in one embodiment of this application, it further includes: a first sending module, configured to, after identifying the actual response state of the target processor using the first communication address of the target processor, generate a second version number reading instruction and send it to the first communication address of the target processor when the version number reading state corresponding to the first version number reading instruction is a reading failure state, until the number of times the version number reading instruction is sent reaches a target number; and a determination module, configured to determine that the actual response state is a non-response state when the version number reading states corresponding to the target number are all reading failure states.

[0189] Optionally, in one embodiment of this application, it further includes: a second generation module, used to determine that the actual response status is a non-response status after all version number read statuses corresponding to the target number of reads are in the read failure status, and then generate a first fault log based on the timestamp, the first communication address, the response code corresponding to the read failure status, and the fault recovery process triggering marker information corresponding to all version number read statuses corresponding to the target number of reads; and a first feedback module, used to feed the first fault log back to the server so as to store the fault status traceability data on the server.

[0190] Optionally, in one embodiment of this application, the first generation module 200 includes: a first determining unit, configured to determine that the first communication address is in a fault state when the actual response state is a non-response state; and a second determining unit, configured to determine that the target processor is in a fault state when the first communication address is in a fault state.

[0191] Optionally, in one embodiment of this application, the first generation module 200 includes: a third determining unit, which determines the self-loading recovery target of the target processor based on the fault state; and a first generation unit, which generates recovery instructions according to the self-loading recovery target.

[0192] Optionally, in one embodiment of this application, it further includes: a fourth acquisition module, configured to acquire the write status of the recovery instruction after sending the recovery instruction to the target processor using the second communication address of the target processor, so as to generate a third version number read instruction if the write status is a successful write status; a second sending module, configured to send the third version number read instruction to the target processor through the target communication bus based on the first communication address; and an identification module, configured to acquire the version number read status corresponding to the third version number read instruction, so as to identify whether the target processor responds to the recovery instruction based on the version number read status corresponding to the third version number read instruction.

[0193] Optionally, in one embodiment of this application, the identification module includes: a first determination unit, configured to determine that the target processor responds to the recovery command when the version number reading status corresponding to the third version number reading command is a successful reading status; and a second determination unit, configured to determine that the target processor does not respond to the recovery command when the version number reading status corresponding to the third version number reading command is a failed reading status.

[0194] Optionally, in one embodiment of this application, it further includes: a third generation module, configured to generate a second fault log based on the recovery instruction, the timestamp and bus status corresponding to the write failure state, the second communication address and the fault recovery process termination marker information after sending a recovery instruction to the target processor using the second communication address of the target processor, when the write state is a write failure state; and a second feedback module, configured to send the second fault log to the server to store the fault state tracing data on the server.

[0195] Optionally, in one embodiment of this application, it further includes: a fourth generation module, used to generate a third fault log based on the timestamp corresponding to the time when the target processor does not respond to the recovery instruction, the successful sending flag information of the third version number read instruction, the return value corresponding to the third version number read instruction, the first communication address, and the hardware anomaly warning flag information after determining that the target processor does not respond to the recovery instruction; and a third feedback module, used to send the third fault log to the server through the target communication bus to store the fault status traceability data on the server.

[0196] Optionally, in one embodiment of this application, it further includes: a fifth generation module, used to generate a fault recovery report based on the repair completion timestamp of the fault recovery process of the target processor, the read value of the first communication address, the total duration of fault processing, and the fault processing success marker information after determining that the target processor has exited the fault state; and a fourth feedback module, used to feed back the fault recovery report to the server.

[0197] For a description of the features in the embodiment corresponding to the fault handling device, please refer to the relevant description of the embodiment corresponding to the fault handling method, which will not be repeated here.

[0198] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described fault handling method embodiments.

[0199] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described fault handling method embodiments when it is run.

[0200] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0201] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0202] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0203] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0204] The above provides a detailed description of a fault handling assessment method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A failure handling method characterized by, Applied to a baseboard management controller, the following steps are included: Determine the real-time status of the server, and if the real-time status meets the detection conditions, identify the actual response status of the target processor using the first communication address of the target processor; If the actual response status is a non-response status, the target processor is determined to be in a fault state, and a recovery instruction for controlling the target processor to self-load is generated; The recovery command is sent to the target processor using the second communication address of the target processor, and if the target processor responds to the recovery command, it is determined that the target processor exits the fault state, thus completing the fault handling.

2. The failure handling method according to claim 1, characterized by, Also includes: Obtain the communication address in the target processor that meets the preset communication conditions, and determine the first communication address; Obtain the flash memory address in the target processor that meets the preset access conditions to determine the second communication address.

3. The failure handling method according to claim 1, characterized by, Before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: Identify the server's startup type based on the target query interface; If the startup type is AC power-on, it is determined that the real-time status meets the detection conditions.

4. The failure handling method according to claim 1, characterized by, Before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: The current power status of the server is detected based on the target management interface; When the power supply is in the off state, it is determined that the real-time state meets the detection conditions.

5. The failure handling method according to claim 1, characterized by, Before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: Obtain the main load power supply status and bus occupancy status of the server; When the main load is not powered on, the bus is not occupied, the external device connected to the server is not running, and the target processor is not accessed or upgraded, the real-time state is determined to meet the detection conditions.

6. The failure handling method according to claim 1, wherein Before identifying the actual response state of the target processor using the first communication address of the target processor, the method further includes: When the server's startup type is AC power-on, the server's current power status is power off, the server's main load power supply status is not powered on, the bus occupancy status is not occupied, the external device connected to the server is not running, and the target processor is not accessed and not upgraded, the real-time status is determined to meet the detection conditions.

7. The failure handling method according to claim 1, wherein The step of identifying the actual response state of the target processor using the first communication address of the target processor includes: Obtain the real-time status of the board where the target processor is located; When the real-time status of the board containing the target processor is in the "in-place" state, the actual response status of the target processor is identified using the first communication address of the target processor.

8. The failure handling method of claim 1, wherein, The step of identifying the actual response state of the target processor using the first communication address of the target processor includes: The first version number read instruction is sent to the first communication address of the target processor via the target communication bus; After sending, obtain the version number reading status corresponding to the first version number reading instruction; The actual response status of the target processor is identified by reading the status based on the version number.

9. The fault handling method of claim 8, wherein, After identifying the actual response state of the target processor using its first communication address, the method further includes: When the version number reading status corresponding to the first version number reading instruction is a reading failure status, a second version number reading instruction is generated and sent to the first communication address of the target processor until the number of times the version number reading instruction is sent reaches the target number; When the version number read status corresponding to the target number is always in the read failure state, the actual response status is determined to be a non-response state.

10. The failure handling method according to claim 9, wherein When the version number read status corresponding to the target number is always in the read failure state, after determining that the actual response status is in the non-response state, the method further includes: Based on the timestamp corresponding to the read status when the version number corresponding to the target number is the read failure status, the first communication address, the response code corresponding to the read failure status, and the fault recovery process triggering flag information, a first fault log is generated. The first fault log is fed back to the server so that the source data of the fault status can be stored on the server.

11. The failure handling method of claim 1, wherein, The step of determining that the target processor is in a fault state when the actual response state is a non-response state includes: If the actual response status is no response, it is determined that the first communication address is in a fault state; When the first communication address is in a fault state, it is determined that the target processor is in a fault state.

12. The fault handling method of claim 1, wherein, In the case that the actual response state is a non-response state, the target processor is determined to be in a fault state, and a recovery instruction for controlling the self-loading of the target processor is generated, including: Based on the fault state, the self-loading recovery target of the target processor is determined; The recovery instructions are generated based on the self-loading recovery target.

13. The failure handling method of claim 1, wherein, After sending the recovery instruction to the target processor using the second communication address of the target processor, the method further includes: Obtain the write status of the recovery instruction, and generate a third version number read instruction if the write status is a successful write status; Based on the first communication address, the third version number read instruction is sent to the target processor through the target communication bus; Obtain the version number read status corresponding to the third version number read instruction, so as to identify whether the target processor responds to the recovery instruction based on the version number read status corresponding to the third version number read instruction.

14. The fault handling method of claim 13, wherein, The step of identifying whether the target processor responds to the recovery instruction based on the version number read status corresponding to the third version number read instruction includes: When the version number read status corresponding to the third version number read instruction is a read success status, it is determined that the target processor responds to the recovery instruction; When the version number read status corresponding to the third version number read instruction is a read failure status, it is determined that the target processor has not responded to the recovery instruction.

15. The fault handling method of claim 13, wherein, After sending the recovery instruction to the target processor using the second communication address of the target processor, the method further includes: When the write status is a write failure status, a second fault log is generated based on the recovery instruction, the timestamp and bus status corresponding to the write failure status, the second communication address, and the fault recovery process termination marker information; The second fault log is sent to the server to store the source data of the fault status on the server.

16. The fault handling method according to claim 1, characterized in that, After determining that the target processor does not respond to the recovery command, the method further includes: If the target processor does not respond to the recovery command, a third fault log is generated based on the timestamp corresponding to the time when the target processor does not respond to the recovery command, the successful sending flag information of the third version number read command, the return value corresponding to the third version number read command, the first communication address, and the hardware anomaly warning flag information. The third fault log is sent to the server via the target communication bus to store the fault status tracing data on the server.

17. The fault handling method according to claim 1, characterized in that, After determining that the target processor has exited the fault state, the process further includes: A fault recovery report is generated based on the repair completion timestamp of the fault recovery process of the target processor, the read value of the first communication address, the total duration of the fault handling, and the fault handling success marker information. The fault recovery report is then sent back to the server.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the fault handling method as described in any one of claims 1 to 17.

19. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault handling method as described in any one of claims 1 to 17.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the fault handling method as described in any one of claims 1 to 17.

Images