Server troubleshooting methods and electronic equipment

Abstract

This application discloses a server fault handling method and electronic device, relating to the field of server technology. The server includes a first processor and a second processor. The method monitors the operating data of the first processor in real time. When the operating data indicates that the first processor is in an abnormal state, the method disconnects the communication links of each port of the first processor. The method translates the physical addresses of each port of the first processor in the server to the corresponding physical addresses in the second processor, and translates the storage addresses of the memory data of the first processor in the server to the corresponding storage addresses in the second processor, so as to build the port communication links and memory space in the second processor. The method constructs a virtual configuration component in the second processor and configures the virtual configuration component according to the configuration information of the first processor, so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor. The method then takes over the first processor through the virtual configuration component in the second processor.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to server fault handling methods and electronic devices. Background Technology

[0002] In traditional dual-processor server architectures, data isolation exists between processors, making cross-domain access impossible. As a result, when one processor fails, its associated firmware and hardware become unusable. Furthermore, the devices it manages are strongly bound to the processor due to physical links and configuration space, making them unmanageable by other processors, leading to server downtime. Summary of the Invention

[0003] This application provides a server fault handling method and electronic device to at least solve the problem that when one of the processors fails, it cannot be taken over by other processors, resulting in server operation interruption.

[0004] This application provides a method for handling server faults, wherein the server includes a first processor and a second processor, comprising:

[0005] Real-time monitoring of the operating data of the first processor;

[0006] When the first processor is determined to be in an abnormal state based on the operating data, the communication links of each port of the first processor are disconnected.

[0007] The physical addresses of each port of the first processor in the server are translated to the corresponding physical addresses in the second processor, and the storage addresses of the memory data of the first processor in the server are translated to the corresponding storage addresses in the second processor, so as to build the communication links and memory space of the ports in the second processor.

[0008] A virtual configuration component is constructed in the second processor, and the virtual configuration component is configured according to the configuration information of the first processor so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor.

[0009] The first processor is taken over by a virtual configuration component in the second processor.

[0010] This application also provides a server fault handling apparatus, the server including a first processor and a second processor, comprising:

[0011] The acquisition module is used to monitor the operating data of the first processor in real time;

[0012] The processing module is used to disconnect the communication links of each port of the first processor when it is determined from the running data that the first processor is in an abnormal state.

[0013] The processing module is also used to translate the physical addresses of each port of the first processor in the server to the corresponding physical addresses in the second processor, and to translate the storage addresses of the memory data of the first processor in the server to the corresponding storage addresses in the second processor, so as to build the communication links and memory space of the ports in the second processor.

[0014] The processing module is also used to construct a virtual configuration component in the second processor and configure the virtual configuration component according to the configuration information of the first processor, so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor.

[0015] The processing module is also used to take over the first processor through a virtual configuration component in the second processor.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described server fault handling methods.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described server fault handling methods.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described server fault handling methods.

[0019] This application describes a server comprising a first processor and a second processor. The server monitors the operational data of the first processor in real time. When the operational data indicates that the first processor is in an abnormal state, the communication links of all ports of the first processor are disconnected. The physical addresses of all ports of the first processor in the server are translated to corresponding physical addresses in the second processor, and the storage addresses of the memory data of the first processor in the server are translated to corresponding storage addresses in the second processor, thereby establishing communication links and memory space for the ports in the second processor. A virtual configuration component is constructed in the second processor, and configured according to the configuration information of the first processor, ensuring that the configuration information of the virtual configuration component is identical to that of the first processor. The virtual configuration component in the second processor then takes over the first processor. This solution achieves rapid takeover of a faulty CPU and seamless migration of device resources in a dual-CPU server architecture, breaking the physical binding relationship between the CPU and PCIe devices in traditional architectures. This allows the other CPU to take over device resources when one CPU fails, completely eliminating the risk of service interruption due to hardware failure, while avoiding the resource waste caused by long-term idle backup devices in traditional dual-machine redundancy solutions. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a server fault handling method provided in this application embodiment Figure 1 ;

[0022] Figure 2 A flowchart of a server fault handling method provided in this application embodiment Figure 2 ;

[0023] Figure 3 A structural diagram of a server fault handling device provided in an embodiment of this application;

[0024] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0026] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0028] In traditional dual-CPU server architectures, the binding relationship between the CPU and PCIe devices presents significant fault tolerance deficiencies. PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard used to connect various hardware components on the motherboard, such as graphics cards, storage devices, and network adapters. When one CPU (CPU0) fails, the PCIe devices it manages are strongly bound to the Root Complex (RC) of the failed CPU due to physical links and configuration spaces, making them impossible for the other, normally functioning CPU (CPU1) to directly take over. This leads to service interruptions or necessitates inefficient software redirection schemes (such as network storage mapping or virtualization migration). The Root Complex (RC) is a core component of the PCIe bus architecture, responsible for connecting the CPU / memory subsystem and PCIe devices, managing the initialization, data communication, address mapping, and device configuration of the entire PCIe subsystem. Current processor fault takeover technologies face the following core challenges:

[0029] PCIe device access isolation: The PCIe bus domain and memory domain are inherently isolated. Cross-domain access requires address mapping through an Address Translation Unit (ATU). In traditional solutions, CPU1 cannot dynamically take over the PCIe bus domain address space of CPU0, resulting in interruption of device access paths. For example, after CPU0 fails, CPU1 cannot directly access the PCIe device BAR space of CPU0 (such as 0xA0000000-0xAFFFFFFF) through the original physical link.

[0030] Redundant architectures waste resources: Traditional dual-machine hot standby or dual-root architectures require a separate PCIe device for each CPU, resulting in low hardware utilization and high costs. For example, financial trading systems need to deploy redundant graphics processing units (GPUs) for each CPU, leading to resource idle rates exceeding 40%.

[0031] Switching latency and reliability bottlenecks: Existing fault switching relies on operating system-level device re-enumeration, which takes several seconds to tens of seconds, failing to meet the requirements of high real-time scenarios (such as autonomous driving and high-frequency trading). In addition, PCIe link status detection relies on only a single signal (such as a heartbeat packet), which has a high risk of misjudgment and may trigger cascading failures.

[0032] Interrupt and DMA consistency challenges: The interrupt vectors (MSI-X) and DMA channels of CPU0 devices are bound to the APIC and memory space of CPU0. During fault switching, interrupt routing and input-output memory management unit (IOMMU) mapping need to be reconfigured, which can easily lead to data loss or system crash.

[0033] Current technologies for processor failure takeover utilize the multiplexing function of PCIe switches to achieve device sharing. The primary and backup servers periodically synchronize device configuration information to shared storage. During failover, the backup server needs to rescan the PCIe bus and load device drivers. The core problem is that device switching relies on operating system-level bus enumeration and driver initialization processes, resulting in interrupt times exceeding 2 seconds. Furthermore, it doesn't resolve cross-CPU domain memory access conflicts, requiring additional software to intercept DMA requests. In addition, this method requires reserving backup resources for each device, leading to low hardware utilization.

[0034] Furthermore, the related technology also discloses that devices can be simultaneously mounted to two independent servers via a PCIe switch. When the primary server fails, the backup server needs to reset the device and reload the driver at the software layer to take over the device. Its drawback is that the switching process requires the operating system to restart the device driver, resulting in service interruption exceeding 5 seconds, and it does not resolve the synchronization issue of device states (such as DMA context and interrupt binding). This solution relies on software-level device reset operations, cannot achieve seamless takeover, and requires hardware resources to configure independent root complexes for both servers, increasing the complexity and cost of device management.

[0035] Furthermore, the related technology also discloses that the baseboard management controller is communicatively connected to an I / O expander and multiple interface modules. Each PCIe port of the processor is communicatively connected to a corresponding switching module. The two PCIe ports of the switching module are respectively communicatively connected to a first type interface and a second type interface in the interface module. The baseboard management controller obtains the presence signal of the first type interface in the target interface module through the I / O expander, and uses this to determine whether a device is connected to the first type interface in the target interface module and generates a corresponding control signal. Based on the control signal, the I / O expander controls one of the PCIe ports in the target switching module connected to the target interface module to connect to the corresponding PCIe port of the processor. This method requires all CPUs to share the same PCIe domain, limiting hardware scalability.

[0036] In summary, in a conventional dual-socket architecture, the system boot process relies on CPU0 (main CPU0) to initialize hardware resources. If CPU0 suffers physical damage (such as a power short circuit or core breakdown), its associated firmware code (such as the BIOS module) and hardware will completely fail, causing CPU1 to be unable to take over the boot process, resulting in a system crash. Traditional PCIe devices are strongly bound to the CPU. When CPU0 fails, the PCIe devices it manages cannot be directly accessed by CPU1 due to physical link isolation and loss of configuration space. When re-establishing a connection, the server needs to interrupt its service before reconnecting, making seamless connectivity impossible.

[0037] To address the aforementioned technical problems, this application provides a server fault handling method. The server includes a first processor and a second processor. The method monitors the operating data of the first processor in real time. When the operating data indicates that the first processor is in an abnormal state, the communication links of each port of the first processor are disconnected. The physical addresses of each port of the first processor in the server are translated to the corresponding physical addresses in the second processor, and the storage addresses of the memory data of the first processor in the server are translated to the corresponding storage addresses in the second processor, thereby establishing communication links and memory space for the ports in the second processor. A virtual configuration component is constructed in the second processor, and configured according to the configuration information of the first processor, so that the configuration information of the virtual configuration component is identical to that of the first processor. The virtual configuration component in the second processor then takes over the first processor. In this solution, a rapid takeover of a faulty CPU and seamless migration of device resources are achieved in a dual-CPU server architecture. This breaks the physical binding relationship between the CPU and PCIe devices in traditional architectures, allowing the other CPU to take over device resources when one CPU fails, completely eliminating the risk of service interruption due to hardware failure, and avoiding the resource waste caused by the long-term idleness of backup devices in traditional dual-machine redundancy solutions.

[0038] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, Figure 1 A flowchart of a server fault handling method provided for embodiments of this application is shown, the method including the following steps:

[0040] 101. Monitor the operating data of the first processor in real time.

[0041] In this embodiment of the application, the server includes a first processor and a second processor. The first processor and the second processor can be the same processor. When neither the first processor nor the second processor is faulty, business processing can be performed by only one of the processors. When one of the processors is faulty, the other processor can take over. Therefore, the operating data of the first processor can be monitored in real time. The first processor is the processor that is currently running. The operating data can include at least the temperature data of the first processor (such as the PROCHOT signal) and the link connection status data (PCIe device link).

[0042] In some embodiments, when monitoring the operating data of the first processor, the operating data of the first processor can be collected at certain time intervals through a BMC management controller or a dedicated monitoring circuit.

[0043] 102. When the first processor is determined to be in an abnormal state based on the running data, disconnect the communication links of each port of the first processor.

[0044] In this embodiment of the application, during the real-time monitoring of the operating data, if it is determined from the operating data that the first processor is in an abnormal state, it means that the first processor cannot operate normally at this time, and the server may experience service interruption. Therefore, the second processor is needed to take over the work of the first processor. However, the communication link of the first processor needs to be disconnected immediately, which is equivalent to cutting off the "command authority" of the first processor. It is equivalent to taking away its official seal and walkie-talkie, so that it can no longer issue orders to subordinate departments (hardware devices), achieving nanosecond-level electrical isolation.

[0045] 103. The physical addresses of each port of the first processor in the server are translated to the corresponding physical addresses in the second processor, and the storage addresses of the memory data of the first processor in the server are translated to the corresponding storage addresses in the second processor, so as to build the communication links and memory space of the ports in the second processor.

[0046] In this embodiment, when the first processor experiences a storage failure, its ports also become unusable. The second processor takes over the work of the first processor. This means that subsequent data transmission and business processing in the server are performed through ports on the second processor. Therefore, it is necessary to translate the port address of the first processor in the server to the corresponding address of the port in the second processor. This allows subsequent data transmission and business processing to be performed directly according to the port address in the second processor. Additionally, the processor has a memory space containing data, which may include system data generated during processor operation, transmitted and received data, and business data. Therefore, it is also necessary to transfer the memory data of the first processor. This can be achieved by translating the storage address of the memory data in the first processor. In other words, after translating the storage address of the memory data in the first processor to the corresponding storage address in the second processor, the second processor can take over the memory data in the first processor according to that storage address, thereby constructing the memory space in the second processor. This is equivalent to copying the memory space of the first processor.

[0047] 104. Construct a virtual configuration component in the second processor, and configure the virtual configuration component according to the configuration information of the first processor so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor.

[0048] In this embodiment of the application, in order to enable the second processor to take over according to the configuration of the first processor and achieve seamless takeover of the processor, a virtual configuration component can be constructed in the second processor. Then, the virtual configuration component is configured according to the configuration information of the first processor so that the configuration information of the configured virtual configuration component is the same as the configuration information of the first processor. In this way, when the second server is running, the user or external device will think that the processor is still intact and will not be able to perceive that the original first processor has been taken over.

[0049] It's important to note that this virtual configuration component can be a virtual Root Complex. The Root Complex (RC) is a core component of the PCIe bus architecture, responsible for connecting the CPU / memory subsystem to PCIe devices and managing the initialization, data communication, address mapping, and device configuration of the entire PCIe subsystem. Simply put, this virtual Root Complex is like a "virtual CEO's office." The layout, telephone extension numbers (ECAM configuration space, Bus / Device / Function numbers) of this "virtual CEO's office" are exactly the same as the primary processor. This way, other employees in the company (external devices, operating systems, and applications) are completely unaware that the CEO has changed and believe everything is normal, thus achieving seamless processor takeover.

[0050] 105. Take over the first processor through the virtual configuration component in the second processor.

[0051] In this embodiment, after the physical addresses of each port, the storage addresses of memory data, and the virtual configuration components in the second processor are all set, the second processor is now equivalent to a copy processor of the first processor. Therefore, the virtual configuration components in the second processor can take over the first processor, and subsequent data transmission and business processing in the server can be implemented through the second processor.

[0052] In this embodiment, the server includes a first processor and a second processor. The server monitors the operating data of the first processor in real time. When the operating data indicates that the first processor is in an abnormal state, the communication links of each port of the first processor are disconnected. The physical addresses of each port of the first processor in the server are translated to the corresponding physical addresses in the second processor, and the storage addresses of the memory data of the first processor in the server are translated to the corresponding storage addresses in the second processor, thereby constructing communication links and memory space for the ports in the second processor. A virtual configuration component is constructed in the second processor, and configured according to the configuration information of the first processor, so that the configuration information of the virtual configuration component is the same as that of the first processor. The virtual configuration component in the second processor takes over the first processor. In this scheme, a rapid takeover of a faulty CPU and seamless migration of device resources are achieved in a dual-CPU server architecture. This breaks the physical binding relationship between the CPU and PCIe devices in traditional architectures, allowing the other CPU to take over device resources when one CPU fails, completely eliminating the risk of service interruption due to hardware failure, and avoiding the resource waste caused by the long-term idleness of backup devices in traditional dual-machine redundancy schemes.

[0053] like Figure 2 As shown, Figure 2 Another flowchart of a server fault handling method provided for embodiments of this application, the method may include the following steps:

[0054] 201. Monitor the operating data of the first processor in real time.

[0055] In this embodiment, the description of step 201 is the same as the detailed description of step 101 in the above embodiments, and will not be repeated in this embodiment.

[0056] 202. When the temperature of the first processor is detected to be lower than the preset temperature value for a preset duration, and / or when the link connection status of the first processor is detected to be in an abnormal state, it is determined that the first processor is in an abnormal state.

[0057] In this embodiment, during normal operation, the processor's temperature rises and significantly exceeds room temperature. If the processor malfunctions and stops operating, its temperature drops, approaching room temperature. Therefore, the processor's temperature can be used to determine if it is faulty. If the processor's temperature remains below a preset value for a preset duration, indicating it has been operating at a low temperature for some time, it can be assumed that the processor is currently faulty. Furthermore, during normal operation, the processor's communication links are functioning correctly. If communication is impossible, indicating a link connection failure, this also suggests a processor malfunction. Therefore, the processor's link connection status can be used to determine if it is faulty. If the first processor's link connection status is abnormal, it can be assumed that the first processor is currently faulty.

[0058] It should be noted that the temperature and link connection status mentioned above are two methods for judging processor faults. The processor can be considered to be faulty if at least one of the above judgment methods is met. That is, the first processor can be considered to be in an abnormal state when the temperature of the first processor is lower than the preset temperature value for a preset duration, or when the link connection status of the first processor is in an abnormal state, or when the temperature of the first processor is lower than the preset temperature value for a preset duration and the link connection status of the first processor is in an abnormal state.

[0059] In this embodiment, the processor is judged to be faulty by two parameters: processor temperature and link connection status, which improves the accuracy of fault diagnosis and allows another processor to take over in a timely manner when a processor fault is detected.

[0060] In some embodiments, dual independent power supplies and signal isolation are required at the hardware level. Independent VRMs (voltage regulator modules) are configured for the first and second processors, and supercapacitors (such as Maxwell 2.7V3000F) are added to the PCIe device slots to maintain power supply to the device for ≥60 seconds after the first processor loses power. The BMC monitors the PROCHOT signal of the first processor via SMBus and simultaneously polls the LTSSM status of the PCIe link (every 10ms). Continuous abnormalities indicate that the first processor has failed.

[0061] 203. When the first processor is determined to be in an abnormal state based on the running data, the communication links of each port of the first processor are disconnected by adjusting the global reset signal value of the first processor.

[0062] In this embodiment, disconnecting the communication link of the first processor's ports can be achieved using a global reset signal, specifically the PCIe_RST# signal. The PCIe_RST# signal is a globally active low-level signal in the PCIe bus used to initialize the internal logic of the device. It triggers the reset of the device registers and state machine through hardware means, ensuring that the device enters a controllable initial state during system startup or in case of an anomaly. The PCIe_RST# signal has an active low characteristic; that is, pulling the signal low triggers a reset, and pulling it high releases the reset. The reset restores the device's internal registers (such as configuration registers and status registers), timing logic, and state machine to their initial values, clearing any temporary states in operation. During normal operation of the first processor, the global reset signal value can be 1. When the first processor is in an abnormal state, the global reset signal value can be adjusted from 1 to 0, thereby disconnecting the communication links of all ports of the first processor.

[0063] In some embodiments, after detecting a failure of the first processor, the FPGA-based arbitration module immediately pulls down its PCIe_RST# signal and cuts off key control signals such as PERST# and CLKREQ# through an optocoupler relay to achieve nanosecond-level electrical isolation.

[0064] 204. Update the physical addresses of each port of the first processor stored in the server's routing table to the physical addresses of the corresponding ports in the second processor, so as to establish communication links for the ports in the second processor.

[0065] In this embodiment, during the translation of the physical addresses of each port of the first processor, these physical addresses can be stored in the server's routing table. Therefore, the physical addresses of the ports stored in the routing table can be updated, that is, the physical addresses of the ports of the first processor are updated to the corresponding physical addresses of the ports in the second processor. The server's routing table is dynamically modified via a PCIe switch, redirecting the PCIe ports (Ports 0-7) originally belonging to the first processor to the second processor domain. This is combined with address space translation (e.g., Outbound ATU maps the device BAR address 0xA0000000 of the first processor to 0xD0000000 in the second processor domain, and Inbound reverse lookup), ensuring transparent physical layer access for the device. Simply put, it can be understood as a company's "internal telephone switchboard" (PCIe switch) transferring all internal calls (data requests) originally destined for the first processor to the second processor's line. This process can be completed within 10 milliseconds (ms).

[0066] 205. In response to a memory access request, determine the storage address of the memory data from the first processor in the server, and redirect the storage address of the memory data to the pre-stored memory pool corresponding to the second processor, so as to construct memory space in the second processor.

[0067] In this embodiment, when converting memory data, it can be done based on a memory access request. This memory access request can be equivalent to a "key" to access the memory space of the first processor. The storage address of the memory data is determined from the first processor and then redirected to the corresponding new address in the second processor. This new address can be understood as a shared memory space, specifically a CMA memory pool or CXL memory pool that can be accessed at high speed by the CPU. This allows the data originally stored in the first processor to be found in the pre-stored memory pool, and new data will also be stored in the pre-stored memory pool.

[0068] In simple terms, the files being processed by the first processor are locked in their own "file cabinet." The system uses the IOMMU (a type of memory management unit) to assign a "master key" to the second processor, dynamically mapping the address of the first processor's file cabinet (e.g., 0xA0000000) to a new address (e.g., 0xD0000000) that the second processor can recognize. This allows the second processor to seamlessly continue processing this data, achieving "zero-copy" transfer—the data itself doesn't need to be physically moved; only a different key is used to open it. This shared file space can be understood as a CMA or CXL memory pool that the CPU can access at high speed.

[0069] In some embodiments, PCIe device takeover and virtualization reconstruction achieve cross-domain transparent access based on a non-transparent bridge (NTB). Core technologies include address isolation mapping, cross-domain interrupt forwarding, and virtual root complex emulation. The NTB maps the physical address of a device in the first processor domain (e.g., 0xA0000000) to the virtual address of a device in the second processor domain (0xD0000000) by configuring an Outbound / Inbound Address Translation Unit (ATU), and isolates the TLP transmission paths between the two domains to avoid address conflicts. At the interrupt level, the NTB intercepts MSI-X messages from the first processor device, rewrites the target address to the APIC ID of the second processor (0xFEE00000→0xFEE01000), and dynamically allocates vector numbers using the kernel interrupt remapping table (IRT) to achieve cross-domain interrupt response. DMA operations redirect device requests to the CMA memory pool reserved in the CPU1 domain through the IOMMU dynamic page table, and achieve zero-copy transfer using NTB reverse address resolution.

[0070] In this embodiment, the continuity of business processes and data consistency are ensured by updating and redirecting the physical addresses of each port in the first processor and the storage addresses of memory data, combined with memory cache synchronization and IO address remapping technology.

[0071] 206. Build a virtual configuration component in the second processor.

[0072] In this embodiment, the description of step 206 is the same as the detailed description of step 104 in the above embodiments, and will not be repeated in this embodiment.

[0073] 207. Start the firmware image in the second processor.

[0074] In this embodiment, through the redundancy design mechanism of BMC, when a failure is detected in the firmware storage area (SPI Flash) of the first processor, the system is automatically switched to the independent firmware image of the second processor to complete the system boot, ensuring high availability under hardware failure.

[0075] It should be noted that the second processor is equipped with an independent SPI Flash storage area to store a UEFI firmware image completely isolated from the first processor. This image contains a complete boot program (such as U-Boot, kernel, and file system) and can independently perform hardware initialization, driver loading, and operating system booting. This firmware image is isolated from the firmware in the first processor to avoid the propagation of a single point of failure. This firmware image contains a complete UEFI boot process (such as SEC, PEI, and DXE stages) and can independently perform hardware initialization, driver loading, and operating system booting without relying on the firmware state of the first processor. The BMC monitors the fault status of the first processor through a hardware timeout mechanism (such as a watchdog timer). If the first processor fails, the BMC automatically switches the SPI interface to the Flash area of ​​the second processor, triggering the backup image boot.

[0076] 208. Retrieve the configuration information of the first processor from the pre-stored global device list.

[0077] 209. Based on the firmware image, configure the virtual configuration components according to the configuration information of the first processor.

[0078] In this embodiment of the application, when configuring the virtual configuration component, it can be done according to the configuration information of the first processor. The configuration information of the first processor can be stored in the global device list. Therefore, the configuration information of the first processor can be retrieved from the global device list for configuration, so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor.

[0079] In some embodiments, the Global Device List (GDT) should actually be the Global Descriptor Table (GDT), which is the core data structure used for memory segmentation management in x86 architecture protection mode. The GDT is a table stored in memory that defines the attributes of each memory segment in the system, including base address, segment limit (i.e., segment size), access permissions, etc. Each memory segment can be understood as a contiguous region in memory used to store information such as code, data, or stack.

[0080] In some embodiments, the BIOS pre-scans dual-domain PCIe device information and writes it into NVRAM (such as Intel OptanePMem) to form a global device list (GDT). The second processor then copies the first processor's ECAM configuration space (Bus / Device / Function number and Class Code) to the virtual root complex, causing the operating system to mistakenly believe that the device is still mounted in its original location.

[0081] In simple terms, the created virtual root complex is like creating a fake "hardware management center" for another "brain" (such as a second processor) within a server, allowing it to take over and control hardware devices (such as network cards and GPUs) that were not originally under its management. When a primary "brain" (such as the primary processor) in the server suddenly fails, to prevent the hardware devices it manages from becoming unusable, the system quickly replicates a virtual hardware configuration identical to the original environment on another healthy "brain" (the second processor), based on a pre-prepared "hardware map" (Global Device List, GDT). Thus, when the operating system runs on the second processor, it mistakenly believes that all hardware is still mounted in its original location, allowing it to continue using these devices without modifying any software, achieving a near-seamless failover.

[0082] In some embodiments, firmware initialization and configuration are seamlessly taken over through dual Boot ROM switching and virtualization context reconstruction. By using a virtual configuration component with the same configuration as the first processor, the second processor can take over and control the hardware that was originally managed by the first processor, thus achieving near-seamless failover of the processor.

[0083] 210. Delete the configuration data of the first processor stored in the configuration management list, and update the configuration management list according to the configuration data of the second processor.

[0084] In this embodiment of the application, since the first processor is faulty, the second processor takes over the first processor in the future. Therefore, the configuration data of the processor can be updated in the configuration management list, that is, the configuration data of the first processor that was originally stored is updated to the configuration data of the second processor.

[0085] In some embodiments, the configuration management list can be an Advanced Configuration and Power Interface (ACPI) table. ACPI is a standard interface for interaction between the operating system and hardware. The ACPI table is used for system hardware configuration and power management and includes sub-tables such as DSDT (Differentiated System Description Table), MADT (Multiple APIC Description Table), and SRAT (System Resource Affinity Table).

[0086] The MADT (Multiple APIC Description Table) is a table in ACPI that describes the configuration of the Advanced Programmable Interrupt Controller (APIC) in the system. It includes a Local APIC entry, typically one for each CPU core, used for interrupt handling and inter-processor communication. When the first processor fails, its Local APIC entry in the MADT can be deleted, meaning the interrupt controller description for that processor is removed from the ACPI table.

[0087] The System Resource Affinity Table (SRAT) is another table in ACPI that describes the affinity between memory and processors, i.e., which memory regions are associated with which processor core or node. When the first processor fails, the SRAT memory topology can be reconstructed, reorganizing the memory-processor associations. Associating memory nodes originally allocated to the first processor with the second processor domain means that memory originally allocated to the first processor is now associated with the second processor's domain, thus reallocating memory nodes to avoid conflicts when the operating system identifies the CPU and memory.

[0088] 211. In response to a hot-plug trigger signal, load the system driver from the second processor.

[0089] 212. Based on system drivers, the first processor is taken over through a virtual configuration component.

[0090] In this embodiment, operating system recovery and driver loading achieve seamless service recovery through kernel hot-plug event triggering and dynamic resource redirection. After the hardware layer switch is complete, the kernel simulates a PCIe device hot-plugging via an ACPI HotPlug event (e.g., _OST0x103), triggering the driver manager (e.g., udev) to load pre-registered backup drivers (e.g., NVIDIA GPU drivers). The kernel receives a simulated "new device inserted" signal (ACPI HotPlug event), which triggers it to rescan the hardware. However, since the virtual configuration components ("virtual office") have already been configured, the kernel scan will find all hardware intact and running in place. Therefore, it will automatically load the drivers for these hardware components (e.g., GPU, network card), allowing services to resume normal operation.

[0091] 213. Monitor the first data sent by the first processor and the second data received in real time.

[0092] 214. Intercept the first data and the second data, and forward both the first data and the second data to the second processor.

[0093] In this embodiment of the application, after the first processor fails, all services of the first processor will be implemented through the second processor. Therefore, the second data sent to the first processor needs to be forwarded to the second processor. Similarly, the data sent by the first processor may be corrupted due to the failure of the first processor, so it is necessary to intercept the first data sent by the first processor.

[0094] In some embodiments, the NTB intercepts MSI-X messages from the first processor's device, rewrites the target address to the APIC ID of the second processor (0xFEE00000→0xFEE01000), and dynamically allocates vector numbers using the kernel interrupt remapping table (IRT) to achieve cross-domain interrupt response. This can be understood as follows: previously, when the hardware department encountered a problem, it needed to request instructions from the first processor (sending an interrupt signal, such as MSI-X). Now, these requests are automatically transferred to the second processor (APIC controller) by the "Interrupt Routing Table (IRT)," ensuring unimpeded communication and timely response.

[0095] In this embodiment, when the first processor malfunctions and cannot continue to work, the data sent by the first processor may be abnormal and therefore needs to be intercepted. At the same time, the data sent to the first processor will not be processed further. Therefore, the data sent and received by the first processor can be forwarded to the second processor. This can ensure data integrity, prevent data loss, and enable the second processor to process the data in a timely manner.

[0096] In this embodiment, a high-availability solution for a dual-CPU server is implemented through a hardware-firmware-operating system collaborative mechanism, allowing the second processor to take over and access its PCIe devices when the first processor fails. First, the BMC reads the PROCHOT (temperature signal) of the first processor via SMBus. If an anomaly is detected three times consecutively, the BMC triggers a system management interrupt to notify the second processor to initiate takeover. It also pulls the PCIe_RST# signal of the first processor low, causing the optocoupler relay to disconnect the control link and trigger the FPGA to perform hardware isolation. Subsequently, the BMC switches the SPI Flash chip select signal, forcing boot from the UEFI image of the second processor. After the UEFI image of the second processor boots, the PCIe switch dynamically switches the device port to the second processor domain, and the non-transparent bridge (NTB) synchronously updates the address mapping table (0xA0000000→0xD0000000), achieving transparent physical layer migration. The operating system activates the standby driver instance through a hot-plug event, dynamically remaps the IOMMU page table, migrates the GPU memory context, and finally restores service.

[0097] In some embodiments, the server fault handling method provided in this application is explained in simple terms below. Under normal circumstances, CPU0 of a high-end server is responsible for handling all daily business, while CPU1 is in standby mode, constantly synchronizing with the work progress of CPU0. To ensure absolute reliability, the entire system is designed with an extremely rapid automatic failover scheme.

[0098] Step 1: The ever-vigilant "health monitor" (fault detection):

[0099] The "health monitor" (usually a BMC management controller or a dedicated monitoring circuit) checks the CPU0's "pulse" (such as the PROCHOT signal) and the PCIe device links it manages every very short time (e.g., 10 milliseconds) to see if communication is smooth.

[0100] Quick determination: Once CPU0 abnormality or PCIe device link loss is detected several times in a row, the monitor can determine that CPU0 has lost its working capability within milliseconds.

[0101] Activate emergency response: The monitor will immediately notify the "emergency command center" (usually an FPGA chip) to activate the emergency plan.

[0102] Step 2: Isolation and Takeover (Hardware Layer Switching):

[0103] Upon receiving an alarm, the Emergency Command Center (FPGA) will immediately perform two key operations:

[0104] Forced Rest: First, it cuts off the "command authority" of CPU0, such as by pulling down its PCIe reset signal, which is equivalent to taking away its official seal and walkie-talkie, making it unable to issue orders to its subordinate departments (hardware devices), thus achieving nanosecond-level electrical isolation.

[0105] Seamless Takeover: The command center will activate a "virtual general manager's office" (virtual root complex) for CPU1. This virtual office's layout, telephone extension numbers (ECAM configuration space, Bus / Device / Function numbers) are identical to those of CPU0. Simultaneously, the company's internal switchboard (PCIe switch) will instantly redirect all internal calls (data requests) originally destined for CPU0 to CPU1. In this way, other employees in the company (operating system and applications) will be completely unaware that the general manager has changed and will believe everything is normal.

[0106] Step 3: Update contacts and permissions (firmware and memory switching):

[0107] CPU1 has now taken over from CPU0 to support the server's normal startup, but it needs to be able to process business operations normally.

[0108] Update the employee handbook: The company's "basic administrative system" (BIOS / UEFI firmware) will initiate a backup plan (standalone UEFI image) specifically for CPU1. It will modify the company's "organizational chart" (ACPI table), cross out CPU0's name, clearly indicate that CPU1 is the new person in charge, and tell the system that all resources (such as memory) now report to CPU1 to avoid confusion of permissions.

[0109] Authorizing access to the shared file cabinet: The most crucial step is handling the "shared file cabinet" (memory data). Files being processed by CPU0 are locked in their own file cabinet. The system uses the IOMMU (a memory management unit) to assign a "master key" to CPU1, dynamically mapping the address of CPU0's file cabinet (e.g., 0xA0000000) to a new address (e.g., 0xD0000000) that CPU1 can recognize. This allows CPU1 to seamlessly continue processing this data, achieving "zero-copy" transfer—the data itself doesn't need to be physically moved; only a different key is used to open it. This shared file space can be understood as a CMA or CXL memory pool that all CPUs can access at high speed.

[0110] Step 4: Restore normal operation (operating system and driver restoration):

[0111] Once the hardware and firmware switch is complete, the company's "management level" (operating system kernel) needs to be notified.

[0112] Simulated hot-plugging: The kernel receives a simulated "new device inserted" signal (ACPI HotPlug event), which triggers it to rescan the hardware. However, since the "virtual office" has already been set up, the kernel will find that all hardware is intact and in place after scanning. Therefore, it will automatically load the drivers for these hardware devices (such as GPUs and network cards) to restore normal operation of the service.

[0113] Redirection work report: Previously, when the hardware department encountered problems, it needed to consult the general manager (by issuing an interrupt signal, such as MSI-X). Now, these requests will be automatically transferred to the "secretariat" (APIC controller) of CPU1 by the "Interrupt Routing Table (IRT)" to ensure smooth communication and timely response.

[0114] In some embodiments, through the collaborative innovation of hardware redundancy design, firmware-level fault isolation, and dynamic resource management of the operating system, a dual-CPU server architecture achieves rapid takeover of a failed CPU and seamless migration of device resources, bringing multiple substantial improvements to high-availability computing scenarios. Its core value lies in breaking the physical binding relationship between the CPU and PCIe devices in traditional architectures through non-transparent bridge technology and virtualization layer reconstruction. This allows another CPU to take over device resources when one CPU fails, completely eliminating the risk of service interruption due to hardware failure, while avoiding the resource waste caused by long-term idle backup devices in traditional dual-machine redundancy solutions. This solution employs a global device table pre-registration and interrupt redirection mechanism to ensure that the operating system can dynamically identify and load cross-domain device drivers without restarting. Combined with memory cache synchronization and IO address remapping technology, it ensures the continuity of business processes and data consistency, making it particularly suitable for real-time transaction systems and high-throughput data processing scenarios with extremely low tolerance for service interruptions. The hardware layer achieves precise isolation of fault domains through intelligent power management and signal isolation mechanisms. Combined with PCIe switching network dynamic routing switching technology, it avoids the risk of fault propagation at the physical level. At the same time, it supports on-demand scheduling of heterogeneous computing devices and parallel processing of mixed loads, significantly improving hardware resource utilization and system flexibility.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0116] like Figure 3 As shown, embodiments of this application also provide a server fault handling apparatus, which may include:

[0117] The acquisition module 301 is used to monitor the operating data of the first processor in real time;

[0118] The processing module 302 is used to disconnect the communication links of each port of the first processor when it is determined from the running data that the first processor is in an abnormal state.

[0119] The processing module 302 is also used to convert the physical addresses of each port of the first processor in the server to the corresponding physical addresses in the second processor, and to convert the storage addresses of the memory data of the first processor in the server to the corresponding storage addresses in the second processor, so as to build the communication links and memory space of the ports in the second processor.

[0120] The processing module 302 is also configured to construct a virtual configuration component in the second processor and configure the virtual configuration component according to the configuration information of the first processor, so that the configuration information of the virtual configuration component is the same as the configuration information of the first processor.

[0121] The processing module 302 is also used to take over the first processor through a virtual configuration component in the second processor.

[0122] In some embodiments, the processing module 302 is further configured to determine that the first processor is in an abnormal state when the temperature of the first processor is detected to be lower than a preset temperature value for a preset duration, and / or when the link connection status of the first processor is detected to be in an abnormal state.

[0123] In some embodiments, the processing module 302 is specifically used to disconnect the communication links of each port of the first processor by adjusting the global reset signal value of the first processor when it is determined from the running data that the first processor is in an abnormal state.

[0124] In some embodiments, the processing module 302 is specifically used to update the physical addresses of each port of the first processor stored in the routing table of the server to the physical addresses of the corresponding ports in the second processor, so as to establish a communication link of the ports in the second processor.

[0125] In some embodiments, the processing module 302 is specifically configured to, in response to a memory access request, determine the storage address of memory data from a first processor in the server, and redirect the storage address of the memory data to a pre-stored memory pool corresponding to a second processor, so as to construct a memory space in the second processor.

[0126] In some embodiments, the processing module 302 is specifically used to start the firmware image in the second processor, wherein the firmware image is isolated from the firmware in the first processor.

[0127] The processing module 302 is specifically used to retrieve the configuration information of the first processor from the pre-stored global device list;

[0128] The processing module 302 is specifically used to configure the virtual configuration component based on the firmware image and the configuration information of the first processor.

[0129] In some embodiments, the processing module 302 is further configured to delete the configuration data of the first processor stored in the configuration management list and update the configuration management list according to the configuration data of the second processor.

[0130] In some embodiments, the processing module 302 is specifically configured to load a system driver from a second processor in response to a hot-plug trigger signal;

[0131] Processing module 302 is specifically used to take over the first processor based on the system driver through a virtual configuration component.

[0132] In some embodiments, the acquisition module 301 is further configured to monitor in real time the first data sent by the first processor and the second data received;

[0133] The processing module 302 is also used to intercept the first data and the second data, and forward both the first data and the second data to the second processor.

[0134] In the embodiments of this application, the description of the features corresponding to the server fault handling device in the embodiments can be found in the relevant description of the server fault handling method in the embodiments, and will not be repeated here.

[0135] like Figure 4 As shown, embodiments of this application also provide an electronic device, including a memory 401 and a processor 402, wherein the memory 401 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above-described server fault handling method embodiments.

[0136] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described server fault handling method embodiments when it is run.

[0137] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0138] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described server fault handling method embodiments.

[0139] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described server fault handling method embodiments.

[0140] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] The foregoing has provided a detailed description of the process monitoring of a storage system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A failure processing method of a server, characterized by, The server comprises a first processor and a second processor, and the method comprises: monitoring running data of the first processor in real time; when it is determined according to the running data that the first processor is in an abnormal state, disconnecting communication links of each port of the first processor; updating physical addresses of the each port of the first processor stored in a routing table of the server to physical addresses of corresponding ports in the second processor, to construct communication links of the ports in the second processor, and in response to a memory access request, determining a storage address of memory data from the first processor in the server, and redirecting the storage address of the memory data to a corresponding pre-stored memory pool in the second processor, to construct a memory space in the second processor; constructing a virtual configuration component in the second processor, and configuring the virtual configuration component according to configuration information of the first processor, so that configuration information of the virtual configuration component is the same as the configuration information of the first processor; taking over the first processor by the virtual configuration component in the second processor.

2. The method of claim 1, wherein, After the running data of the first processor is monitored in real time, the method further comprises: when a duration in which the temperature of the first processor is less than a preset temperature value reaches a preset duration, and / or when a link connection state of the first processor is detected to be in an abnormal state, it is determined that the first processor is in an abnormal state.

3. The method of claim 1, wherein, When it is determined according to the running data that the first processor is in an abnormal state, the communication links of the each port of the first processor are disconnected, comprising: when it is determined according to the running data that the first processor is in an abnormal state, disconnecting the communication links of the each port of the first processor by adjusting a global reset signal value of the first processor.

4. The method of claim 1, wherein, The virtual configuration component is configured according to the configuration information of the first processor, comprising: starting a firmware image in the second processor, the firmware image being isolated from a firmware in the first processor; calling configuration information of the first processor from a pre-stored global device list; based on the firmware image, configuring the virtual configuration component according to the configuration information of the first processor.

5. The method of claim 1, wherein, The method further comprises: deleting configuration data of the first processor stored in a configuration management list, and updating the configuration management list according to configuration data of the second processor.

6. The method of claim 1, wherein, The first processor is taken over by the virtual configuration component in the second processor, comprising: in response to a hot plug trigger signal, loading a system driver from the second processor; based on the system driver, taking over the first processor by the virtual configuration component.

7. The method of claim 1, wherein, The method further comprises: monitoring first data sent by the first processor and second data received by the first processor in real time; intercepting the first data and the second data, and forwarding the first data and the second data to the second processor.

8. An electronic device, comprising: comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the method for handling a failure of a server according to any one of claims 1 to 7.

Images