A method for implementing a dual-machine hot standby system based on FPGA fault detection

Abstract

The present invention relates to a method for implementing a dual-machine hot standby system based on FPGA fault detection, and belongs to the field of fault detection. The hardware environment of the dual-machine hot standby system based on FPGA fault detection of the present invention is composed of two mainboards integrated with FPGA modules and sensor chips, which serve as the master and slave respectively; each mainboard runs fault detection driver software, system monitoring software, and IP migration control software. The dual-machine hot standby system can detect system operation failures or hardware failures within milliseconds and perform master-slave switching, and is suitable for application scenarios with high requirements for real-time fault detection. The present invention performs fault detection through the FPGA module, which improves the real-time performance of fault detection. It can not only detect the operating status of the operating system and the status of key processes, but also detect hardware failures.

Description

Technical Field

[0001] The present invention belongs to the field of fault detection and relates to a method for realizing a dual-machine hot standby system based on FPGA fault detection, and specifically relates to the hardware composition, fault detection mode and high-availability realization of the dual-machine hot standby system. Background Art

[0002] Previous dual-machine hot standby technologies mostly used high-availability software to achieve dual-machine hot standby. High-availability software uses network heartbeat messages as a fault detection method. The master sends heartbeat messages to the slave at a fixed time period. If the slave fails to receive the master's heartbeat message within three consecutive heartbeat cycles, it determines that the master has a fault and performs a master-slave switch. The fault detection method based on network heartbeat messages takes a long time, usually in seconds. For some devices that require real-time master-slave switching, this method is difficult to meet high real-time indicators. In addition, high-availability software lacks an effective means of detecting hardware failures, especially hardware failures that occur during the initial power-on of the device. Summary of the Invention

[0003] (1) Technical issues to be resolved

[0004] The technical problem to be solved by the present invention is how to provide an implementation method of a dual-machine hot standby system based on FPGA fault detection, so as to solve the problems that the previous dual-machine hot standby technology is difficult to meet higher real-time indicators and cannot detect hardware faults that occur in the early stage of equipment power-on.

[0005] (2) Technical solution

[0006] In order to solve the above technical problems, the present invention proposes a method for implementing a dual-machine hot standby system based on FPGA fault detection. The hardware environment of the dual-machine hot standby system consists of two mainboards integrating FPGA modules and sensor chips, and the two mainboards serve as the master and slave respectively; the FPGA of each mainboard communicates with the sensor chip and CPU via IIC and PCIe respectively, and the FPGAs of the two mainboards communicate via the baseboard serial port; fault detection driver software, system monitoring software, and IP migration control software run on the CPU of each mainboard, and the system monitoring software and IP migration control software communicate with the FPGA via the fault detection driver software; wherein the FPGA module implements fault detection, heartbeat detection, and fault reporting functions, and fault detection is divided into hardware fault detection, software fault detection, and FPGA fault detection.

[0007] Furthermore, the sensor is a digital voltage, current, and temperature monitor, which realizes the collection of key hardware status of the motherboard. The key hardware status serves as the basis for motherboard fault judgment. The key hardware status includes RTC voltage, DDR voltage, CPU voltage, input voltage, bridge chip temperature, and CPU temperature; the collected data is sent to the FPGA module via IIC.

[0008] Furthermore, the hardware fault detection includes power-on timing detection and hardware critical state detection. The FPGA realizes the power-on timing detection by reading the "power normal" PG signal of the power chip. The detection of the hardware critical state is that the FPGA controls the sensor to collect the key hardware state of the mainboard, and compares the collected value with the rated value. If it exceeds the rated range, it is determined to be a hardware fault.

[0009] Furthermore, the software fault detection includes operating system running status detection. The operating system status detection is implemented through the interrupt mechanism, and the running status of the driver software indirectly reflects whether the operating system has crashed. The FPGA sends an interrupt signal to the CPU every 3ms. If the interrupt can be cleared in time by the driver software, it is considered that the operating system is running normally. If the interrupt cannot be cleared within 9ms, it is determined that the operating system is malfunctioning.

[0010] Furthermore, the software fault detection includes system resource usage status detection. The detection of system resource usage status depends on the system monitoring software. The system monitoring software monitors the usage of key processes, CPU and memory of the operating system, sends the key process status to the FPGA module every 300ms, and sends the CPU and memory usage status to the FPGA every 1s. Finally, the FPGA determines whether there is a key process abnormality, CPU or memory usage abnormality; if the host's FPGA operates normally, then after the mainboard hardware and software fault occurs, the host's FPGA module can detect the fault content and send a fault message to the slave FPGA module. After receiving the fault message, the slave determines that the host has a fault; if the host FPGA operates abnormally, the slave's FPGA can determine that the host FPGA has a fault through heartbeat detection between FPGAs.

[0011] Furthermore, the heartbeat detection between FPGAs is implemented as follows: the FPGAs of the host and slave send heartbeat messages to each other through the serial port every 1ms. If the host FPGA operates abnormally, it will not be able to send heartbeat messages on time. After the slave fails to receive heartbeat messages for 3 consecutive ms, it will determine that the host FPGA is faulty; after the slave FPGA module detects the host fault, it will temporarily store the fault information through a dedicated register and immediately generate an interrupt signal. The interrupt is processed by the driver software and finally transmitted to the IP migration control software.

[0012] Furthermore, the driver software implements the interrupt processing and the ability of application software to read and write FPGA modules. The interrupt processing content is divided into operating system operation status interrupt processing and fault reporting interrupt processing. The driver software will judge the source of the interrupt. If it is a 3ms interrupt signal generated by the FPGA, that is, the interrupt signal used to judge the operating status of the operating system, the interrupt will be cleared immediately. If it is a fault reporting interrupt, the fault reporting message will be sent to the application layer through the asynchronous notification mechanism, and handed over to the IP migration control software of the application layer for further processing.

[0013] Furthermore, the driver software maps the FPGA module as a PCI device, enabling the IP migration control software and system monitoring software to read and write the FPGA's data registers through the POSIX standard interface, thereby realizing the functions of sending the system's key process status, CPU, and memory usage to the FPGA and reading fault information from the FPGA.

[0014] Furthermore, the system monitoring software implements the monitoring of operating system resources. The software runs automatically as a daemon process after the operating system is started, periodically checks the system's key processes and CPU and memory usage, and sends the monitoring results to the FPGA through the driver software interface.

[0015] Furthermore, the IP migration control software automatically runs after the system is turned on and configures the external communication IP. The IP migration control software can receive the fault reporting message sent by the FPGA driver. After receiving the fault reporting message, the driver software reads the FPGA register, determines the fault content, and controls the external communication IP to migrate from the master to the slave. The external communication IP is unique. When the master is able to work, the IP migration control software on the master binds the external communication IP to the master network card. When the master fails, the IP migration control software of the slave binds the external communication IP to the slave network card. The master and slave run the same service application and communicate with other client devices in the local area network in a "request-response" manner. If the client device does not generate a request when the fault occurs, since the external communication IP address remains unchanged before and after the master-slave switch, after the slave takes over the master, the client device generates a new request and sends it to the slave, which responds to it. If the host fails in the process of responding to the request, the request cannot be responded normally. After determining that the response has timed out, the client device sends a request again, and this request will be processed by the slave.

[0016] (3) Beneficial effects

[0017] This invention proposes a method for implementing a dual-machine hot standby system based on FPGA fault detection. This dual-machine hot standby system can detect system operational or hardware failures within milliseconds and perform master-slave switching, making it suitable for applications requiring high-speed fault detection. By utilizing an FPGA module for fault detection, this method improves real-time fault detection and can detect not only the operating system status and key process status, but also hardware faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a system architecture diagram of the present invention;

[0019] Figure 2 For the embodiment of the present invention;

[0020] Figure 3 Execute a flow chart for host failure detection;

[0021] Figure 4 This is the execution flow chart of the slave after discovering the master fault. DETAILED DESCRIPTION

[0022] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0023] The present invention provides a method for implementing a dual-machine hot standby system based on FPGA fault detection. The hardware environment of this dual-machine hot standby system consists of two mainboards, each integrating an FPGA module and a sensor chip, serving as the master and slave machines, respectively. Each mainboard runs fault detection driver software, system monitoring software, and IP migration control software. This dual-machine hot standby system can detect system operational or hardware faults within milliseconds and perform master-slave switching, making it suitable for applications requiring high-speed fault detection.

[0024] In response to the shortcomings and problems of the existing technology, the present invention provides a method for implementing a dual-machine hot standby system based on FPGA fault detection. Fault detection is performed through the FPGA module, which improves the real-time performance of fault detection. It can not only detect the operating system running status and key process status, but also detect hardware faults.

[0025] The dual-machine hot standby system hardware environment consists of two mainboards with integrated FPGA modules and sensor chips, serving as the master and slave devices, respectively. Each mainboard's FPGA communicates with the sensor chip and CPU via IIC and PCIe, respectively, and the FPGAs of both mainboards communicate via the baseboard serial port. Each mainboard's CPU runs fault detection driver software, system monitoring software, and IP migration control software, which communicate with the FPGA via the fault detection driver software.

[0026] The FPGA module realizes the functions of fault detection, heartbeat detection and fault reporting.

[0027] Fault detection is divided into hardware fault detection, software fault detection, and FPGA fault detection.

[0028] Hardware fault detection includes power-on sequence detection and hardware critical status detection. The FPGA can detect the power-on sequence by reading the "power good" PG signal from the power chip. Hardware critical status detection involves the FPGA controlling sensors to collect key hardware status of the motherboard, such as voltage, current, and temperature, and comparing the collected values with the rated values. If they exceed the rated range, a hardware fault is determined.

[0029] Software fault detection is divided into operating system running status detection and system resource usage status detection. Operating system status detection is implemented through the interrupt mechanism, and the running status of the driver software indirectly reflects whether the operating system has crashed. The FPGA sends an interrupt signal to the CPU every 3ms. If the interrupt can be cleared in time by the driver software, the operating system is considered to be running normally. If the interrupt cannot be cleared within 9ms, the operating system is judged to be faulty. The detection of system resource usage depends on the system monitoring software. The system monitoring software monitors the usage of key processes, CPU and memory of the operating system, sends key process status to the FPGA module every 300ms, and sends CPU and memory usage information to the FPGA every 1s. Finally, the FPGA determines whether there are key process anomalies, CPU or memory usage anomalies; if the host FPGA is running normally, then after the mainboard hardware and software failure occurs, the host FPGA module can detect the fault content and send a fault message to the slave FPGA module. After receiving the fault message, the slave determines that the host has a fault; if the host FPGA is running abnormally, the slave FPGA can determine that the host FPGA has a fault through heartbeat detection between FPGAs.

[0030] Heartbeat detection between FPGAs is implemented by having the master and slave FPGAs exchange heartbeat messages every 1ms via the serial port. If the master FPGA is operating abnormally, it will not send heartbeat messages on time. If the slave fails to receive heartbeat messages for 3 consecutive ms, it will determine that the master FPGA has failed. Upon detecting a master failure, the slave FPGA module temporarily stores the failure information in a dedicated register and immediately generates an interrupt signal. This interrupt is processed by the driver software and ultimately transmitted to the IP migration control software.

[0031] The sensor is a digital voltage, current, and temperature monitor that collects key motherboard hardware status information. This information serves as the basis for motherboard fault diagnosis. Key hardware status includes RTC voltage, DDR voltage, CPU voltage, input voltage, bridge chip temperature, and CPU temperature. The collected data is transmitted to the FPGA module via the IIC.

[0032] The driver software implements interrupt processing and the ability for application software to read and write to the FPGA module. Interrupt processing is divided into operating system operation status interrupt processing and fault reporting interrupt processing. The driver software determines the source of the interrupt. If it is a 3ms interrupt signal generated by the FPGA, which is used to determine the operating system operation status, the interrupt is immediately cleared. If it is a fault reporting interrupt, the fault report message is sent to the application layer through an asynchronous notification mechanism and handed over to the application layer's IP migration control software for further processing. The driver software maps the FPGA module as a PCI device, enabling the IP migration control software and system monitoring software to read and write the FPGA's data registers through the POSIX standard interface, thereby enabling the system to send key process status, CPU, and memory usage to the FPGA and read fault information from the FPGA.

[0033] The system monitoring software monitors operating system resources. It runs automatically as a daemon process after the operating system is started, periodically checks key system processes and CPU and memory usage, and sends the monitoring results to the FPGA through the driver software interface.

[0034] The IP migration control software automatically runs after the system is turned on and configures the external communication IP. The IP migration control software can receive the fault report message sent by the FPGA driver. After receiving the fault report message, the driver software reads the FPGA register, determines the fault content, and controls the external communication IP to migrate from the master to the slave. The external communication IP is unique. When the master is working, the IP migration control software on the master binds the external communication IP to the master network card; when the master fails, the IP migration control software on the slave binds the external communication IP to the slave network card. The master and slave run the same service application and communicate with other client devices in the local area network in a "request-response" manner. If the client device does not generate a request when the fault occurs, since the external communication IP address remains unchanged before and after the master-slave switch, after the slave takes over the master, the client device generates a new request and sends it to the slave, which responds to it. If the master fails in the process of responding to the request, the request cannot be responded to normally. After determining that the response has timed out, the client device sends a request again, and this request will be processed by the slave. Since the time for fault detection and IP migration binding is in milliseconds, the client timeout period can also be shortened to the same level, thereby reducing the time for fault recovery.

[0035] The present invention is described in detail below through a set of deployed embodiments to further illustrate the present invention, but it is not intended to limit the present invention.

[0036] like Figure 2 FIG. 1 is a schematic diagram of an embodiment of the present invention.

[0037] Specifically, in this embodiment, the two main boards in the chassis form a master-slave relationship, the FPGA modules on the two main boards are interconnected through the baseboard serial port, and both main boards can be connected to the switch through the chassis to communicate with other devices in the local area network.

[0038] After the chassis is powered on, the FPGA module first checks the power-on sequence and key hardware status. If the timing or key hardware status is abnormal, the host computer will not be able to boot successfully. The host FPGA module immediately notifies the slave FPGA module of the host failure through a heartbeat message. After the slave computer boots normally, the IP migration control software first reads the register data in the FPGA that stores the failure information. Upon detecting that the host computer failed to boot, it immediately binds the external communication IP address to the slave computer.

[0039] If the host is powered on normally, the host's IP migration control software will bind the external communication IP to the host. During normal operation of the host, the application on the host communicates with other devices in the LAN through the external communication IP. The slave will start the same application as the host, but because the external communication is bound to the host network card, it cannot communicate and is in standby mode. The FPGA module collaborates with the driver software and system monitoring software to monitor the motherboard's hardware and software faults, and ultimately the FPGA determines whether a fault has occurred. Fault types are divided into hardware faults, system operation faults, abnormal resource usage, system critical process faults, and FPGA faults. The fault judgment conditions are as follows:

[0040] 1. The critical hardware status exceeds the rated range, and the motherboard hardware is determined to be faulty;

[0041] 2. Failure to clear the FPGA's 3ms interrupt within 9ms leads to a system failure.

[0042] 3. The FPGA compares the CPU and memory usage collected by the system monitoring software with normal thresholds. If the CPU and memory usage exceed the normal threshold for 10 consecutive seconds, it determines that resource usage is abnormal.

[0043] 4. FPGA determines the status of the key process collected by the system monitoring software. If the key process is abnormal, it is determined that the system key process is faulty.

[0044] 5. If the slave FPGA module fails to receive the heartbeat message from the master FPGA module within 3ms, the master FPGA is deemed faulty.

[0045] Except for resource usage anomalies that require a long period of accumulation for judgment, other fault phenomena can be detected in milliseconds.

[0046] If the fault type is non-FPGA fault, the host's FPGA module will generate a fault reporting interrupt after discovering the fault and send a fault heartbeat message at the same time. The fault reporting interrupt will be processed by the host's driver software and generate a fault reporting message, which will be notified to the host's IP migration control software. To prevent IP conflicts, the host's IP migration control software will complete the unbinding of the external communication IP and the host's network card after receiving the fault reporting message; after the slave's FPGA module discovers the host fault through the fault message, it will generate a fault reporting interrupt. The slave's IP migration control software will execute the binding of the external communication IP and the slave's network card, and send an ARP broadcast message to update the ARP cache table of each device in the LAN.

[0047] If the fault type is a host FPGA module failure, the slave machine will be determined to be faulty after it fails to receive the 1ms heartbeat message generated by the host for 3 consecutive ms. The slave machine's IP migration control software will bind the external communication IP to the slave machine's network card and send an ARP broadcast message to update the ARP cache table of each device in the LAN. New request messages generated by client devices in the LAN will be sent to the slave machine's network card. At the same time, after receiving the slave machine's ARP broadcast message, the host machine's IP migration control software will immediately unbind the host machine's network card from the external communication IP and will not receive client request messages.

[0048] Since the master and slave machines run the same application, the IP addresses remain unchanged before and after IP migration, so the slave machine can take over the work of the master machine and communicate with other devices in the LAN.

[0049] The dual-machine hot standby system of the present invention can detect system operation failure or hardware failure and execute master-slave switching within milliseconds, and is suitable for application scenarios with high requirements for real-time fault detection.

[0050] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing a dual-machine hot standby system based on FPGA fault detection, characterized in that: The hardware environment of the dual-machine hot standby system consists of two mainboards integrated with FPGA modules and sensor chips, with the two mainboards serving as the master and slave, respectively. The FPGA of each mainboard communicates with the sensor chip and CPU via IIC and PCIe, respectively, and the FPGAs of the two mainboards communicate via the baseboard serial port. Fault detection driver software, system monitoring software, and IP migration control software run on the CPU of each mainboard, and the system monitoring software and IP migration control software communicate with the FPGA via the fault detection driver software. The FPGA module implements fault detection, heartbeat detection, and fault reporting functions, and fault detection is divided into hardware fault detection, software fault detection, and FPGA fault detection.

2. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1, wherein: The sensor is a digital voltage, current, and temperature monitor that collects the key hardware status of the motherboard. The key hardware status serves as the basis for motherboard fault diagnosis. The key hardware status includes RTC voltage, DDR voltage, CPU voltage, input voltage, bridge chip temperature, and CPU temperature. The collected data is sent to the FPGA module via IIC.

3. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The hardware fault detection includes power-on timing detection and hardware key status detection. The FPGA realizes power-on timing detection by reading the "power good" PG signal of the power chip. The hardware key status detection is that the FPGA controls the sensor to collect the key hardware status of the mainboard and compares the collected value with the rated value. If it exceeds the rated range, it is determined to be a hardware fault.

4. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The software fault detection includes operating system running status detection. The operating system status detection is implemented through the interrupt mechanism, and the running status of the driver software indirectly reflects whether the operating system has crashed. The FPGA sends an interrupt signal to the CPU every 3ms. If the interrupt can be cleared in time by the driver software, the operating system is considered to be running normally. If the interrupt cannot be cleared within 9ms, the operating system is determined to be faulty.

5. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The software fault detection includes system resource usage detection. The detection of system resource usage depends on the system monitoring software. The system monitoring software monitors the usage of key processes, CPU and memory of the operating system. The system monitoring software sends the key process status to the FPGA module every 300ms and sends the CPU and memory usage information to the FPGA every 1s. The FPGA finally determines whether there is any abnormality in key processes, CPU or memory usage. If the host's FPGA is operating normally, then after a mainboard hardware or software fault occurs, the host's FPGA module can detect the fault content and send a fault message to the slave's FPGA module. After receiving the fault message, the slave determines that the host has a fault; if the host's FPGA is operating abnormally, the slave's FPGA can determine that the host's FPGA has a fault through heartbeat detection between FPGAs.

6. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 5, wherein: Heartbeat detection between FPGAs is implemented as follows: the FPGAs of the master and slave devices exchange heartbeat messages every 1ms via the serial port. If the master FPGA is operating abnormally, it will not be able to send heartbeat messages on time. If the slave device fails to receive heartbeat messages for 3 consecutive ms, it will determine that the master FPGA has failed. When the slave device FPGA module detects a master failure, it will temporarily store the failure information in a dedicated register and immediately generate an interrupt signal. This interrupt is processed by the driver software and ultimately transmitted to the IP migration control software.

7. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The driver software implements interrupt processing and the ability of application software to read and write FPGA modules. The interrupt processing content is divided into operating system operation status interrupt processing and fault reporting interrupt processing. The driver software will determine the source of the interrupt. If it is a 3ms interrupt signal generated by the FPGA, that is, the interrupt signal used to determine the operating system operation status, the interrupt will be cleared immediately. If it is a fault reporting interrupt, the fault reporting message will be sent to the application layer through the asynchronous notification mechanism, and handed over to the IP migration control software of the application layer for further processing.

8. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 7, wherein: The driver software maps the FPGA module as a PCI device, enabling the IP migration control software and system monitoring software to read and write the FPGA's data registers through the POSIX standard interface, thereby enabling the system's key process status, CPU, and memory usage to be sent to the FPGA and fault information to be read from the FPGA.

9. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The system monitoring software monitors operating system resources. It runs automatically as a daemon process after the operating system is started, periodically checks key system processes and CPU and memory usage, and sends the monitoring results to the FPGA through the driver software interface.

10. The method for implementing a dual-machine hot standby system based on FPGA fault detection according to claim 1 or 2, characterized in that: The IP migration control software automatically runs after the system is powered on, configures the external communication IP, and can receive fault report messages sent by the FPGA driver. After receiving the fault report message, the driver software reads the FPGA register, determines the fault content, and controls the migration of the external communication IP from the master to the slave. The external communication IP is unique. When the master is working, the IP migration control software on the master binds the external communication IP to the master network card. When the master fails, the IP migration control software on the slave binds the external communication IP to the slave network card. The master and slave machines run the same service application and communicate with other client devices on the local area network in a "request-response" manner. If the client device does not generate a request when the fault occurs, since the external communication IP address remains unchanged before and after the master-slave switch, after the slave takes over as the master, the client device generates a new request and sends it to the slave, which responds. If the master fails during the request response process, the request cannot be responded to normally. The client device sends a request again after determining that the response has timed out, and this request will be processed by the slave.

Images