A fault recovery method based on smart network interface cards

Abstract

The invention discloses a fault recovery method based on an intelligent network card, and the recovery method comprises the following steps: a first step, triggering a timeout reset unit, a second step, updating a status register, a third step, restarting a server, a fourth step, continuously detecting a status of a status register, and a fifth step, completing the restart; the invention utilizes the design of a timeout reset unit, an FPGA, a status register and a detection unit, and expands a status register in virtio so that the status register represents the status of the entire network card, and a system on the network card takes over part of the state machine change logic of the status register of the virtio device, so that the server can sense a problem with a backend device without modifying a server driver, trigger a driver reload, and keep synchronization with the backend fault recovery, so that such fault recovery avoids restarting the server and reduces the fault recovery time.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a fault recovery method based on an intelligent network card. Background Art

[0002] In data centers, smart network cards are gradually being used on a large scale. Smart network cards present network and storage devices to server hosts in the form of PCIE cards, while offloading VPC network and cloud storage functions that were previously completed by server hosts to the network cards.

[0003] Since the smart network card carries a large number of functions and the business is complex, various types of failures are inevitable. However, in bare metal products, the smart network card and the server are only connected by the PCIE channel, and only standard virtio devices are often implemented on the PCIE channel. Therefore, when the smart network card fails, there is a lack of necessary fault-tolerant mechanism between the network card and the server to ensure that the system can detect the problem and restore the business. Summary of the invention

[0004] The purpose of the present invention is to provide a fault recovery method based on an intelligent network card to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a fault recovery method based on an intelligent network card, the recovery method comprising the following steps:

[0006] The first step is to trigger the timeout reset unit. When the smart network card fails, the timeout reset unit will be triggered to restart the FPGA and CPU.

[0007] The second step is to update the status register. The FPGA sends a signal to the virtio status register to change the status register to NEED_RESET state.

[0008] Step 3: The server is restarted, and the status register changes to trigger the server driver to reload.

[0009] The fourth step is to continuously detect the status of the status register. After the server is restarted, the status of the status register is continuously detected.

[0010] Step 5. Restart is completed. When the FPGA and CPU of the smart network card are restarted successfully, the write reset unit is performed to release the trigger of the timeout reset unit, so that the status register becomes available. The server can continue to load normally and complete the fault recovery.

[0011] Preferably, a CPLD is arranged inside the smart network card, the timeout reset unit is arranged inside the CPLD, the FPGA and the CPU are arranged inside the smart network card, the virtio is arranged inside the smart network card, and the virtio is used to connect the smart network card and the server.

[0012] Preferably, the timeout reset unit includes an input module, a counting module and an output module;

[0013] The input module is used to receive information transmitted by the write reset unit;

[0014] The counting module is used to calculate and determine whether the time exceeds the threshold;

[0015] The output module is used to output a reset signal to the FPGA.

[0016] Preferably, a trigger logic is provided inside the FPGA, and the trigger logic is used to output a signal to a status register after receiving a reset signal.

[0017] Preferably, a detection unit is disposed inside the server, and the detection unit includes a detection module, a control module, a cleaning module and a connection module.

[0018] Preferably, the connection module is used to connect the server and virtio;

[0019] The detection module is used to detect the state of virtio transmitted through the connection module;

[0020] The control module is used to control the restart of the server driver;

[0021] The cleaning module is used to clear the status register of virtio.

[0022] Preferably, the input and output ends of the connection module are electrically connected to the output end of the status register and the input end of the detection module respectively, the output end of the detection module is electrically connected to the input end of the control module, the output end of the control module is electrically connected to the input end of the cleaning module and the input end of the server driver, and the output end of the cleaning module is electrically connected to the input end of the status register.

[0023] Preferably, the status of the status register includes a NEED_RESET status and a device available status.

[0024] Preferably, the state of the status register represents the state of the entire smart network card.

[0025] Technical effects and advantages of the present invention:

[0026] The present invention utilizes the design of a timeout reset unit, an FPGA, a status register and a detection unit, and through the triggering logic of the timeout reset unit and the FPGA, the device can complete fault monitoring and emergency operations under a fault state. By expanding the status register in virtio, the status register represents the state of the entire network card. The system on the network card takes over part of the state machine change logic of the status register of the virtio device, so that the server can sense that a problem occurs in the back-end device without modifying the server driver, trigger the driver to reload, and keep synchronization with the fault recovery of the back-end. Such fault recovery avoids restarting the server and reduces the fault recovery time. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a schematic diagram of the implementation process of the recovery method of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention provides Figure 1 A fault recovery method based on an intelligent network card is shown, and the recovery method includes the following steps:

[0030] The first step is to trigger the timeout reset unit. When the smart network card fails, the timeout reset unit will be triggered to restart the FPGA and CPU.

[0031] Furthermore, the server restart will not be triggered at this time. When the FPGA or CPU fails, the write reset unit will not work, resulting in a timeout of the write reset unit. The input module receives the information and transmits it to the counting module. The counting module compares the transmitted information to determine whether the value exceeds the threshold. If it exceeds the threshold, it indicates a timeout. Then a signal is sent to the output module. The output module sends a timeout signal to the FPGA, triggering the trigger logic of the FPGA.

[0032] The second step is to update the status register. The FPGA sends a signal to the virtio status register to change the status register to NEED_RESET state.

[0033] Step 3: The server is restarted, and the status register changes to trigger the server driver to reload.

[0034] Furthermore, at this time, the connection module of the server transmits the current state of the status register: NEED_RESET state, to the detection module. After detection, the detection module sends it to the control module. The control module interrupts the server driver and restarts the server driver. At the same time, the control module transmits the signal to the cleaning module. The cleaning module will clear the virtio status register. During this period, the connection module continues to transmit the signal of the status register to the detection module to continuously detect the status of the status register.

[0035] The fourth step is to continuously detect the status of the status register. After the server is restarted, the status of the status register is continuously detected.

[0036] Furthermore, during a restart, the status register is not available.

[0037] Step 5. Restart is completed. When the FPGA and CPU of the smart network card are restarted successfully, the write reset unit is performed to release the trigger of the timeout reset unit, so that the status register becomes available. The server can continue to load normally and complete the fault recovery.

[0038] Furthermore, only when the smart network card is restarted, the FPGA and CPU are restarted, and the drive write reset unit is running, can the timeout period be adjusted to prevent the timeout reset unit from timing out, thereby preventing the FPGA trigger logic from sending a signal to the status register, causing the status of the status register to change to the device available state, thereby completing fault recovery.

[0039] Specifically, a CPLD is arranged inside the smart network card, a timeout reset unit is arranged inside the CPLD, an FPGA and a CPU are arranged inside the smart network card, virtio is arranged inside the smart network card, virtio is used to connect the smart network card and the server, the timeout reset unit includes an input module, a counting module and an output module, the input module is used to receive information transmitted by the write reset unit, the counting module is used to calculate and judge whether the time exceeds the threshold, and the output module is used to output a reset signal to the FPGA.

[0040] Furthermore, the server and the smart network card are two independent systems. During the power-on process of the whole machine, the server and the network card start initialization respectively. CPLD is an integrated circuit specially used to realize the functions of digital logic circuits. It is highly programmable and configurable, suitable for realizing various complex logic functions, with the characteristics of small size, low power consumption and high speed. It can be flexibly configured through programming to meet the needs of different application scenarios. Virtio is the de facto io device standard in the cloud computing era. It is widely used in virtualization scenarios. After the smart network card is widely used, virtio is also presented as the standard device for the smart network card to the server. In this way, the smart network card can be seamlessly applied in mainstream operating systems, and can also be compatible with virtualization scenarios and bare metal scenarios at the same time. FPGA is a field programmable gate array. It is further developed on the basis of programmable devices such as PAL, GAL, and CPLD. The product of the smart network card, it appears as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices. The timeout reset unit is the hardware added to the smart network card. The timeout reset function can be realized through the timeout reset unit, that is, when the timeout time is reached, the timeout reset unit will send a reset signal to the FPGA and CPU to restart the FPGA and CPU, and then resume the service. The write reset unit is used to adjust the timeout time. During the operation of the FPGA and CPU, the write reset unit will be continuously run. The timeout time is adjusted by the write reset unit. As long as the FPGA and CPU are in normal working state, it can be guaranteed that the timeout reset unit will never time out. When the FPGA or CPU fails to function normally, the timeout reset unit will be triggered due to the lack of the write timeout unit, thereby helping the FPGA or CPU to recover from the failure.

[0041] Specifically, the FPGA is internally provided with a trigger logic, and the trigger logic is used to output a signal to the status register after receiving a reset signal. The server is internally provided with a detection unit, and the detection unit includes a detection module, a control module, a cleaning module and a connection module. The connection module is used to connect the server and virtio, the detection module is used to detect the status of virtio transmitted through the connection module, the control module is used to control the restart of the server driver, and the cleaning module is used to clear the status register of virtio. The input and output ends of the connection module are electrically connected to the output end of the status register and the input end of the detection module respectively, the output end of the detection module is electrically connected to the input end of the control module, the output end of the control module is electrically connected to the input end of the cleaning module and the input end of the server driver, and the output end of the cleaning module is electrically connected to the input end of the status register.

[0042] Furthermore, a trigger logic is retained inside the FPGA, which is not affected by the reset or other logics, to ensure normal operation at any time. This trigger logic is triggered and executed when a reset signal is received to update the status register of virtio. When a problem occurs in the smart network card, the timeout reset unit of the smart network card will initiate a reset operation to the FPGA. The trigger logic of the FPGA borrows the status register of the virtio device, writes NEED_RESET to the status register, and sends the change of the status register to the detection unit of the server to interrupt the server, thereby triggering the server driver to reload the driver. At the same time, the cleaning module will clear the device status register of virtio and wait for the status register to change to the device being available. In the process of restoring the service of the smart network card, the status register of virtio is always in the NEED_RESET state, which indicates that the device is unavailable. The detection module will always detect whether the status of the status register has changed through the connection module. When the FPGA and CPU of the smart network card are reloaded, the status of the virtio status register is modified to the device being available, so that the driver loading is continued after the detection module detects it, and the traffic of the service flow is restored.

[0043] Specifically, the status of the status register includes the NEED_RESET status and the device available status, and the status of the status register represents the status of the entire smart network card.

[0044] Furthermore, virtio is a device specification that has a status register set internally. The status register is used to indicate the status of the virtio device. The NEED_RESET status is the device notifying the server driver that a problem has occurred with the virtio device and that the driver needs to reinitialize the virtio device. At the same time, the device-level status register function is expanded so that the status register function not only represents the status of the virtio device, but also represents the status of the entire smart network card. By using the virtio status register, the initialization timing synchronization problem in the smart network card application scenario is solved. The smart network card informs the server through the status register that initialization is complete, thereby synchronizing the initialization timing of the server and the smart network card.

[0045] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A fault recovery method based on an intelligent network card, characterized in that: The recovery method includes the following steps: The first step is to trigger the timeout reset unit. When the smart network card fails, the timeout reset unit will be triggered to restart the FPGA and CPU. At this time, the server restart will not be triggered. When the FPGA or CPU fails, the write reset unit will not work, resulting in a timeout in the write reset unit. The input module receives the information and transmits it to the counting module. The counting module compares the transmitted information to determine whether the value exceeds the threshold. If it exceeds the threshold, it means a timeout occurs. Then a signal is sent to the output module. The output module sends a timeout signal to the FPGA, triggering the trigger logic of the FPGA. The second step is to update the status register. The FPGA sends a signal to the virtio status register to change the status register to NEED_RESET state. Step 3: The server is restarted, and the status register changes to trigger the server driver to reload. The server's connection module transmits the current state of the status register: NEED_RESET state, to the detection module. After detection, the detection module sends it to the control module. The control module interrupts the server driver and restarts the server driver. At the same time, the control module transmits the signal to the cleanup module. The cleanup module clears the virtio status register. During this period, the connection module continues to transmit the status register signal to the detection module to continuously detect the status of the status register. The fourth step is to continuously detect the status of the status register. After the server restarts, the status of the status register is continuously detected. During the restart period, the status register is always unavailable. Step 5: After the restart is completed, when the FPGA and CPU of the smart network card are successfully restarted, the write reset unit is performed to release the trigger of the timeout reset unit, so that the status register becomes available, and the server can continue to load normally, completing the fault recovery; Only when the smart network card is restarted, the FPGA and CPU are running again, and the drive write reset unit is running, can the timeout period be adjusted to prevent the timeout reset unit from timing out, thereby preventing the FPGA trigger logic from sending a signal to the status register, so that the status of the status register is changed to the device available state, thereby completing the fault recovery; Specifically, a CPLD is arranged inside the smart network card, a timeout reset unit is arranged inside the CPLD, an FPGA and a CPU are arranged inside the smart network card, virtio is arranged inside the smart network card, virtio is used to connect the smart network card and the server, the timeout reset unit includes an input module, a counting module and an output module, the input module is used to receive information transmitted by the write reset unit, the counting module is used to calculate and judge whether the time exceeds the threshold, and the output module is used to output a reset signal to the FPGA.

2. A fault recovery method based on smart network card according to claim 1, characterized in that: The smart network card is provided with a CPLD inside, the timeout reset unit is provided in the CPLD, the FPGA and the CPU are both provided inside the smart network card, the virtio is provided inside the smart network card, and the virtio is used to connect the smart network card and the server.

3. The fault recovery method based on smart network card according to claim 1, characterized in that: The timeout reset unit includes an input module, a counting module and an output module; The input module is used to receive information transmitted by the write reset unit; The counting module is used to calculate and determine whether the time exceeds the threshold; The output module is used to output a reset signal to the FPGA.

4. A fault recovery method based on smart network card according to claim 3, characterized in that: The FPGA is internally provided with a trigger logic, and the trigger logic is used to output a signal to the status register after receiving a reset signal.

5. The fault recovery method based on smart network card according to claim 1, characterized in that: A detection unit is arranged inside the server, and the detection unit comprises a detection module, a control module, a cleaning module and a connection module.

6. A fault recovery method based on smart network card according to claim 5, characterized in that: The connection module is used to connect the server and virtio; The detection module is used to detect the state of virtio transmitted through the connection module; The control module is used to control the restart of the server driver; The cleaning module is used to clear the status register of virtio.

7. A fault recovery method based on smart network card according to claim 6, characterized in that: The input and output ends of the connection module are electrically connected to the output end of the status register and the input end of the detection module respectively, the output end of the detection module is electrically connected to the input end of the control module, the output end of the control module is electrically connected to the input end of the cleaning module and the input end of the server driver, and the output end of the cleaning module is electrically connected to the input end of the status register.

8. The fault recovery method based on smart network card according to claim 1, characterized in that: The status of the status register includes a NEED_RESET status and a device available status.

9. A fault recovery method based on smart network card according to claim 8, characterized in that: The state of the status register represents the state of the entire smart network card.

Images