Troubleshooting method, device and server

Abstract

The present application provides a fault handling method, device, and server. The method includes: the baseboard management controller BMC receives the first fault information, and when it is determined that the fault will interfere with the normal operation of the server according to the first fault information, the BMC actively obtains the faulty target The second fault information corresponding to the PCIe device, the second fault information includes the fault information inside the target PCIe device and the modules connected to the target PCIe device, and then the BMC locates the server where the target PCIe device is located according to the first fault information and the second fault information failure. When the BMC determines that the fault will interfere with the normal operation of the server based on the initially obtained fault information, the BMC can further collect the fault information of the target PCIe device, so that the collection of fault information is more comprehensive and the fault location is more accurate. You can use the out-of-band BMC to collect fault information and accurately locate faults.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a fault handling method, device and server. Background technique [0002] In AI computing in the field of artificial intelligence (AI), the demand for computing power of servers continues to increase, and at the same time, the requirements for reliability, availability and serviceability (RAS) of servers are also getting higher and higher. . [0003] In order to provide sufficient computing power, a server that integrates multiple AI chips (such as image processing unit (graphics processing unit, GPU), neural processing unit (neural processing unit, NPU), and tensor processing unit (tensor processing unit, TPU), etc.) Came into being. Through the interconnection of multiple AI chips to form a multi-P system, it provides stronger computing power for AI computing. In a server including multiple AI chips, from the perspective of the host management system, the AI ...

AI Technical Summary

Problems solved by technology

In a server including multiple AI chips, from the perspective of the host management system, the AI ​​chip is a peripheral component interconnect express (PCIe) device. Fault location relies on advanced error reporting (advanced error reporting, AER). Fault location is achieved through in-band logs such as PCIe AER Code. However, when locating based on PCIe AERCode, it can only be located when it is connected to the central processing unit (CPU). The fault location granularity of AI chip is relatively coarse, so it cannot provide more fault data to support fault repair, and when the host management system fails, the server cannot collect fault information and fault location

Images