Diagnosis method and computer program product

Abstract

The present application discloses a diagnosis method and a computer program product. The diagnosis method is applied to a server; the server comprises a central processing unit (CPU), a baseboard management controller (BMC) and a neural processing unit (NPU); and the BMC is connected to both the CPU and the NPU. The method comprises: the CPU parsing a BMC log to obtain a pre-diagnosis result; when the pre-diagnosis result indicates that the NPU is faulty, determining a test script on the basis of the pre-diagnosis result; testing the NPU on the basis of the test script, and reading a BMC log generated in the test process; and when the BMC log generated in the test process indicates that the BMC raises an alarm, generating a diagnosis result on the basis of an in-band log generated in the test process and the BMC log that triggers the BMC to raise the alarm. The reliance on an in-band log of the OS of the customer is eliminated, thereby resolving the difficulty of fault diagnosis caused by the inability to obtain logs due to customer data security concerns.

Description

Diagnostic methods and computer program products

[0001] This application claims priority to Chinese Patent No. 202410984695.5, filed on July 19, 2024, entitled "Diagnostic Method and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a diagnostic method and a computer program product. Background Technology

[0003] NPU (Neural Processing Unit) servers are designed specifically for computationally intensive tasks such as deep learning and neural networks. These servers play a crucial role in customer model training scenarios, efficiently processing large amounts of data and accelerating the model training process. However, in practical use, NPU servers may encounter issues such as speed reduction and GPU failures, which directly impact the efficiency and quality of model training.

[0004] In related technologies, problem localization relies on collecting and analyzing in-band logs. In-band logs typically refer to logs generated within the server, such as system logs, application logs, and hardware logs. These logs contain detailed information about server operation and are crucial for analyzing and locating problems. However, because this data may contain sensitive information (such as user data and model parameters), customers may be unwilling to provide in-band logs due to data security concerns, thus hindering problem localization. Summary of the Invention

[0005] This application provides a diagnostic method and computer program product for diagnosing NPU faults without relying on the in-band logs of the client OS system.

[0006] To address the above problems, the technical solutions provided in this application are as follows:

[0007] The first aspect of this application provides a diagnostic method applied to a server, the server including a central processing unit (CPU), a motherboard management controller (BMC), and a neural network processor (NPU), wherein the BMC is connected to both the CPU and the NPU, including:

[0008] The CPU parses the BMC logs to obtain pre-diagnostic results;

[0009] When the pre-diagnosis result indicates a fault in the NPU, a test script is determined based on the pre-diagnosis result, and the test script is used to test the NPU.

[0010] The NPU is tested based on the test script, and the BMC logs generated during the test are read.

[0011] When the BMC logs generated during the test indicate the BMC alarm, a diagnostic result is generated based on the in-band logs generated during the test and the BMC logs that triggered the BMC alarm.

[0012] The method provided in this application embodiment, when the pre-diagnosis result indicates an NPU fault, the CPU calls the corresponding test script based on the pre-diagnosis result to perform targeted testing on the NPU. During the test, the CPU monitors the motherboard BMC log. If a BMC alarm occurs during the test, the CPU can parse the diagnostic result based on the motherboard BMC log that triggered the alarm and the in-band log. The in-band log provides the NPU's operating status and software-level error information during the test, while the BMC log reveals abnormal hardware-level states. The final diagnostic result can be obtained based on these two types of logs without accessing or relying on the client OS's in-band log. This means that even if the client restricts access to the client operating system for data security reasons, this application embodiment can still achieve NPU fault diagnosis, solving the problem of fault diagnosis caused by the inability to obtain logs due to client data security considerations.

[0013] In one possible implementation, after determining the test script based on the pre-diagnosis results, the method further includes: loading the NPU driver, which is used to establish a communication connection between the CPU and the NPU.

[0014] The process of testing the NPU based on the test script and reading the BMC logs generated during the test includes:

[0015] When the NPU driver is loaded, the NPU is tested based on the test script, and the BMC logs generated during the test are read.

[0016] In one possible implementation, the BMC log includes the sensor event log (SEL) of the BMC and the sensor event log (SEL) of the NPU's BMC. The pre-diagnostic result includes the NPU fault type. The CPU parses the BMC log to obtain the pre-diagnostic result, which includes:

[0017] The SEL of the BMC is parsed;

[0018] When the SEL of the BMC includes the BMC alarm information of the NPU, the SEL of the BMC of the NPU is parsed based on the BMC alarm information of the NPU to obtain the NPU fault type corresponding to the BMC alarm information of the NPU.

[0019] In one possible implementation, the method further includes:

[0020] When the SEL of the BMC includes the BMC alarm information of the NPU, obtain the fault code in the SEL of the BMC of the NPU;

[0021] The obtained fault codes are compared with a preset fault code table to determine the NPU fault type corresponding to the fault code. The preset fault code table includes NPU fault types corresponding to each fault code.

[0022] In one possible implementation, determining the test script based on the pre-diagnosis results includes:

[0023] Based on the pre-diagnosis results, a test task list is determined, and the parameters of each test task in the test task list are assigned values ​​to obtain the test script corresponding to the pre-diagnosis results.

[0024] In one possible implementation, the step of testing the NPU based on the test script and reading the BMC logs generated during the test includes:

[0025] According to the order of the test tasks in the test script, the NPU is tested sequentially, and the BMC logs generated during the test are read until the BMC logs generated during the test indicate a BMC alarm.

[0026] In one possible implementation, when the BMC logs generated during the test characterize the BMC alarm, a diagnostic result is generated based on the in-band logs generated during the test and the BMC logs that triggered the BMC alarm, including:

[0027] When a BMC log generated during the test indicates a BMC alarm, the currently executing test task in the test script is terminated.

[0028] The NPU is tested sequentially according to the order of the test tasks in the test script until all the test tasks in the test script have been executed.

[0029] Diagnostic results are generated based on the in-band logs generated during the test and the BMC logs that triggered the BMC alarm.

[0030] In one possible implementation, before the CPU parses the BMC log to obtain the pre-diagnostic result, it further includes:

[0031] The CPU sends a first preset instruction to the BMC to request the BMC log, so that after receiving the first preset instruction, the BMC returns the BMC log to the CPU. The BMC log includes out-of-band information of the NPU's BMC sent to the BMC log when the NPU fails.

[0032] In one possible implementation, before the CPU parses the BMC log to obtain the pre-diagnostic result, it further includes:

[0033] The CPU sends a first preset instruction to the BMC to request the BMC log, so that after receiving the first preset instruction, the BMC sends a request to the NPU's BMC to obtain out-of-band information of the NPU's BMC, so that the NPU's BMC sends the out-of-band information to the BMC.

[0034] After the BMC obtains the NPU's out-of-band information returned by the NPU's BMC, the CPU receives the server BMC log, which contains the NPU's out-of-band information, returned by the BMC.

[0035] In one possible implementation, the method further includes:

[0036] Based on the built-in case library, a processing strategy corresponding to the diagnostic result is generated. The processing strategy includes a solution for the diagnostic result. The built-in case library includes multiple historical diagnostic results and processing strategies corresponding to each historical diagnostic result.

[0037] A second aspect of this application provides a computer program product that, when run on a computer, executes the diagnostic method described in the first aspect above. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of this embodiment, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 is a schematic diagram of the application scenario of the diagnostic system provided in the embodiment of this application;

[0040] Figure 2 is a schematic diagram of the diagnostic system framework provided in an embodiment of this application;

[0041] Figure 3 is a flowchart illustrating the first diagnostic method provided in this application embodiment;

[0042] Figure 4 is a flowchart illustrating the second diagnostic method provided in an embodiment of this application;

[0043] Figure 5 is a schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of this application;

[0044] Figure 6 is a schematic diagram illustrating the classification of parsing methods provided in the embodiments of this application;

[0045] Figure 7 is a schematic diagram of the NPU log provided in an embodiment of this application;

[0046] Figure 8 is a schematic diagram of the SEL file content provided in an embodiment of this application;

[0047] Figure 9 is a schematic diagram of the pre-diagnosis results provided in an embodiment of this application;

[0048] Figure 10 is a schematic diagram of the test script provided in an embodiment of this application;

[0049] Figure 11 is a schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of this application. Detailed Implementation

[0050] To enable those skilled in the art to better understand the embodiments of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0051] To facilitate understanding of the technical solutions provided in the embodiments of this application, the terms involved in the embodiments of this application will be explained below.

[0052] A Neural Processing Unit (NPU) is a processor or chip specifically designed for performing computations on artificial neural networks. It is widely used to accelerate artificial intelligence tasks, particularly deep learning and machine learning algorithms.

[0053] A BMC is a dedicated microcontroller used to monitor and manage various aspects of server hardware, including but not limited to temperature, power status, fan speed, and the health of the processor and memory. Each major hardware component, such as the motherboard and NPU, can have its own BMC for more granular management and monitoring.

[0054] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0055] This application provides a diagnostic method in which the CPU reads and parses the motherboard BMC log to obtain a pre-diagnostic result. When the pre-diagnostic result indicates an NPU fault, the CPU calls the corresponding test script to test the NPU based on the pre-diagnostic result. During the test, the CPU monitors the motherboard BMC log to capture the reproduction of NPU faults caused by the test. When the motherboard BMC alarms, the CPU obtains the motherboard BMC log that triggered the alarm, as well as the in-band log generated by the NPU during the test. The in-band log reflects the NPU's operating status and software-level error information during the test, while the motherboard BMC log that triggered the alarm reflects the hardware-level faults that occurred in the NPU during the test. This information records in detail the hardware and software states during the diagnostic process, as well as possible problems or abnormal situations. Therefore, this application can perform fault diagnosis without directly accessing or entering the customer's operating system, and the diagnostic result can be obtained based on the in-band and out-of-band logs generated during the diagnostic process, without relying on the in-band logs of the customer's OS system. This solves the problem of fault diagnosis caused by the inability to obtain logs due to customer data security considerations.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0057] The following example illustrates the diagnostic system provided in this application.

[0058] Referring to Figure 1, which is a schematic diagram of an application scenario for the diagnostic system provided in this embodiment, the server includes a CPU, a BMC, and an NPU. The BMC is connected to both the CPU and the NPU. This embodiment provides an externally connectable diagnostic device, which includes a diagnostic system. The diagnostic device can be a hard disk or a USB storage device. The diagnostic device has sufficient capacity to store the diagnostic system and all its required resources.

[0059] In practical applications, diagnostic devices containing diagnostic systems can be plugged into the corresponding interfaces of the server, such as SATA, SAS, and USB. After the diagnostic device is connected to the server, the CPU can load and run the diagnostic system within the device.

[0060] After the diagnostic system runs on the CPU, the CPU can be manually triggered to start the diagnostic method. When the CPU receives the instruction to trigger the diagnostic method, it runs the fault diagnosis module in the diagnostic system to acquire out-of-band information of the BMC and performs a preliminary diagnosis based on the BMC out-of-band information. If the preliminary diagnosis indicates an NPU fault, further diagnosis is performed, and the NPU is tested to reproduce the NPU fault and generate a processing strategy.

[0061] For details on the functions of each part of the diagnostic system, please refer to Figure 2, which is a schematic diagram of the diagnostic system framework provided in the embodiment of this application.

[0062] The fault pre-diagnosis module is used to parse the server motherboard BMC logs provided by the motherboard BMC to determine the location and type of output faults in problematic devices, providing targeted testing support for the fault diagnosis module. Parsing methods can include register parsing, SEL parsing, and FDM parsing.

[0063] The fault diagnosis module is used to perform hardware and software performance tests on the NPU based on the fault location and type determined by the fault pre-diagnosis module, monitor the test process, and output test results. The test types of the fault diagnosis module can include HBM media testing, TDP power consumption testing, ROCE network port testing, bandwidth testing, and EDP power consumption testing. Specifically, HBM media testing includes: Read: stress test performing read operations; Write: stress test performing write operations. Bandwidth testing includes: d2h: test of data transmission from the NPU to the CPU; h2d: test of data transmission from the CPU to the NPU device; d2d: data transmission test within the NPU; p2p: data transmission test within the CPU. Cache memory testing includes:

[0064] The log parsing module collects and parses log information generated during testing to produce diagnostic results and processing strategies. The log information generated during testing includes out-of-band BMC information that triggers motherboard BMC alarms, and in-band logs generated when the CPU runs the diagnostic system. The module can parse the in-band and out-of-band information separately to generate corresponding diagnostic strategies for each. Alternatively, it can combine the in-band and out-of-band information to generate a single diagnostic strategy.

[0065] The diagnostic method provided in the embodiments of this application is described below. Referring to Figure 3, Figure 3 is a flowchart illustrating the first diagnostic method provided in the embodiments of this application. The method includes:

[0066] S110. The CPU parses the BMC log to obtain the pre-diagnosis result.

[0067] The CPU reads and analyzes the BMC logs. The BMC is a component responsible for managing server hardware status (such as temperature, voltage, fan speed, etc.) and generating corresponding logs. By parsing the BMC logs, the CPU distinguishes between common hardware failures (such as hard drive, memory, CPU, expansion cards, etc.) and NPU-related failures, thus obtaining a preliminary diagnostic result.

[0068] S111. When the pre-diagnosis result indicates a fault in the NPU, a test script is determined based on the pre-diagnosis result, and the test script is used to test the NPU.

[0069] When the pre-diagnosis result indicates an NPU fault, the test script corresponding to the pre-diagnosis result is called to perform targeted tests on the NPU in order to reproduce the NPU fault.

[0070] S112. Test the NPU based on the test script and read the BMC logs generated during the test.

[0071] By monitoring the BMC logs generated during the testing process, we can ensure the accuracy and effectiveness of the tests.

[0072] S113. When the BMC log generated during the test indicates the BMC alarm, a diagnostic result is generated based on the in-band log generated during the test and the BMC log that triggered the BMC alarm.

[0073] If the BMC log indicates a BMC alarm during the test, the CPU generates a diagnostic result by comprehensively analyzing the in-band logs generated during the test and the BMC logs that triggered the BMC alarm.

[0074] Therefore, in this embodiment, when the pre-diagnosis result indicates an NPU fault, the CPU calls the corresponding test script based on the pre-diagnosis result to perform targeted testing on the NPU. During the test, the CPU monitors the motherboard BMC log. If a BMC alarm occurs during the test, the CPU can parse the diagnostic result based on the motherboard BMC log that triggered the alarm and the in-band log. The in-band log provides the NPU's operating status and software-level error information during the test, while the BMC log reveals abnormal hardware-level states. The final diagnostic result can be obtained based on these two types of logs without accessing or relying on the client OS's in-band log. This means that even if the client restricts access to the client operating system for data security reasons, this embodiment can still achieve NPU fault diagnosis, solving the problem of fault diagnosis caused by the inability to obtain logs due to client data security considerations. The diagnostic method executed by the diagnostic system is described below. Referring to Figure 4, Figure 4 is a schematic diagram of the second diagnostic method provided in this embodiment, which includes:

[0075] S101, the CPU reads the server motherboard BMC log through the first preset instruction.

[0076] The CPU running the diagnostic system sends a first preset command to the server motherboard BMC to request the server motherboard BMC logs. The motherboard BMC can acquire and integrate information from the BMCs of different components.

[0077] The server motherboard BMC log can be used to store important information about server hardware status, events, and potential failures. The server motherboard BMC can record NPU fault information to the server motherboard BMC log file and can forward the server motherboard BMC log file to the server's management system, such as the diagnostic system provided in this application embodiment. That is, the server motherboard BMC log includes not only out-of-band information from the server motherboard BMC but also out-of-band information from the NPU BMC. The motherboard BMC and the NPU BMC, by recording hardware status and anomalies respectively, combine to provide a comprehensive hardware health overview, allowing the diagnostic system to gain a more complete understanding of the NPU and motherboard's operating status.

[0078] In the first scenario, after receiving the first preset instruction, the server motherboard's BMC sends a request to the NPU's BMC to retrieve the NPU's out-of-band information, causing the NPU's BMC to send the out-of-band information to the server motherboard's BMC. After the server motherboard's BMC receives the NPU's out-of-band information, it returns a server motherboard BMC log containing this information to the CPU.

[0079] In another scenario, when the NPU's BMC detects a fault in the NPU, the NPU's BMC can transmit the NPU's out-of-band information to the server motherboard's BMC via a preset interface. The server motherboard's BMC will then record the received out-of-band information from the NPU's BMC, either directly or after processing, into its log. This log will then be returned to the CPU by the server motherboard's BMC upon receiving the first preset instruction from the CPU.

[0080] For example, during out-of-band information transmission from the NPU or motherboard, the out-of-band information can be packaged into data packets and transmitted using secure encryption protocols such as HTTPS, XML, or JSON to ensure transmission to the server motherboard BMC. The default interface can be Redfish. Redfish is a modern management interface that allows hardware status and configuration data to be remotely accessed and exchanged in a standardized manner, providing efficiency and security. The NPU's BMC can utilize this interface to send out-of-band information containing NPU-related fault details to the server motherboard BMC in real time. Furthermore, Redfish can use HTTPS-based communication to ensure data security during transmission.

[0081] After receiving out-of-band information from the NPU's BMC, the server motherboard's BMC can parse and convert this information into a user-readable format, displaying it on the web interface. This transformation of the NPU's out-of-band information into intuitive data allows maintenance personnel or IT administrators to visually assess the NPU's operational status through the web interface, improving transparency and responsiveness.

[0082] S102. The CPU analyzes the server motherboard BMC logs to determine the preliminary diagnostic results.

[0083] This refers to the fault pre-diagnosis module in the CPU operation diagnostic system. By analyzing the server motherboard BMC logs provided by the motherboard BMC, it distinguishes between ordinary hardware faults (such as hard drive, memory, CPU, and expansion cards) and NPU-related faults. When an NPU fault is determined, it parses the out-of-band information of the NPU's BMC in the server motherboard BMC logs to determine the pre-diagnosis result. This pre-diagnosis result is used to guide subsequent performance testing of the NPU.

[0084] Once the server motherboard BMC log indicates an NPU failure, the CPU running the diagnostic system will analyze the out-of-band information of the NPU's BMC in the server motherboard BMC log to obtain more specific problem details and corresponding diagnostic strategies. The NPU lower-level log refers to the log information generated by the underlying hardware or firmware during NPU task execution. This information can include the NPU's operating status, error codes, performance data, etc.

[0085] The workflow of the fault pre-diagnosis module can be seen in Figure 5. Figure 5 is a schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of this application, including:

[0086] First, the fault pre-diagnosis module obtains the server motherboard BMC logs. The server motherboard BMC logs include fault information provided by the server motherboard BMC, which may be fault information caused by server motherboard failure and / or fault information caused by NPU failure.

[0087] For example, the fault pre-diagnosis module can use a first preset command to collect server motherboard BMC logs with a single click. The collected server motherboard BMC logs are then preprocessed, which may involve compression, formatting, or other preprocessing tasks to facilitate subsequent parsing. The first preset command could be `ipmcget -d diaginfo`.

[0088] The preprocessed server motherboard BMC logs are then parsed, as shown in Figure 2. Parsing methods can include register parsing, SEL (System Event Log) parsing, and FDM (Frequency-division multiplexing) parsing. It should be noted that these three parsing methods are used to parse the server motherboard BMC logs during the pre-diagnosis process. These three methods can be performed simultaneously or sequentially.

[0089] The first method is register parsing. The health status and fault information of hardware devices (such as NPU, memory, hard drives, etc.) are often recorded in specific registers. These registers contain a series of binary bits, each representing a status or fault flag. The server motherboard BMC log includes register values. By reading these register values ​​and interpreting them according to a predetermined bit mapping table, it can be determined whether a fault has occurred and its specific type. During log parsing, the collected logs can be analyzed based on register-related information to obtain precise fault outputs related to the Power Supply Unit (PSU). If the preliminary diagnostic result only indicates a PSU fault, and other parsing methods indicate that the current NPU is not faulty, the current diagnostic process is terminated.

[0090] The second type is FDM, an intelligent fault management engine in servers. It monitors various server components and promptly reports and analyzes potential hardware failures. FDM works in conjunction with other systems to parse the server's motherboard BMC logs to determine the presence and type of faults. Specifically, the FDM engine analyzes the server's motherboard BMC logs, looking for any anomalies or error codes that may indicate hardware failures. Analysis includes, but is not limited to, CPU errors, memory errors, power supply problems, fan failures, and over-temperature. Based on the parsed error codes and anomalies, FDM can match them to a pre-defined fault mode database to identify specific fault types. These patterns can involve failures of individual components, such as the CPU or memory modules, or more complex system-level problems.

[0091] Therefore, by analyzing the FDM logs, the specific source of the fault can be quickly located, such as obtaining precise fault outputs related to Dual In-line Memory Modules (DIMMs), disks, and Redundant Array of Independent Disks (RAID). If the pre-diagnostic results only indicate that there is a fault in the DIMM, disk, or RAID, while other parsing methods indicate that there is no fault in the current NPU, the diagnosis is terminated.

[0092] The third method is SEL parsing. The SEL log is generated by the BMC and is used to record important information about the server hardware and its health status. When events such as hardware failure, abnormal temperature, power problems, or system restarts are detected, the BMC records these events in the SEL log. Each event has a unique fault code (SEL Record ID). The server motherboard BMC log includes the motherboard BMC's SEL file and the NPU's BMC's SEL file. In this embodiment, the server includes a motherboard and an NPU, each with its own BMC. The NPU's BMC and the motherboard BMC each have a SEL file. The motherboard's SEL file contains fault information related to memory and hard disk, while the NPU's BMC's SEL file contains NPU-related fault information. During the operation of the NPU's BMC, the two SEL files can communicate to transmit SEL alarms from the NPU's BMC to the motherboard BMC's SEL file. Therefore, the motherboard BMC's SEL file can also contain alarm information transmitted from the NPU.

[0093] In practical applications, some NPU alarm information is transmitted to the motherboard's BMC, but not all NPU alarm information is transmitted to the motherboard BMC. Therefore, to more accurately and comprehensively determine NPU alarm information, the NPU's SEL file can be further analyzed to determine the preliminary diagnostic results. Thus, after finding transmitted NPU alarm information in the motherboard BMC's SEL file, the NPU's SEL file is further parsed. During the parsing of the NPU's SEL file, preliminary diagnostics can be performed using at least one of two methods: the NPU's BMC alarm information or the NPU fault ID within the NPU's SEL file.

[0094] Referring to Figure 6, which is a schematic diagram of the parsing methods provided in this application embodiment, the first parsing method in Figure 6 determines the NPU fault number or fault phenomenon through the alarm information of the NPU's BMC. This parsing method can be applied to the following situations: card drop, bandwidth reduction, overheating, carrier board failure, and temperature acquisition failure. The above-mentioned fault problems may cause immediately visible and direct failures or performance degradation, affecting the stability and availability of the server. Therefore, when the interfaces between BMCs are relatively limited, only the fault information with greater impact from the NPU's BMC is allowed to be passed through to the SEL file of the motherboard's BMC. This information can be recorded by the motherboard BMC and the NPU's BMC as alarm information of the NPU's BMC and recorded in the SEL file of the NPU's BMC for quick response when querying faults. At the same time, the NPU's BMC can also continue to record sensor events during operation to the SEL file of the NPU's BMC, including the passed-through alarm information and more detailed event records. Therefore, when parsing the SEL file of the NPU's BMC, the pre-diagnostic results can be read based on the NPU alarm information recorded in the SEL file.

[0095] For example, when the CPU reads the motherboard BMC's SEL file, it contains NPU BMC alarm information passed through by the NPU BMC: NPU bandwidth reduction. To ensure the accuracy of the pre-diagnostic results, the CPU selectively reads the NPU BMC's SEL file based on the NPU bandwidth reduction. If the NPU BMC's SEL file contains NPU bandwidth reduction, and the current server only includes one NPU module, then the pre-diagnostic result includes NPU bandwidth reduction.

[0096] In one possible implementation, if the number of NPU modules on the current server is greater than one, to more accurately determine which module should be tested next, the CPU can further parse the systemcom.dat file of the NPU in the motherboard BMC log based on the read NPU BMC alarm information: NPU bandwidth reduction. Referring to Figure 7, which is a schematic diagram of the NPU log provided in this embodiment, this file contains NPU link establishment information, i.e., how many NPU cards were identified and what the bandwidth is. During the parsing process, the pre-diagnosis module uses LINKSTS as the keyword to perform orthogonal matching on the log file. As shown in Figure 7, it can identify that the link establishment status of NPU 10 is 0x2045, and conclude that the NPU 10 module has reduced bandwidth as a pre-diagnosis result. Subsequently, targeted testing and diagnosis of the NPU 10 module can be performed.

[0097] As shown in Figure 6, the second parsing method identifies the fault phenomenon through the NPU fault ID. Figure 6 also includes some situations where pre-diagnosis can be performed using the fault ID, i.e., querying the NPU fault ID recorded in the SEL file. This parsing method is applicable to the following situations: intermittent network port disconnection, ECC isolation, and L2 buffering, etc. These types of faults are more insidious; they may not immediately cause system crashes, but they will affect system reliability and data integrity. Given limited BMC interface resources, the above fault information is not transparently transmitted to the motherboard BMC's SEL file. Therefore, this parsing method can correspond to NPU fault types that are not transparently transmitted to the motherboard BMC SEL file.

[0098] Referring to Figure 8, which is a schematic diagram of the SEL file content provided in an embodiment of this application, the CPU can traverse the NPU's SEL file and attempt to obtain the fault IDs recorded in the SEL file. The fault IDs extracted in Figure 8 are, for example, 0X81078603. Fault IDs are typically associated with specific fault types. The pre-diagnosis module compares the fault ID 0X81078603 with a predefined fault code table. The fault code table lists all possible fault IDs and their corresponding fault descriptions and suggested handling strategies. By looking up the table, the pre-diagnosis module can determine the specific NPU fault type represented by fault ID 0X81078603, such as NPU memory or bandwidth abnormalities, or network port connection problems. After identifying the fault ID, the diagnostic system can not only locate the problem but also query specific fault information, as shown in the fault details in Figure 8: 1. The network port chip detected a fault in its own link status; 2. The network port chip detected a fault in the peer's link status. The fault type is: Network port function unavailable. The handling measures adopted for the current fault are: 1. Reporting the fault event to fault management; 2. Recording logs. Current fault level: critical.

[0099] Subsequently, the pre-diagnosis module can output the specific location of the fault and suggested handling strategies. Referring to Figure 9, which is a schematic diagram of the pre-diagnosis results provided in the embodiment of this application, the pre-diagnosis results include: fault ID 0X81078603. This pre-diagnosis result is used to guide maintenance personnel to take corresponding actions, such as hardware checks, firmware upgrades, or parameter adjustments.

[0100] The fault pre-diagnosis module can obtain the output fault location and / or fault type of the problematic device by parsing the server motherboard BMC log. Based on the above pre-diagnosis results, the fault pre-diagnosis module outputs a diagnosis strategy to enable the fault diagnosis module to perform hardware and software performance testing on the NPU.

[0101] S103. The CPU calls the test script corresponding to the pre-diagnosis result based on the pre-diagnosis result.

[0102] This step can be implemented by the fault diagnosis module in the CPU running diagnostic system. The following is an explanation of the fault diagnosis module in the diagnostic system.

[0103] The main function of this module is to invoke appropriate testing tools to perform targeted stress tests on the NPU based on the location and type of problems identified in the pre-diagnosis phase. By monitoring the testing process, it ensures the accuracy and effectiveness of the tests and ultimately outputs the test results to assist maintenance personnel in quickly locating and resolving faults.

[0104] The test script includes setting test parameters, such as test type, duration, and load level. The purpose of stress testing is to simulate high load conditions to check the performance and stability of the NPU under extreme conditions. The content and flow of the test script call by the fault diagnosis module can be seen in Figure 10, which is a schematic diagram of the test script provided in this embodiment. This flow describes how, when the fault diagnosis module starts running, the logger is initialized using `init_logger()`, a task list (TASK_LIST) is set, and values ​​are assigned to the test tasks and test-related parameters to be executed during this test. The test script obtained after initialization in Figure 10 contains five classes, indicating that this test requires the execution of five tasks: bandwidth (bandwidth test), roce (RoCE test), hbm (cache memory test), tdp_power (TDP power test), and edp_power (EDP power test). These classes correspond to their respective test tasks; for example, `bandwidth()` is used to obtain bandwidth values. According to the test script, the CPU communicates with the NPU so that the NPU can run each test task in the test script sequentially.

[0105] Test scripts can be predefined automated scripts based on different pre-diagnostic results and diagnostic strategies, ensuring targeted testing rather than blindly testing the entire spectrum. Instead, they focus on areas suspected of being problematic. For example, if the diagnostic strategy indicates a decline in NPU performance, the focus is on bandwidth testing; if there are memory errors, the emphasis is on memory verification. This targeted testing reduces irrelevant testing, saves time, avoids a broad, indiscriminate search, directly addresses the problem, and improves diagnostic efficiency.

[0106] S104. The CPU loads the NPU driver and performs a stress test on the NPU computing unit.

[0107] This step can be implemented by the fault diagnosis module in the CPU's diagnostic system. The goal of stress testing is to check the stability and performance of the NPU under high load, thereby verifying whether the NPU has the problems or performance bottlenecks indicated by the pre-diagnostic results.

[0108] The NPU driver is a program that enables communication between the automatic diagnostic system and the NPU. It is used to establish a communication connection between the CPU running the automatic diagnostic system and the NPU. When the NPU driver is loaded, the fault diagnosis module can control the NPU computing unit to perform stress test related tasks.

[0109] After the driver is loaded, the fault diagnosis module can start a stress test. For example, the CPU sends calculation instructions to the NPU computing unit so that the NPU computing unit can perform calculations and obtain the test results of the current stress test.

[0110] During the S105 and NPU tests, the CPU reads the server motherboard BMC log through a second preset command and determines whether the server motherboard BMC is alarming based on the server motherboard BMC log.

[0111] Embed a second preset command in the test script, such as a command using the ipmitool tool, like ipmitool sel list, to periodically (e.g., every few seconds) or in real time read the server motherboard BMC logs to check for any abnormalities or alarm information, and wait for the NPU failure to reproduce during the execution of the test script.

[0112] The server motherboard BMC logs being read include, but are not limited to, key indicators such as temperature, power consumption, and error rate. Based on the read BMC logs, it is determined whether the server motherboard BMC has triggered an alarm, which may include hardware events such as overheating, abnormal voltage, or fan failure.

[0113] S106. When the motherboard BMC alarms, stop the currently executing test task in the test script.

[0114] If the server motherboard BMC logs read during testing trigger a motherboard BMC alarm, the fault diagnosis module immediately stops the currently running test task. This means that even during testing, the CPU can obtain hardware status and alarm information in real time. This instant feedback mechanism can quickly respond to anomalies to prevent further hardware damage.

[0115] In one possible implementation, the number of test scripts in the task list is greater than one. For example, the test tasks for testing the NPU include three types: bandwidth test, hbm (cache memory test), and tdp_power (TDP power test). If the motherboard BMC alarm is not triggered when the bandwidth test is executed, the next cache memory test will be executed according to the order of the test tasks.

[0116] When the second test task is executed, if a motherboard BMC alarm is triggered, the current test task is terminated, and the log parsing module obtains the out-of-band information of the motherboard BMC that triggered the alarm. A third test task, the TDP power test, is then performed on the NPU. After all test tasks are completed, the log parsing module analyzes and generates diagnostic results based on the server motherboard BMC logs from the two instances of triggered alarms.

[0117] S107. The CPU obtains the in-band log of the automatic diagnostic system and generates diagnostic results based on the in-band log of the automatic diagnostic system and the server motherboard BMC log that triggers the motherboard BMC alarm.

[0118] In-band logs refer to the logs generated by the CPU running the automatic diagnostic system. Relative to the client OS system, the automatic diagnostic system provided in this embodiment is an independent OS system, and the automatic diagnostic system has the authority to access these in-band logs. The in-band logs provide the NPU's running status and software-level error information during the testing process.

[0119] The server motherboard BMC logs that trigger motherboard BMC alarms include fault information detected by the BMC. During operation, the NPU may also generate in-band alarms. Log information generated internally by the NPU reflects the NPU's status and encountered errors during task execution, including but not limited to: NPU tool runtime exceptions, memory management problems, abnormal performance metrics, application crashes or hangs, and algorithm execution errors. Therefore, it is necessary to obtain the in-band logs generated by the CPU running the automatic diagnostic system, as well as the motherboard BMC out-of-band information that triggers motherboard BMC alarms. This information records in detail the hardware and software status throughout the diagnostic process, as well as any possible problems or anomalies.

[0120] The diagnostic results for this diagnosis can be generated based on at least one of the following: the in-band log of the automatic diagnostic system and the out-of-band information of the motherboard BMC that triggered the motherboard BMC alarm. The diagnostic results may include the NPU fault type and the cause of the fault.

[0121] Example of in-band log triggering a motherboard BMC alarm:

[0122] Timestamp: 2024-07-17 12:35:45

[0123] Event: NPU performance degraded, processing latency increased.

[0124] Detailed information: Over the past 30 minutes, the average latency of the NPU executing neural network inference tasks increased from 10ms to 20ms. Simultaneously, CPU utilization rose from 30% to 70%, indicating that the NPU may not be effectively distributing the computational load.

[0125] Example of a server motherboard BMC log that triggers a motherboard BMC alarm:

[0126] Timestamp: 2024-07-17 12:36:00

[0127] Event: Overheat alarm triggered.

[0128] Detailed information: The NPU core temperature reached 85°C, exceeding the warning threshold of 80°C. The BMC has automatically reduced the fan speed to the maximum value, but the temperature has not yet dropped significantly.

[0129] Based on the in-band logs and out-of-band information mentioned above, the diagnostic results are determined, including the fault type and the cause of the fault.

[0130] Fault type: NPU overheating leading to performance degradation.

[0131] Cause of the failure: High-intensity, continuous computing tasks caused the NPU core temperature to rise. The cooling system may be insufficient to handle the high load, or dust accumulation on the heatsink and fan may be affecting cooling efficiency.

[0132] In one possible implementation, the corresponding processing strategy in the built-in case library is obtained based on the diagnostic results corresponding to the in-band logs of the automatic diagnostic system and the out-of-band information of the motherboard BMC that triggers the motherboard BMC alarm.

[0133] The diagnostic system also includes a built-in case library, which is used to generate subsequent operations for the NPU based on the diagnostic results, including solutions for various problems.

[0134] After the test, the log parsing module can do more than simply output raw log data. Based on a large library of fault cases and solutions built into the diagnostic system, it can generate corresponding processing strategies for the diagnostic results. These strategies provide solutions or suggestions tailored to the specific diagnostic findings. This eliminates the need for users to spend significant time searching and trying different solutions; instead, they can directly follow the system's recommendations, greatly accelerating problem-solving.

[0135] For example, the processing strategies provided by the built-in case library for the diagnostic results in the above example may include:

[0136] 1. Immediately reduce the workload of the NPU to avoid overheating and hardware damage over a prolonged period.

[0137] 2. Clean the dust inside the server and check and potentially upgrade the cooling system.

[0138] 3. Monitor temperature changes. If the problem persists, it may be necessary to replace the NPU or related heat dissipation components.

[0139] After diagnosis, the diagnostic system can output the results to a file on a server or send them to a remote location over a network. The diagnostic results can be a detailed report listing all detected problems and possible solutions. Once the diagnosis is complete and the results are obtained, the diagnostic device currently storing the diagnostic system can be removed from the server so that the diagnostic system can be reused on other servers.

[0140] Furthermore, to facilitate user operation, automated diagnostic systems can be equipped with a user-friendly interface. Users can easily initiate fault diagnosis, view diagnostic results, and obtain solutions through simple operations. This intuitive and easy-to-use user interface design lowers the barrier to entry for diagnostic systems, allowing more users to benefit from the convenience brought by automated diagnostic systems.

[0141] In real-world applications, new problems constantly arise as systems or devices are continuously updated and upgraded. Diagnostic systems can store the diagnostic results, test scripts, and fault handling strategies obtained from each diagnosis in a built-in case library. This enables the diagnostic system to continuously learn and improve its diagnostic effectiveness by learning from new fault cases. This self-optimization capability allows the system to always remain in optimal condition to meet ever-changing challenges.

[0142] For the application flow of each module of the diagnostic system in the diagnostic method, please refer to Figure 11. Figure 11 is a schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of this application. It includes:

[0143] S21. Read the server motherboard BMC log.

[0144] After the diagnostic system starts running, the fault pre-diagnosis module reads the server motherboard BMC log, which includes the NPU's BMC out-of-band information passed through by the NPU's BMC.

[0145] S22. Generate pre-diagnostic results based on the server motherboard BMC logs.

[0146] The fault pre-diagnosis module analyzes the server motherboard BMC logs to determine if it is an NPU fault. If it is an NPU fault, it generates a pre-diagnosis result and sends the result to the fault diagnosis module.

[0147] S23. Call the test script, load the NPU driver, and test the NPU computing unit.

[0148] Based on the received pre-diagnosis results, the fault diagnosis module calls the test script corresponding to the pre-diagnosis results and loads the NPU driver. After the driver is loaded, the NPU computing unit is tested according to the test script.

[0149] S24. During the test, read the server motherboard BMC log.

[0150] During the test, the fault diagnosis module reads the server motherboard BMC log and determines whether the server motherboard BMC is alarming based on the read server motherboard BMC log.

[0151] S25, Server motherboard BMC alarm, terminate test script.

[0152] When the fault diagnosis module detects an alarm in the server motherboard BMC, it stops the currently executing test script.

[0153] S26. Generate diagnostic results based on the server motherboard BMC logs that trigger motherboard BMC alarms.

[0154] After the server motherboard BMC alarms, the log parsing module obtains the in-band logs generated by the diagnostic system, as well as the server motherboard BMC logs that triggered the alarm, and analyzes the obtained logs to obtain diagnostic results.

[0155] In summary, the diagnostic system provided in this application has the following beneficial effects:

[0156] 1. Compared to relying on in-band logs for diagnosis, this application's embodiments utilize out-of-band information synchronization to collect hardware alarm status, combine intelligent analysis to predict fault location and type, and selectively perform performance tests. This ensures that fault diagnosis and performance testing are performed without directly accessing or entering the customer's operating system, eliminating the need to rely on the customer's OS system's in-band logs and solving the problem of difficulty in fault location caused by the inability to obtain logs due to customer data security considerations.

[0157] 2. Compared to the inefficiency of manually executing test commands, the diagnostic system, after completing pre-fault diagnosis, can call upon its built-in case library to automatically generate fault handling strategies based on the diagnostic results. These strategies can include targeted test commands, configuration adjustment suggestions, etc. Automating strategy generation not only improves efficiency but also reduces the possibility of human error.

[0158] 3. Compared to the time-consuming nature of full-scale testing, which fails to meet the requirements for rapid repair and business recovery, this application's embodiments, combined with the results of fault pre-diagnosis, can design precise testing schemes. These schemes only test areas that may have problems, rather than performing full-scale testing on the entire system. This significantly shortens testing time while ensuring the relevance and effectiveness of the testing. Furthermore, once a problem is discovered, processing strategies from the built-in case library can be immediately invoked for repair, achieving rapid recovery.

[0159] 4. In this embodiment, by connecting a diagnostic device storing the diagnostic system to the server to be diagnosed, the diagnostic system can be run via the CPU to diagnose the NPU. The diagnostic system, as an independent service, can be quickly deployed in various environments without the need for repeated complex tool installation and configuration, significantly simplifying the deployment and configuration process in different customer environments and reducing operational complexity and the probability of errors.

[0160] The above are some specific implementations of the diagnostic method provided in the embodiments of this application. Based on this, the embodiments of this application also provide a computer program product. When the computer program product is run on a computer, the computer implements the diagnostic method provided in the embodiments of this application.

[0161] This application also provides corresponding devices and computer storage media for implementing the diagnostic methods provided in this application.

[0162] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to enable the device to perform the diagnostic method described in any embodiment of this application.

[0163] The computer storage medium stores code, and when the code is run, the device running the code implements the diagnostic method described in any embodiment of this application.

[0164] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0165] It should be understood that in the embodiments of this application, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0166] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0167] It should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0168] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the embodiments of this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the embodiments of this application. Therefore, the embodiments of this application are not to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

A diagnostic method characterized in that, The application is applied to a server, the server comprises a central processing unit (CPU), a baseboard management controller (BMC) and a neural network processing unit (NPU), the BMC is connected with the CPU and the NPU, and the method comprises the following steps: The CPU analyzes the BMC log to obtain a pre-diagnosis result; When the pre-diagnosis result represents that the NPU is faulty, a test script is determined based on the pre-diagnosis result, and the test script is used for testing the NPU; The NPU is tested based on the test script, and the BMC log generated in the testing process is read; When the BMC log generated in the testing process represents that the BMC alarms, a diagnosis result is generated based on the in-band log generated in the testing process and the BMC log triggering the BMC alarm. The method of claim 1, wherein The BMC log comprises a sensor event log (SEL) of the BMC and a sensor event log (SEL) of the NPU, the pre-diagnosis result comprises an NPU fault type, the CPU analyzes the BMC log to obtain the pre-diagnosis result, which comprises the following steps: The SEL of the BMC is analyzed; When the SEL of the BMC comprises NPU BMC alarm information, the SEL of the NPU BMC is analyzed based on the NPU BMC alarm information to obtain an NPU fault type corresponding to the NPU BMC alarm information. The method according to claim 2, characterized in that The method further comprises the following steps: When the SEL of the BMC comprises NPU BMC alarm information, a fault code in the SEL of the NPU BMC is obtained; The obtained fault code is compared with a preset fault code table to determine an NPU fault type corresponding to the fault code, and the preset fault code table comprises an NPU fault type corresponding to each fault code. The method of claim 1, wherein The test script is determined based on the pre-diagnosis result, which comprises the following steps: Based on the pre-diagnosis result, a test task list is determined, and parameters of each test task in the test task list are assigned to obtain a test script corresponding to the pre-diagnosis result. The method according to claim 4, characterized in that The NPU is tested based on the test script, and the BMC log generated in the testing process is read, which comprises the following steps: According to the arrangement order of each test task in the test script, the NPU is tested in sequence, and the BMC log generated in the testing process is read until the BMC log generated in the testing process represents that the BMC alarms. The method according to claim 4, characterized in that When the BMC log generated in the testing process represents that the BMC alarms, the diagnosis result is generated based on the in-band log generated in the testing process and the BMC log triggering the BMC alarm, which comprises the following steps: When the BMC log generated in the testing process represents that the BMC alarms, the test task currently executed in the test script is terminated; According to the arrangement order of each test task in the test script, the NPU is tested in sequence until each test task in the test script is executed completely; The diagnosis result is generated based on the in-band log generated in the testing process and the BMC log triggering the BMC alarm. The method of claim 1, wherein Before the CPU analyzes the BMC log to obtain the pre-diagnosis result, the following steps are further included: The CPU sends a first preset instruction for requesting to acquire a BMC log to the BMC, so that the BMC returns the BMC log to the CPU after receiving the first preset instruction, and the BMC log comprises out-of-band information of the BMC of the NPU sent by the BMC of the NPU to the BMC when the NPU fails. The method of claim 1, wherein Before the CPU analyzes the BMC log to obtain a pre-diagnosis result, the method further comprises: The CPU sends a first preset instruction for requesting to acquire a BMC log to the BMC, so that the BMC sends a request for acquiring out-of-band information of the BMC of the NPU to the BMC of the NPU after receiving the first preset instruction, so that the BMC of the NPU sends the out-of-band information to the BMC; After the BMC acquires the out-of-band information of the BMC of the NPU returned by the BMC of the NPU, the CPU receives the server BMC log returned by the BMC, which contains the out-of-band information of the BMC of the NPU. The method of claim 1, wherein The method further comprises: Based on a built-in case library, a processing strategy corresponding to the diagnosis result is generated, the processing strategy comprises a solution for the diagnosis result, and the built-in case library comprises a plurality of historical diagnosis results and processing strategies corresponding to the historical diagnosis results. A computer program product, characterized in that When the computer program product runs on the computer, the computer executes the method according to any one of claims 1-9.

Images