Diagnostic method and computer program product

Abstract

The present application discloses a diagnostic method and computer program product, which are applied to a server, wherein the server includes a central processing unit (CPU), a baseboard management controller (BMC), and a neural network processor (NPU), wherein the BMC is connected to both the CPU and the NPU. The method comprises: the CPU parses the BMC log to obtain a pre-diagnosis result; when the pre-diagnosis result indicates a fault in the NPU, a test script is determined based on the pre-diagnosis result; the NPU is tested based on the test script, and the BMC log generated during the test is read; when the BMC log generated during the test indicates a BMC alarm, a diagnostic result is generated based on the in-band log generated during the test and the BMC log that triggers the BMC alarm. This eliminates the need to rely on the in-band log of the client OS system, solving the problem of fault diagnosis caused by the inability to obtain logs due to customer data security considerations.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a diagnostic method and a computer program product. Background Art

[0002] NPU (Neural Processing Unit) servers are designed specifically for compute-intensive tasks such as deep learning and neural networks. These servers play a key role in customer model training scenarios, efficiently processing large amounts of data and accelerating model training. However, in actual use, NPU servers may encounter issues such as slowdowns and card failures, which directly impact the efficiency and quality of model training.

[0003] In related technologies, problem locating relies on collecting and analyzing in-band logs. In-band logs typically refer to logs generated within the server, such as system logs, application logs, and hardware logs. These logs contain detailed information about the server's runtime and are crucial for analyzing and locating problems. However, because this data may contain sensitive information (such as user data and model parameters), customers may be reluctant to provide in-band logs for data security reasons, making problem locating impossible. Summary of the Invention

[0004] Embodiments of the present application provide a diagnostic method and a computer program product for diagnosing NPU faults without relying on in-band logs of the client OS system.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] A first aspect of an embodiment of the present application provides a diagnostic method applied to a server, wherein the server includes a central processing unit (CPU), a baseboard management controller (BMC), and a neural network processor (NPU), wherein the BMC is connected to both the CPU and the NPU, and includes:

[0007] The CPU parses the BMC log to obtain a pre-diagnosis result;

[0008] When the pre-diagnosis result indicates that the NPU is faulty, determining a test script based on the pre-diagnosis result, the test script being used to test the NPU;

[0009] Test the NPU based on the test script and read the BMC log generated during the test;

[0010] When the BMC log generated during the test indicates the BMC alarm, a diagnosis result is generated based on the in-band log generated during the test and the BMC log that triggers the BMC alarm.

[0011] The method provided by the embodiment of the present application is that when the pre-diagnosis result indicates an NPU failure, the CPU calls the corresponding test script according to the pre-diagnosis result to perform targeted testing on the NPU. During the test, the CPU monitors the motherboard BMC log. If a BMC alarm occurs during the test, the CPU can parse and obtain the diagnostic result based on the motherboard BMC log and the in-band log that triggered the alarm. Among them, the in-band log provides the operating status of the NPU during the test and error information at the software level, while the BMC log reveals the abnormal status at the hardware level. The final diagnostic result can be obtained based on the above two logs without accessing or relying on the in-band log of the client OS. This means that even if the customer restricts access to the client operating system for data security reasons, the embodiment of the present application can still implement fault diagnosis for the NPU, solving the fault diagnosis problem caused by the inability to obtain logs due to customer data security considerations.

[0012] In a possible implementation, after determining the test script based on the pre-diagnosis result, the method further includes: loading an NPU driver, wherein the NPU driver is used to establish a communication connection between the CPU and the NPU;

[0013] Testing the NPU based on the test script and reading the BMC log generated during the test includes:

[0014] When the NPU driver is loaded, the NPU is tested based on the test script, and the BMC log generated during the test is read.

[0015] In one possible implementation, the BMC log includes a sensor event log SEL of the BMC and a sensor event log SEL of the BMC of the NPU, the pre-diagnosis result includes an NPU fault type, and the CPU parses the BMC log to obtain a pre-diagnosis result, including:

[0016] Parsing the SEL of the BMC;

[0017] When the SEL of the BMC includes the BMC alarm information of the NPU, the SEL of the BMC of the NPU is parsed based on the BMC alarm information of the NPU to obtain the NPU fault type corresponding to the BMC alarm information of the NPU.

[0018] In a possible implementation, the method further includes:

[0019] When the SEL of the BMC includes the BMC alarm information of the NPU, obtaining a fault code in the SEL of the BMC of the NPU;

[0020] The acquired fault code is compared with a preset fault code table to determine the NPU fault type corresponding to the fault code, where the preset fault code table includes the NPU fault type corresponding to each fault code.

[0021] In a possible implementation, determining a test script based on the pre-diagnosis result includes:

[0022] Based on the pre-diagnosis result, a test task list is determined, and parameters of each test task in the test task list are assigned to obtain a test script corresponding to the pre-diagnosis result.

[0023] In one possible implementation, testing the NPU based on the test script and reading the BMC log generated during the test includes:

[0024] According to the arrangement order of each test task in the test script, the NPU is tested in sequence, and the BMC log generated during the test is read until the BMC log generated during the test indicates a BMC alarm.

[0025] In one possible implementation, when the BMC log generated during the test indicates the BMC alarm, generating a diagnosis result based on the in-band log generated during the test and the BMC log that triggered the BMC alarm includes:

[0026] When the BMC log generated during the test indicates the BMC alarm, terminate the test task currently being executed in the test script;

[0027] Testing the NPU in sequence according to the order in which each test task in the test script is arranged until all test tasks in the test script are completed;

[0028] A diagnosis result is generated based on the in-band log generated during the test and the BMC log that triggers the BMC alarm.

[0029] In a possible implementation, before the CPU parses the BMC log to obtain the pre-diagnosis result, the following steps are further included:

[0030] The CPU sends a first preset instruction to the BMC for requesting to obtain a BMC log, so that the BMC returns the BMC log to the CPU after receiving the first preset instruction, wherein the BMC log includes BMC out-of-band information of the NPU sent to the BMC log by the BMC of the NPU when the NPU fails.

[0031] In a possible implementation, before the CPU parses the BMC log to obtain the pre-diagnosis result, the following steps are further included:

[0032] The CPU sends a first preset instruction for requesting to obtain a BMC log to the BMC, so that after receiving the first preset instruction, the BMC sends a request for obtaining out-of-band information of the BMC of the NPU to the BMC of the NPU, so that the BMC of the NPU sends the out-of-band information to the BMC;

[0033] After the BMC obtains the BMC out-of-band information of the NPU returned by the BMC of the NPU, the CPU receives the server BMC log returned by the BMC, which includes the BMC out-of-band information of the NPU.

[0034] In a possible implementation, the method further includes:

[0035] Based on a built-in case library, a processing strategy corresponding to the diagnosis result is generated, wherein the processing strategy includes a solution for the diagnosis result. The built-in case library includes multiple historical diagnosis results and processing strategies corresponding to each historical diagnosis result.

[0036] A second aspect of an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the diagnostic method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of this embodiment, the following is a brief introduction to the drawings required for use in the description of the embodiment. Obviously, the drawings described below are only some embodiments of the embodiment of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A schematic diagram of an application scenario of the diagnostic system provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the diagnostic system framework provided in an embodiment of the present application;

[0040] Figure 3 A schematic flow chart of the first diagnostic method provided in an embodiment of the present application;

[0041] Figure 4 A schematic flow chart of the second diagnostic method provided in the embodiments of the present application;

[0042] Figure 5A schematic diagram of the operation flow of the fault pre-diagnosis module provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of the classification of analysis methods provided in the embodiments of the present application;

[0044] Figure 7 Schematic diagram of the NPU log provided in the embodiment of this application;

[0045] Figure 8 Schematic diagram of the SEL file content provided in the embodiment of this application;

[0046] Figure 9 A schematic diagram of the pre-diagnosis results provided in an embodiment of the present application;

[0047] Figure 10 A schematic diagram of a test script provided for an embodiment of the present application;

[0048] Figure 11 A schematic diagram of the operation flow of the fault pre-diagnosis module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the embodiments of the present application.

[0050] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the terms involved in the embodiments of the present application will be explained below.

[0051] A Neural Processing Unit (NPU) is a processor or chip specifically designed to perform artificial neural network calculations. It is widely used to accelerate artificial intelligence tasks, especially deep learning and machine learning algorithms.

[0052] A BMC is a specialized microcontroller used to monitor and manage various aspects of server hardware, including but not limited to temperature, power status, fan speed, and the health of the processor and memory. Each major hardware component, such as the motherboard and NPU, can have its own BMC for more detailed management and monitoring.

[0053] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0054] In an embodiment of the present application, a diagnostic method is provided in which a CPU reads and parses the motherboard BMC log to obtain a pre-diagnosis result. When the pre-diagnosis result indicates an NPU fault, the CPU calls the corresponding test script to test the NPU based on the pre-diagnosis result. During the test, the CPU monitors the motherboard BMC log to capture the recurrence of the NPU fault caused by the test. When the motherboard BMC alarm sounds, the motherboard BMC log that triggered the motherboard BMC alarm and the in-band log generated by the NPU during the test are obtained. The in-band log reflects the operating status of the NPU during the test and the error information at the software level. The motherboard BMC log that triggered the motherboard BMC alarm reflects the hardware-level fault that occurred in the NPU during the test. This information records in detail the hardware and software status during the diagnostic process, as well as possible problems or abnormal conditions. Therefore, the embodiment of the present application can perform fault diagnosis without directly contacting or entering the client operating system, and can obtain diagnostic results based on the in-band logs and out-of-band logs generated during the diagnostic process, without relying on the in-band logs of the client OS system, solving the fault diagnosis problem caused by the inability to obtain logs due to customer data security considerations.

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the embodiments of the present application.

[0056] The following describes the diagnostic system provided in the embodiment of the present application through an embodiment.

[0057] See also Figure 1 , Figure 1 This is a schematic diagram of an application scenario of the diagnostic system provided in an embodiment of the present application. The server includes a CPU, a BMC, and an NPU. The BMC is connected to both the CPU and the NPU. This embodiment of the present application provides an external diagnostic device that includes a diagnostic system. The diagnostic device can be a hard drive or a USB storage device. The diagnostic device has sufficient capacity to store the diagnostic system and all the resources required.

[0058] In actual application, a diagnostic device containing a diagnostic system can be inserted into a corresponding interface of the server, such as SATA, SAS, USB, etc. After the diagnostic device is connected to the server, the CPU can load and run the diagnostic system in the diagnostic device.

[0059] After the diagnostic system runs on the CPU, the CPU can be manually triggered to initiate the diagnostic method. Upon receiving the trigger, the CPU executes the fault diagnosis module in the diagnostic system to obtain out-of-band information from the BMC and performs a preliminary diagnosis based on this information. If the preliminary diagnosis indicates an NPU fault, further diagnosis and testing of the NPU are performed to reproduce the fault and generate a treatment strategy.

[0060] For the functions of each part of the diagnostic system, please refer to Figure 2 , Figure 2 A schematic diagram of the diagnostic system framework provided in an embodiment of the present application.

[0061] The fault pre-diagnosis module parses the server motherboard BMC logs provided by the motherboard BMC to determine the output fault location and fault type of the problematic device, providing targeted testing support for the fault diagnosis module. Analysis methods include register analysis, SEL analysis, and FDM analysis.

[0062] The fault diagnosis module is used to perform software and hardware performance testing on the NPU based on the fault location and fault type determined by the fault pre-diagnosis module, monitor the test process, and output test results. The test types of the fault diagnosis module may include HBM media testing, TDP power consumption testing, ROCE network port testing, bandwidth testing, and EDP power consumption testing. Among them, the HBN media test includes: Read: stress testing for read operations. Write: stress testing for write operations. Bandwidth testing includes: d2h: testing of data sent from the NPU to the CPU. h2d: testing of data sent from the CPU to the NPU device. d2d: data transmission test within the NPU. p2p: data transmission test within the CPU. Cache memory testing includes:

[0063] The log parsing module collects and analyzes log information generated during the test process to generate diagnostic results and action strategies. This log information includes out-of-band information from the mainboard BMC that triggers alarms, as well as in-band logs generated when the CPU runs diagnostic systems. The module can parse both in-band and out-of-band logs separately to generate corresponding diagnostic strategies. Alternatively, the module can combine in-band and out-of-band logs to generate a single diagnostic strategy.

[0064] The following describes the diagnostic method provided in the examples of the present application. Figure 3 , Figure 3 This is a flow chart of the first diagnostic method provided in an embodiment of the present application, which includes:

[0065] S110 : The CPU parses the BMC log to obtain a pre-diagnosis result.

[0066] The CPU reads and analyzes BMC logs. The BMC is responsible for managing server hardware status (such as temperature, voltage, and fan speed) and generating corresponding logs. By parsing BMC logs, the CPU can distinguish between common hardware failures (such as those in the hard drive, memory, CPU, and motherboard) and NPU-related failures, thereby providing a preliminary pre-diagnosis result.

[0067] S111 . When the pre-diagnosis result indicates that the NPU is faulty, determine a test script based on the pre-diagnosis result, where the test script is used to test the NPU.

[0068] When the pre-diagnosis result indicates an NPU fault, the test script corresponding to the pre-diagnosis result is called according to the pre-diagnosis result to perform targeted testing on the NPU to reproduce the NPU fault.

[0069] S112: Test the NPU based on the test script, and read the BMC log generated during the test.

[0070] Ensure the accuracy and effectiveness of the test by monitoring the BMC logs generated during the test.

[0071] S113 : When the BMC log generated during the test indicates the BMC alarm, generate a diagnosis result based on the in-band log generated during the test and the BMC log that triggers the BMC alarm.

[0072] If the BMC log indicates a BMC alarm during the test, the CPU generates a diagnostic result by comprehensively analyzing the in-band logs generated during the test and the BMC log that triggered the BMC alarm.

[0073] Therefore, in the embodiment of the present application, when the pre-diagnosis result indicates an NPU failure, the CPU calls the corresponding test script according to the pre-diagnosis result to perform targeted testing on the NPU. During the test, the CPU monitors the motherboard BMC log. If a BMC alarm occurs during the test, the CPU can parse and obtain the diagnostic result based on the motherboard BMC log and the in-band log that triggered the alarm. Among them, the in-band log provides the operating status of the NPU during the test and error information at the software level, while the BMC log reveals the abnormal status at the hardware level. The final diagnostic result can be obtained based on the above two logs without accessing or relying on the in-band log of the client OS. This means that even if the customer restricts access to the client operating system for data security reasons, the embodiment of the present application can still implement fault diagnosis for the NPU, solving the fault diagnosis problem caused by the inability to obtain logs due to customer data security considerations. The following describes the diagnostic method performed by the diagnostic system, see Figure 4 , Figure 4This is a schematic diagram of a second diagnostic method provided in an embodiment of the present application, which includes:

[0074] S101. The CPU reads the server mainboard BMC log through a first preset instruction.

[0075] The CPU running the diagnostic system sends a first preset instruction to the server motherboard BMC, requesting to obtain the server motherboard BMC log. The motherboard BMC can obtain and integrate information from different component BMCs.

[0076] The server motherboard BMC log can be used to save important information about the server hardware status, events, and potential failures. The server motherboard BMC can record the fault information of the NPU in the server motherboard BMC log file, and can forward the server motherboard BMC log file to the server's management system, such as the diagnostic system provided in the embodiment of the present application. In other words, the server motherboard BMC log includes not only the BMC out-of-band information of the server motherboard, but also the BMC out-of-band information of the NPU. The motherboard BMC and the NPU's BMC each record the hardware status and anomalies, and the two are combined to provide a comprehensive hardware health overview, allowing the diagnostic system to have a more comprehensive understanding of the operation of the NPU and the motherboard.

[0077] In the first scenario, after receiving the first preset instruction, the server motherboard's BMC sends a request to the NPU's BMC to obtain the NPU's BMC's out-of-band information, causing the NPU's BMC to send the out-of-band information to the server motherboard's BMC. After the server motherboard's BMC obtains the NPU's BMC out-of-band information returned by the NPU's BMC, the server motherboard's BMC returns the server motherboard's BMC log containing the NPU's BMC out-of-band information to the CPU.

[0078] In another scenario, when the NPU's BMC detects an NPU fault, the NPU's BMC can transmit the NPU's out-of-band information to the server's mainboard BMC via a preset interface. The server's mainboard BMC records the received NPU's BMC out-of-band information directly or after processing it in the server's mainboard BMC log. This allows the server's mainboard BMC to return the server's mainboard BMC log containing the NPU's BMC out-of-band information to the CPU upon receiving the first preset instruction from the CPU.

[0079] For example, during the out-of-band information transmission process of the NPU or the motherboard, the out-of-band information can be packaged into data packets, and HTTPs or XML, JSON and other formats can be used to ensure secure encryption protocol transmission and ensure transmission to the server motherboard BMC. Among them, the preset interface can be Redfish. Redfish is a modern management interface that allows hardware status and configuration data to be remotely accessed and exchanged in a standardized manner, providing efficiency and security. Using this interface, the NPU's BMC can notify the server motherboard BMC of out-of-band information containing NPU-related fault details in real time. In addition, Redfish can use HTTPS-based communication to ensure the security of data during transmission.

[0080] After receiving the NPU's out-of-band BMC information, the server motherboard's BMC parses it and converts it into a user-readable format, displaying it on the web interface. This conversion allows maintenance personnel or IT administrators to intuitively view the NPU's operating status through the web interface, improving transparency and responsiveness.

[0081] S102: The CPU parses the server mainboard BMC log to determine a pre-diagnosis result.

[0082] The fault pre-diagnosis module in the CPU operation diagnostic system analyzes the server motherboard BMC logs to distinguish between common hardware faults (such as hard disk, memory, CPU, and motherboard) and NPU-related faults. If an NPU fault is determined, the module parses the out-of-band information about the NPU in the server motherboard BMC logs to determine the pre-diagnosis result. This pre-diagnosis result is used to guide subsequent NPU performance testing.

[0083] Once the server motherboard BMC log indicates an NPU failure, the CPU running the diagnostic system will analyze the out-of-band information from the NPU's BMC in the server motherboard BMC log to obtain more specific problem details and the corresponding diagnostic strategy. NPU lower-layer logs refer to log information generated by the underlying hardware or firmware during the NPU's task execution. This information may include the NPU's operating status, error codes, performance data, etc.

[0084] The workflow of the fault prediction module can be found in Figure 5 , Figure 5 The schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of the present application includes:

[0085] First, the fault pre-diagnosis module obtains the server mainboard BMC log, which includes fault information provided by the server mainboard BMC, which may be fault information caused by server mainboard fault and / or fault information caused by NPU fault.

[0086] Exemplarily, the fault pre-diagnosis module can use a first preset command to collect server motherboard BMC logs with one click. The collected server motherboard BMC logs are pre-processed, which may involve compression, formatting, or other pre-processing tasks to facilitate subsequent analysis. The first preset command can be ipmcget -d diaginfo.

[0087] Then parse the pre-processed server motherboard BMC log, such as Figure 2 As shown, the parsing methods can include register parsing, SEL (System Event Log) parsing, and FDM (Frequency-division multiplexing) parsing. It should be noted that during the pre-diagnosis process, the server motherboard BMC log is parsed using these three parsing methods. The three parsing methods can be performed simultaneously or sequentially.

[0088] The first one is register parsing. The health status and fault information of hardware devices (such as NPU, memory, hard disk, etc.) are often recorded in specific registers. These registers contain a series of binary bits, each bit representing a state or fault flag. The server motherboard BMC log includes the values of the registers. By reading the values of these registers and then interpreting them according to the predetermined bit mapping table, it can be determined whether a fault has occurred and the specific type of fault. During the log parsing process, the collected logs can be parsed based on the register-related information to obtain the precise fault output of the power supply unit (PSU). If the pre-diagnosis result only indicates that there is a fault in the PSU, and other parsing methods indicate that there is no fault in the current NPU, the diagnosis is terminated.

[0089] Secondly, FDM is an intelligent fault management engine in the server that monitors each component of the server and reports and analyzes possible hardware failures in a timely manner. FDM can work in conjunction with the server to parse the server motherboard BMC log to determine whether there is a fault in the server and its type. Specifically, the FDM engine can parse the server motherboard BMC log to find any anomalies or error codes that may indicate a hardware failure. The analysis includes but is not limited to CPU errors, memory errors, power supply problems, fan failures, temperature exceeds the limit, etc. Based on the parsed error codes and anomalies, FDM can match the preset fault pattern database to identify the specific fault type. These patterns can involve the failure of a single component, such as a CPU or memory module, or more complex system-level problems.

[0090] Therefore, by analyzing FDM logs, we can quickly locate the specific source of the fault, such as the precise fault output related to the dual in-line memory modules (DIMMs), disks (DISKs), and redundant arrays of independent disks (RAIDs). If the pre-diagnosis results indicate only DIMM, DISK, or RAID faults, and other analysis methods indicate that the current NPU is not faulty, the diagnosis is terminated.

[0091] The third type is SEL parsing. The SEL log is generated by the BMC and is used to record important information about the server's hardware and health status. When events such as hardware failure, temperature anomaly, power supply problem, system restart, etc. are detected, the BMC will record these events in the SEL log, and each event has a unique fault code (SELRecord ID). The server motherboard BMC log includes the sel file of the motherboard BMC and the sel file of the NPU's BMC. In the embodiment of the present application, the server includes a motherboard and an NPU, each of which has a BMC. The NPU's BMC and the motherboard BMC each have a SEL file. The motherboard's SEL file includes fault information of the memory and hard disk, and the NPU's BMC's SEL file includes NPU-related fault information. During the operation of the NPU's BMC, the two SEL files can communicate to pass the SEL alarm of the NPU's BMC to the SEL file of the motherboard BMC. Therefore, the alarm information transmitted by the NPU can also be seen in the SEL file of the motherboard BMC.

[0092] In actual application scenarios, some NPU alarm information will be transparently transmitted to the motherboard's BMC, but not all NPU alarm information can be transparently transmitted to the motherboard BMC. Therefore, in order to more accurately and comprehensively determine the NPU alarm information, the NPU's SEL file can be further analyzed to determine the pre-diagnosis result. Therefore, after querying the motherboard BMC's SEL file for the presence of transparent NPU alarm information, the NPU's SEL file is further parsed. In the process of parsing the NPU's SEL file, pre-diagnosis can be performed through at least one of the two methods: the NPU's BMC alarm information or the NPU fault ID in the NPU's SEL file.

[0093] See also Figure 6 , Figure 6 A schematic diagram of the classification of the analysis methods provided in the embodiments of this application, such as Figure 6 The first analysis method uses the NPU's BMC alarm information to determine the NPU fault sequence number or fault symptom. This analysis method is applicable to the following scenarios: card loss, bandwidth reduction, overtemperature, carrier board failure, and temperature acquisition failure. These faults can cause immediate and visible failures or performance degradation, impacting server stability and availability. Therefore, when the interface between BMCs is limited, only the NPU's BMC's high-impact fault information is transparently transmitted to the motherboard BMC's SEL file. This information can be recorded by both the motherboard BMC and the NPU's BMC as NPU BMC alarm information and recorded in the NPU's BMC's SEL file, enabling rapid response when faults are detected. Furthermore, the NPU's BMC can continue to record sensor events during operation in the NPU's BMC's SEL file, including the transparently transmitted alarm information and more detailed event records. Therefore, when parsing the NPU's BMC's SEL file, pre-diagnostic results can be obtained based on the NPU alarm information recorded in the SEL file.

[0094] For example, when the CPU reads the motherboard BMC's SEL file, it contains the NPU BMC alarm information transparently transmitted from the NPU BMC: NPU bandwidth reduction. To ensure the accuracy of the pre-diagnosis result, the CPU specifically reads the NPU BMC's SEL file based on the NPU bandwidth reduction. If the NPU BMC's SEL file contains NPU bandwidth reduction, if the current server has only one NPU module, the pre-diagnosis result includes NPU bandwidth reduction.

[0095] In one possible implementation, if the number of NPU modules in the current server is greater than one, in order to more accurately determine which module should be tested later, the CPU can read the NPU BMC alarm information: NPU bandwidth reduction, and further analyze the NPU systemcom.dat file in the motherboard BMC log, refer to Figure 7 , Figure 7 This is a schematic diagram of the NPU log provided by the embodiment of this application. The file contains the link establishment information of the NPU, that is, how many NPU cards are identified and what the bandwidth is. During the parsing process, the pre-diagnosis module uses LINKSTS as the keyword to perform orthogonal matching on the log file, such as Figure 7 As shown in the figure, the link establishment status of NPU 10 can be identified as 0x2045. The NPU 10 module bandwidth is reduced as a pre-diagnosis result. Subsequently, targeted testing and diagnosis of the NPU 10 module are performed.

[0096] like Figure 6 The second analysis method is to determine the fault phenomenon through the NPU fault ID. Figure 6 This also includes some situations that can be pre-diagnosed using the fault ID, that is, querying the NPU fault ID recorded in the SEL file. This parsing method can be applied to the following situations: network port disconnection, ECC isolation, and L2 Buff. These faults are more subtle and may not immediately cause a system crash, but they can affect system reliability and data integrity. When BMC interface resources are limited, these fault information is not transparently transmitted to the motherboard BMC SEL file. Therefore, this parsing method can be used for NPU fault types that are not transparently transmitted to the motherboard BMC SEL file.

[0097] refer to Figure 8 , Figure 8 The CPU can traverse the SEL file of the NPU and try to obtain the fault ID recorded in the SEL file. Figure 8 Extract the fault ID, such as 0X81078603. Fault IDs are usually associated with specific fault types. The pre-diagnosis module compares the fault ID: 0X81078603 with the pre-defined fault code table. The fault code table lists all possible fault IDs and their corresponding fault descriptions and recommended processing strategies. By looking up the table, the pre-diagnosis module can determine the specific NPU fault type represented by the 0X81078603 fault ID, such as NPU memory, bandwidth anomalies, or network port connection problems. After clarifying the fault ID, the diagnostic system can not only locate the problem, but also query the specific information of the fault, such as Figure 8The fault details shown are: 1. The network port chip detected a link failure on its own; 2. The network port chip detected a link failure on the peer end. Fault type: Network port function unavailable. The current fault resolution measures are: 1. Reporting the fault to fault management; 2. Logging. Current fault severity: Major.

[0098] The pre-diagnosis module can then output the specific location of the fault and the recommended handling strategy, refer to Figure 9 , Figure 9 The schematic diagram of the pre-diagnosis results provided for the embodiment of the present application includes: the fault ID is 0X81078603, and the pre-diagnosis result is used to guide maintenance personnel to take corresponding actions, such as hardware inspection, firmware upgrade or parameter adjustment.

[0099] The fault pre-diagnosis module can parse the server motherboard BMC log to obtain the output fault location and / or fault type of the problem device. Based on the above pre-diagnosis results, the fault pre-diagnosis module outputs a diagnostic strategy to enable the fault diagnosis module to perform software and hardware performance testing on the NPU.

[0100] S103 : The CPU calls a test script corresponding to the pre-diagnosis result according to the pre-diagnosis result.

[0101] This step can be implemented by the CPU running a fault diagnosis module in the diagnosis system. The fault diagnosis module in the diagnosis system is described below.

[0102] This module's primary function is to invoke appropriate testing tools to perform targeted stress testing on the NPU based on the problem location and type identified during the pre-diagnosis phase. It monitors the test progress to ensure accuracy and effectiveness, and ultimately outputs test results to help maintenance personnel quickly locate and resolve faults.

[0103] The test script includes setting test parameters, such as test type, duration, load level, etc. The purpose of stress testing is to simulate high load conditions to check the performance and stability of the NPU under extreme conditions. For the content and process of calling the test script in the fault diagnosis module, please refer to Figure 10 , Figure 10 This is a schematic diagram of the test script provided by the embodiment of the present application. This process describes how, when the fault diagnosis module starts running, init_logger() is used to initialize the logger, set a task list (TASK_LIST), and assign values to the test tasks and test-related parameters that need to be executed during this test. Figure 10The test script generated after initialization contains five classes, indicating the five tasks required for this test: bandwidth (bandwidth test), roce (RoCE test), hbm (cache memory test), tdp_power (TDP power test), and edp_power (EDP power test). These classes correspond to their respective test tasks, such as bandwidth() for obtaining bandwidth values. According to the test script, the CPU communicates with the NPU, allowing the NPU to run each test task in the test script in sequence.

[0104] Test scripts can be automated, predefined scripts based on different diagnostic strategies based on pre-diagnostic results. This ensures targeted testing, rather than blindly measuring everything. Instead, it focuses on suspected problem areas. For example, if the diagnostic strategy indicates NPU performance degradation, bandwidth testing is focused; if memory errors are detected, memory verification is emphasized. This targeted testing reduces irrelevant testing, saves time, and avoids a broad-based troubleshooting, directly identifying the problem and improving diagnostic efficiency.

[0105] S104: The CPU loads the NPU driver and performs a stress test on the NPU computing unit.

[0106] This step can be implemented by the fault diagnosis module in the CPU operation diagnosis system. The goal of the stress test is to check the stability and performance of the NPU under high load, thereby verifying whether the NPU has the problems or performance bottlenecks indicated by the pre-diagnosis results.

[0107] Among them, the NPU driver is a program for communication between the automatic diagnosis system and the NPU, which is used to establish a communication connection between the CPU running the automatic diagnosis system and the NPU. When the NPU driver is loaded, the fault diagnosis module can control the NPU computing unit to perform related tasks of the stress test.

[0108] After the driver is loaded, the fault diagnosis module may start a stress test. For example, the CPU sends a calculation instruction to the NPU calculation unit so that the NPU calculation unit performs calculations, thereby obtaining a test result of the current stress test.

[0109] S105 , during the NPU test, the CPU reads the server mainboard BMC log through a second preset command, and determines whether the server mainboard BMC has issued an alarm based on the server mainboard BMC log.

[0110] A second preset command is embedded in the test script, such as a command using the ipmitool tool, such as ipmitoolsel list, to read the server motherboard BMC log periodically (for example, once every few seconds) or in real time to check whether there are any abnormal conditions or alarm messages, and wait for the NPU fault to reappear during the execution of the test script.

[0111] The server motherboard BMC logs read include, but are not limited to, key indicators such as temperature, power consumption, and error rate. Based on the read BMC logs, it is determined whether the server motherboard BMC has triggered an alarm, which can include hardware events such as overtemperature, abnormal voltage, and fan failure.

[0112] S106. When the mainboard BMC alarms, the test task currently being executed in the test script is stopped.

[0113] If the server motherboard BMC log read during testing triggers a motherboard BMC alarm, the fault diagnosis module immediately stops the currently running test task. This means that even during testing, the CPU can obtain real-time hardware status and alarm information. This immediate feedback mechanism enables rapid response to anomalies to prevent further hardware damage.

[0114] In one possible implementation, the number of test scripts in the task list is greater than 1. For example, the test tasks for testing the NPU include bandwidth (bandwidth test), hbm (cache memory test), and tdp_power (TDP power test). When executing the bandwidth test, the motherboard BMC alarm is not triggered, and the next cache memory test is executed in the order of the test tasks.

[0115] When the second test task triggers a mainboard BMC alarm, the current test task is terminated. The log parsing module then retrieves the out-of-band BMC information that triggered the alarm. The third test task, TDP power testing, is performed on the NPU. After all test tasks are completed, the log parsing module analyzes the server mainboard BMC logs that triggered the two mainboard BMC alarms and generates diagnostic results.

[0116] S107 : The CPU obtains an in-band log of the automatic diagnosis system, and generates a diagnosis result based on the in-band log of the automatic diagnosis system and the server mainboard BMC log that triggers the mainboard BMC alarm.

[0117] In-band logs are logs generated by the CPU's automatic diagnostic system. The automatic diagnostic system provided in this embodiment is independent of the client OS and has access to these logs. These logs provide information about the NPU's operating status and software-level errors during testing.

[0118] The server's mainboard BMC logs, which trigger BMC alarms, include information about faults detected by the BMC. The NPU may also generate in-band alarms during operation. Logs generated internally by the NPU reflect its status and errors encountered during task execution, including but not limited to: NPU tool runtime anomalies, memory management issues, performance metric anomalies, application crashes or hangs, and algorithm execution errors. Therefore, it is necessary to obtain both in-band logs generated by the CPU's automatic diagnostic system and out-of-band BMC information that triggers mainboard BMC alarms. This information details the hardware and software status throughout the diagnostic process, as well as any potential issues or anomalies.

[0119] The diagnosis result of this diagnosis can be generated based on at least one of the in-band log of the automatic diagnosis system and the out-band information of the motherboard BMC that triggered the motherboard BMC alarm. The diagnosis result can include the fault type and cause of the NPU.

[0120] Example of an in-band log that triggers a mainboard BMC alarm:

[0121] Timestamp: 2024-07-17 12:35:45

[0122] Event: NPU performance degraded and processing latency increased.

[0123] Details: Over the past 30 minutes, the average latency of the NPU performing neural network inference tasks increased from 10ms to 20ms. At the same time, CPU utilization increased from 30% to 70%, indicating that the NPU may not be able to effectively share the computing load.

[0124] Example of a server motherboard BMC log that triggers a motherboard BMC alarm:

[0125] Timestamp: 2024-07-17 12:36:00

[0126] Event: Overheat alarm triggered.

[0127] Details: The NPU core temperature reached 85°C, exceeding the warning threshold of 80°C. The BMC automatically reduced the fan speed to the maximum, but the temperature did not drop significantly.

[0128] The diagnosis result is determined by combining the above in-band logs and out-of-band information. The diagnosis result includes the fault type and cause.

[0129] Fault type: NPU overheating causes performance degradation.

[0130] Cause: High-intensity, continuous computing tasks cause the NPU core temperature to rise. The cooling system may not be sufficient to handle the high load, or dust accumulation on the radiator and fan may affect cooling efficiency.

[0131] In a possible implementation, a corresponding processing strategy in a built-in case library is obtained based on the diagnosis result corresponding to the in-band log of the automatic diagnosis system and the out-band information of the mainboard BMC that triggers the mainboard BMC alarm.

[0132] The diagnostic system also includes a built-in case library, which is used to generate subsequent operations to be performed on the NPU based on the diagnostic results, including solutions corresponding to various problems.

[0133] After the test is complete, the log parsing module goes beyond simply outputting raw log data. Based on the diagnostic system's built-in case library, which contains numerous failure cases and solutions, it generates a corresponding processing strategy for the diagnostic results. This strategy then provides solutions or suggestions for the diagnostic results. This eliminates the need for users to spend time searching and trying different solutions, allowing them to directly follow the system's suggestions, significantly accelerating problem resolution.

[0134] For example, for the diagnostic results of the above example, the processing strategies provided by the built-in case library may include:

[0135] 1. Immediately reduce the workload of the NPU to avoid prolonged overheating and damage to the hardware.

[0136] 2. Clean the dust inside the server, check and possibly upgrade the cooling system.

[0137] 3. Monitor temperature changes. If the problem persists, the NPU or related cooling components may need to be replaced.

[0138] After the diagnosis is complete, the diagnostic system can output the results to a file on the server or send them to a remote location over the network. This diagnostic result can be a detailed report listing all detected issues and possible solutions. Once the diagnosis is complete and the results are retrieved, the diagnostic device currently storing the diagnostic system can be removed from the server so that the diagnostic system can be reused on another server.

[0139] To facilitate user experience, automated diagnostic systems can be equipped with user-friendly interfaces. With simple operations, users can easily initiate fault diagnosis, view diagnostic results, and obtain solutions. This intuitive and user-friendly interface lowers the barrier to entry for diagnostic systems, allowing more users to benefit from the convenience they bring.

[0140] In real-world applications, new problems constantly arise as systems or devices are constantly updated and upgraded. The diagnostic system can store the diagnostic results, test scripts, and troubleshooting strategies obtained from each diagnosis in a built-in case library. This enables the system to continuously learn and improve its diagnostic effectiveness by learning new fault cases. This self-optimizing capability ensures that the system remains in optimal condition to meet ever-changing challenges.

[0141] For the application process of each module of the diagnostic system in the diagnostic method, please refer to Figure 11 , Figure 11 This is a schematic diagram of the operation flow of the fault pre-diagnosis module provided in the embodiment of the present application. It includes:

[0142] S21. Read the server motherboard BMC log.

[0143] After the diagnosis system starts running, the fault pre-diagnosis module reads the server mainboard BMC log, wherein the server mainboard BMC log includes the NPU BMC out-of-band information transparently transmitted by the NPU BMC.

[0144] S22. Generate a pre-diagnosis result based on the server motherboard BMC log.

[0145] The fault pre-diagnosis module analyzes whether the fault is an NPU fault based on the server motherboard BMC log it reads. If it is an NPU fault, it generates a pre-diagnosis result and sends it to the fault diagnosis module.

[0146] S23. Call the test script, load the NPU driver, and test the NPU computing unit.

[0147] Based on the received pre-diagnosis results, the fault diagnosis module calls the test script corresponding to the pre-diagnosis results and loads the NPU driver. After the driver is loaded, the NPU computing unit is tested according to the test script.

[0148] S24. During the test, read the server motherboard BMC log.

[0149] During the test, the fault diagnosis module reads the server motherboard BMC log and determines whether the current server motherboard BMC alarm is triggered based on the read server motherboard BMC log.

[0150] S25. The server motherboard BMC alarms and terminates the test script.

[0151] When the fault diagnosis module detects a BMC alarm on the server motherboard, it stops the currently executing test script.

[0152] S26. Generate a diagnosis result based on the server mainboard BMC log that triggers the mainboard BMC alarm.

[0153] After the server motherboard BMC alarm, the log parsing module obtains the in-band log generated by the diagnostic system and the server motherboard BMC log that triggered the motherboard BMC alarm, and analyzes the obtained logs to obtain the diagnosis results.

[0154] In summary, the diagnostic system provided in the embodiments of the present application has the following beneficial effects:

[0155] 1. Compared to relying on in-band logs for diagnosis, the embodiments of this application utilize out-of-band information synchronization to collect hardware alarm status, combine intelligent analysis to predict fault location and type, and selectively perform performance testing. This ensures that fault diagnosis and performance testing can be performed without directly accessing or entering the client operating system, eliminating the need to rely on the client OS's in-band logs. This solves the problem of difficulty in locating faults caused by the inability to obtain logs due to customer data security concerns.

[0156] 2. Compared to the inefficient manual execution of test commands, after completing fault pre-diagnosis, the diagnostic system can call upon a built-in case library to automatically generate troubleshooting strategies based on the diagnostic results. These strategies can include targeted test commands and configuration adjustment suggestions. Automated strategy generation not only improves efficiency but also reduces the possibility of human error.

[0157] 3. Compared to full-scale testing, which takes a long time and cannot meet the requirements for rapid repair and service recovery, the embodiments of the present application can design precise testing plans based on the results of fault pre-diagnosis. These plans only test areas where problems may exist, rather than fully testing the entire system. This can greatly shorten testing time while ensuring the targeted and effective testing. In addition, once a problem is discovered, the processing strategy in the built-in case library can be immediately called to repair it, achieving rapid recovery.

[0158] 4. In this embodiment, by connecting a diagnostic device storing a diagnostic system to the server to be diagnosed, the CPU can run the diagnostic system to diagnose the NPU. As an independent service, the diagnostic system can be quickly deployed in a variety of environments without the need for repeated complex tool installation and configuration. This greatly simplifies the deployment and configuration process in different customer environments, reducing operational complexity and the probability of error.

[0159] The above are some specific implementation methods of the diagnostic method provided in the embodiments of the present application. Based on this, the embodiments of the present application also provide a computer program product. When the computer program product runs on a computer, the computer implements the diagnostic method provided in the embodiments of the present application.

[0160] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the diagnostic method provided in the embodiments of the present application.

[0161] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device performs the diagnostic method described in any embodiment of the present application.

[0162] The computer storage medium stores code, and when the code is executed, the device executing the code implements the diagnostic method described in any embodiment of the present application.

[0163] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0164] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0166] It should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0167] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0168] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] The above description of the disclosed embodiments enables those skilled in the art to implement or use the embodiments of the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the embodiments of the present application. Therefore, the embodiments of the present application are not limited to the embodiments shown herein, but are intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A diagnostic method, characterized in that The method is applied to a server, the server including a central processing unit (CPU), a baseboard management controller (BMC), and a neural network processor (NPU). The BMC is connected to both the CPU and the NPU. The server is connected to a diagnostic device including a diagnostic system. The CPU loads and runs the diagnostic system to start the method. The method includes: The CPU parses the BMC log to obtain a pre-diagnosis result; When the pre-diagnosis result indicates that the NPU is faulty, determining a test script based on the pre-diagnosis result, the test script being used to test the NPU; wherein determining the test script includes setting test parameters; Test the NPU based on the test script and read the BMC log generated during the test; When the BMC log generated during the test indicates the BMC alarm, a diagnostic result is generated based on the in-band log generated during the test and the BMC log that triggers the BMC alarm; wherein the in-band log is a log generated by the CPU running the diagnostic system.

2. The method according to claim 1, characterized in that The BMC log includes a sensor event log SEL of the BMC and a sensor event log SEL of the BMC of the NPU. The pre-diagnosis result includes an NPU fault type. The CPU parses the BMC log to obtain a pre-diagnosis result, including: Parsing the SEL of the BMC; When the SEL of the BMC includes the BMC alarm information of the NPU, the SEL of the BMC of the NPU is parsed based on the BMC alarm information of the NPU to obtain the NPU fault type corresponding to the BMC alarm information of the NPU.

3. The method according to claim 2, characterized in that The method further comprises: When the SEL of the BMC includes the BMC alarm information of the NPU, obtaining a fault code in the SEL of the BMC of the NPU; The acquired fault code is compared with a preset fault code table to determine the NPU fault type corresponding to the fault code, where the preset fault code table includes the NPU fault type corresponding to each fault code.

4. The method according to claim 1, wherein Determining a test script based on the pre-diagnosis result includes: Based on the pre-diagnosis result, a test task list is determined, and parameters of each test task in the test task list are assigned to obtain a test script corresponding to the pre-diagnosis result.

5. The method according to claim 4, characterized in that The NPU is tested based on the test script and the BMC log generated during the test is read, including: According to the arrangement order of each test task in the test script, the NPU is tested in sequence, and the BMC log generated during the test is read until the BMC log generated during the test indicates a BMC alarm.

6. The method according to claim 4, characterized in that When the BMC log generated during the test indicates the BMC alarm, generating a diagnosis result based on the in-band log generated during the test and the BMC log that triggered the BMC alarm includes: When the BMC log generated during the test indicates the BMC alarm, terminate the test task currently being executed in the test script; Testing the NPU in sequence according to the order in which each test task in the test script is arranged until all test tasks in the test script are completed; A diagnosis result is generated based on the in-band log generated during the test and the BMC log that triggers the BMC alarm.

7. The method according to claim 1, characterized in that Before the CPU parses the BMC log to obtain the pre-diagnosis result, the following steps are further performed: The CPU sends a first preset instruction to the BMC for requesting to obtain a BMC log, so that the BMC returns the BMC log to the CPU after receiving the first preset instruction, wherein the BMC log includes BMC out-of-band information of the NPU sent to the BMC log by the BMC of the NPU when the NPU fails.

8. The method according to claim 1, characterized in that Before the CPU parses the BMC log to obtain the pre-diagnosis result, the following steps are further performed: The CPU sends a first preset instruction for requesting to obtain a BMC log to the BMC, so that after receiving the first preset instruction, the BMC sends a request for obtaining out-of-band information of the BMC of the NPU to the BMC of the NPU, so that the BMC of the NPU sends the out-of-band information to the BMC; After the BMC obtains the BMC out-of-band information of the NPU returned by the BMC of the NPU, the CPU receives the server BMC log returned by the BMC, which includes the BMC out-of-band information of the NPU.

9. The method according to claim 1, characterized in that The method further comprises: Based on a built-in case library, a processing strategy corresponding to the diagnosis result is generated, wherein the processing strategy includes a solution for the diagnosis result. The built-in case library includes multiple historical diagnosis results and processing strategies corresponding to each historical diagnosis result.

10. A computer program product, characterized in that When the computer program product is run on a computer, the computer executes the method according to any one of claims 1 to 9.

Images