Software and hardware information monitoring and fault diagnosis method applied to AI server

Abstract

The invention relates to the technical field of server fault diagnosis, belongs to a software and hardware information monitoring and fault diagnosis method applied to an AI server, provides a topological structure applied to software and hardware state monitoring of the AI server on one hand, and provides a software and hardware information monitoring and fault diagnosis method applied to the AI server on the other hand. According to the technical scheme, hardware key information of main board cards (a mainboard, an exchange board, a GPU board and the like) in the server is collected, and the monitoring function of hardware signals is achieved; main running state data of system software are collected, and the function of monitoring the running state of server software is achieved; a trained AI large model is embedded in the fault diagnosis module, so that the functions of quickly positioning hardware and software faults and generating fault processing suggestions are realized.

Description

Technical Field

[0001] This invention relates to the field of server fault diagnosis technology, and pertains to a method for monitoring and diagnosing hardware and software information and faults in AI servers. Background Technology

[0002] With the explosive growth in AI computing power demand, the scale and complexity of AI server clusters continue to rise. The reliability and stability of their hardware systems (such as GPUs, high-speed interconnects, power supplies, and heat dissipation modules) have become major factors affecting overall performance. Traditional hardware monitoring technologies rely on threshold alarms and statistical analysis of structured data (such as temperature, voltage, and power consumption), but they have limitations when dealing with high-dimensional, non-linearly correlated fault modes. At the same time, intelligent diagnostic technologies based on Large Language Models (LLMs) are rapidly developing. Through multimodal data fusion (log text, time-series indicators, and hardware topology diagrams) and contextual reasoning capabilities, they can uncover hidden fault features that are difficult to capture using traditional methods.

[0003] Figure 1 This is a flowchart of a server fault diagnosis method, apparatus, device, and readable storage medium disclosed in patent number CN113886120A. This patent uses various server faults and corresponding solutions as a knowledge source to construct a fault knowledge graph. Based on real-time fault information, the graph nodes and adjacent nodes automatically searched can include key content and solutions of the current fault information, improving the efficiency and accuracy of server fault diagnosis. This patent only describes the fault diagnosis method at the server system level; it does not include the specific methods for obtaining fault information or its hardware topology. However, server fault diagnosis methods are inseparable from specific hardware structures.

[0004] The existing technical solutions mainly have the following three problems: 1. The current solution only designs fault diagnosis methods / algorithms and does not provide a hardware topology that can realize whole-machine fault detection; 2. The current solution is designed only for hardware fault diagnosis or software fault diagnosis, without simultaneously monitoring and diagnosing both hardware and software faults; 3. The current solution is based on a fault knowledge graph, but does not use a large model for server hardware and software fault diagnosis. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for monitoring and diagnosing hardware and software information and faults in AI servers.

[0006] To achieve the above objectives, the present invention provides the following technical solution: On one hand, a topology for monitoring the hardware and software status of an AI server is provided, including an AI server and a fault diagnosis module based on a large AI model. The AI ​​server includes hardware boards and server software. The hardware boards include a motherboard, a switching board, a GPU board, and a fan power supply board. The fault diagnosis module is located on the switching board. The switching board is connected to the motherboard uplink and to the GPU board downlink. The fan power supply board is connected to the switching board. The motherboard, switching board, GPU board, and fan power supply board are all equipped with complex programmable logic devices (CPLDs). The server software is connected to a software information monitoring module. The fault diagnosis module contains an MCU chip and a TF card. The TF card and the MCU chip transmit information through the SDMMC bus. The motherboard is equipped with a controller BMC, which is connected to the MCU chip via an I2C bus. VR chips are located on the motherboard, GPU board, and fan power board.

[0007] Preferably, the GPU board also integrates an OAM module and a PHY Retimer chip. The OAM module is connected to the complex programmable logic device (CPLD) on the GPU board via a JTAG interface, and the PHY Retimer chip is connected to the CPLD on the GPU board via a UART interface.

[0008] Preferably, the switching board carries an SW switching chip, which is connected to a complex programmable logic device (CPLD) on the switching board via a UART interface. The SW switching chip is also connected to CPLDs on the GPU board and the fan power supply board via an I2C bus.

[0009] Preferably, the controller BMC connects to the MCU chip via the I2C bus and interacts with the server software via HTTP / HTTPS protocol and JSON data format.

[0010] On the other hand, a method for monitoring and diagnosing hardware and software information and faults in AI servers is provided, including the following steps: S1. The complex programmable logic device (CPLD) set on each hardware card board performs real-time monitoring and collection of the key hardware information of the corresponding hardware card, and the software information monitoring module performs real-time monitoring and collection of the main operating status data of the system software. S2. Process the data collected in S1; S3. Based on the data processed in S2, the fault diagnosis module based on the AI ​​large model mounted on the switching board diagnoses hardware and software faults. S4. Generate fault diagnosis and fault handling suggestions, and quickly locate the fault location.

[0011] Preferably, the server hardware and software operation logs obtained by the MCU chip are stored via a TF card; the controller BMC upgrades the MCU chip via the I2C bus and updates the fault diagnosis model in the MCU chip.

[0012] Preferably, the status of each hardware board is determined based on high level, low level, and voltage value. Specifically, a complex programmable logic device (CPLD) monitors and records the ENABLE signal, POWER GOOD signal, and voltage value of the VR chip. When the voltage value fluctuates beyond a set threshold, the CPLD records the abnormal status. When the ENABLE signal of the VR chip is high and the POWER GOOD signal becomes low, the CPLD records the abnormal status.

[0013] Preferably, for the GPU board, the Complex Programmable Logic Device (CPLD) on it obtains the operating status information of the OAM module through the JTAG interface. The CPLD performs data processing, extracts relevant log information, and records the abnormal state if an error is found. The CPLD on it also obtains the operating status information of the PHY Retimer chip through the UART interface. The CPLD performs data processing, extracts relevant log information, and records the abnormal state if an error is found.

[0014] As a preferred embodiment, for the motherboard, the controller BMC obtains CPU temperature information through the PECI bus and obtains the board status information monitored by the complex programmable logic device (CPLD) on the motherboard through the I2C bus. When the obtained information is inconsistent with the preset status, the complex programmable logic device (CPLD) will report the relevant information to the motherboard controller BMC through the I2C bus and report the information to the MCU of the fault handling module through the peripheral interface SPI.

[0015] Preferably, for the fan power supply board, the complex programmable logic device (CPLD) on the board can control the fan and obtain fan speed information. When the fan stops running abnormally, the CPLD records the abnormal state.

[0016] As a preferred embodiment, for the switching board, the complex programmable logic device (CPLD) on the switching board can monitor the status information of important devices on the board. It will also act as the I2C Master to collect the status information of the GPU board's CPLD and the fan power supply board's CPLD. This information will be reported to the motherboard controller (BMC) via the I2C bus and to the microcontroller (MCU) of the fault handling module via the peripheral interface (SPI).

[0017] Preferably, the software information monitoring module monitors the status of local service processes and system resource usage in real time, and then sends the monitored information to the fault diagnosis module for diagnosis. Specifically, the controller BMC obtains software information through RedFishAPI, and the microprogram controller MCU of the fault diagnosis module obtains the above software information from the controller BMC through the I2C bus and performs fault diagnosis.

[0018] Compared with existing technologies, this invention provides a method for monitoring and diagnosing hardware and software information and diagnosing faults in AI servers, which has the following beneficial effects: 1. Provides a hardware solution for real-time monitoring, recording, and diagnosis of the hardware and software operating status of computing nodes in AI rack servers. When a fault occurs in a certain board or system in the server, it can be located and recorded in a timely manner, providing hardware protection for the stable operation of the server.

[0019] 2. Add a fault diagnosis module to the key board of the AI ​​server computing node—the switch board. The core of this module is an MCU chip, which can be used to collect the server hardware and system operation logs and error logs. At the same time, a trained fault diagnosis model is embedded in the MCU chip. Based on inference, it can realize the rapid location of faults and generate preliminary solutions, realizing the functions of server software and hardware operation log recording and fault diagnosis.

[0020] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0021] Figure 1 The above is a flowchart of server fault diagnosis disclosed in patent number CN113886120A; Figure 2 This is a design block diagram of a software and hardware information monitoring and fault diagnosis method for AI servers according to the present invention; Figure 3 This is a hardware topology block diagram of a software and hardware information monitoring and fault diagnosis method for AI servers according to the present invention. Figure 4This is a software information acquisition structure block diagram of a software and hardware information monitoring and fault diagnosis method for AI servers according to the present invention. Figure 5 This is a block diagram of a lightweight AI large model structure for a software and hardware information monitoring and fault diagnosis method for AI servers according to the present invention. Figure 6 This is a fault diagnosis structure block diagram of a software and hardware information monitoring and fault diagnosis method for AI servers according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] See Figures 2-4 This invention provides a topology for monitoring the hardware and software status of an AI server, including an AI server and a fault diagnosis module based on a large AI model. The AI ​​server includes hardware boards and server software. The hardware boards include a motherboard, a switching board, a GPU board, and a fan power supply board. The fault diagnosis module is located on the switching board, which is connected to the motherboard uplink and the GPU board downlink. The fan power supply board is connected to the switching board. The motherboard, switching board, GPU board, and fan power supply board are all equipped with a Complex Programmable Logic Device (CPLD). The server software is connected to a software information monitoring module. The fault diagnosis module internally includes an MCU chip and a TF card. The TF card and the MCU chip transmit information via an SDMMC bus. The motherboard is equipped with a controller BMC, which is connected to the MCU chip via an I2C bus. The motherboard, GPU board, and fan power supply board are all equipped with VR chips.

[0024] Specifically, the GPU board also integrates an OAM module and a PHY Retimer chip. The OAM module is connected to the complex programmable logic device (CPLD) on the GPU board via a JTAG interface, and the PHY Retimer chip is connected to the complex programmable logic device (CPLD) on the GPU board via a UART interface.

[0025] Specifically, the switching board carries an SW switching chip, which is connected to a complex programmable logic device (CPLD) on the switching board via a UART interface. The SW switching chip is also connected to CPLDs on the GPU board and the fan power supply board via an I2C bus.

[0026] Specifically, the controller BMC connects to the MCU chip via the I2C bus and interacts with the server software via HTTP / HTTPS protocol and JSON data format.

[0027] This invention provides a method for monitoring and diagnosing hardware and software information and diagnosing faults in AI servers, comprising the following steps: S1. The complex programmable logic device (CPLD) set on each hardware card board performs real-time monitoring and collection of the key hardware information of the corresponding hardware card, and the software information monitoring module performs real-time monitoring and collection of the main operating status data of the system software. S2. Process the data collected in S1; S3. Based on the data processed in S2, the fault diagnosis module based on the AI ​​large model mounted on the switching board diagnoses hardware and software faults. S4. Generate fault diagnosis and fault handling suggestions, and quickly locate the fault location.

[0028] Specifically, the server's hardware and software operation logs obtained by the MCU chip are stored via a TF card; the controller BMC upgrades the MCU chip via the I2C bus and updates the fault diagnosis model in the MCU chip.

[0029] Specifically, the status of each hardware board is determined based on high level, low level, and voltage value. Specifically, the ENABLE signal, POWER GOOD signal, and voltage value of the VR chip are monitored and recorded by a complex programmable logic device (CPLD). When the voltage value fluctuates beyond the set threshold, the CPLD records the abnormal state. When the ENABLE signal of the VR chip is high and the POWER GOOD signal becomes low, the CPLD records the abnormal state.

[0030] Specifically, for the GPU board, the Complex Programmable Logic Device (CPLD) on it obtains the operating status information of the OAM module through the JTAG interface. The CPLD performs data processing, extracts relevant log information, and records the abnormal state if an error is found. The CPLD on it also obtains the operating status information of the PHY Retimer chip through the UART interface. The CPLD performs data processing, extracts relevant log information, and records the abnormal state if an error is found.

[0031] Specifically, for the motherboard, the controller BMC obtains CPU temperature information through the PECI bus and obtains the board status information monitored by the complex programmable logic device (CPLD) on the motherboard through the I2C bus. When the obtained information is inconsistent with the preset status, the complex programmable logic device (CPLD) will report the relevant information to the motherboard controller BMC through the I2C bus and report the information to the MCU of the fault handling module through the peripheral interface SPI.

[0032] Specifically, for the fan power supply board, the complex programmable logic device (CPLD) on the board can control the fan and obtain fan speed information. When the fan stops running abnormally, the CPLD records the abnormal state.

[0033] Specifically, for the switching board, the complex programmable logic device (CPLD) on the switching board can monitor the status information of important devices on the board. It also acts as the I2C Master to collect the status information of the GPU board's CPLD and the fan power supply board's CPLD. This information is then reported to the motherboard controller (BMC) via the I2C bus and to the microcontroller controller (MCU) of the fault handling module via the peripheral interface (SPI).

[0034] Specifically, the software information monitoring module monitors the status of local service processes and system resource usage in real time, and then sends the monitored information to the fault diagnosis module for diagnosis; specifically, the controller BMC obtains software information through the RedFish API, and the microprogram controller MCU of the fault diagnosis module obtains the above software information from the controller BMC through the I2C bus and performs fault diagnosis.

[0035] To better illustrate the control logic, the following specific explanation is provided: 1. Implementation of hardware information monitoring function AI servers have a relatively large number and variety of boards, and some major boards involve CPLDs. To achieve more stable, reliable, and cost-effective monitoring of server hardware information, based on the concept of decoupling, the hardware information of each board is monitored by its own CPLD, and then the monitored information is processed and reported to the fault diagnosis module. The main boards for which the hardware monitoring function needs to be implemented in this invention include motherboards, switching boards, GPU boards, fan power supply boards, etc.

[0036] like Figure 3The diagram shows the hardware topology for implementing hardware status information monitoring in this invention. In this structure, the switching board serves as a key board connecting the motherboard upstream and the GPU board downstream. A fault diagnosis module is added to the switching board. The fault diagnosis module contains an MCU chip, a TF card, and related hardware circuitry. The TF card and the MCU chip transmit information via the SDMMC bus. The TF card is used to store the server software and hardware operation logs acquired by the MCU. The BMC can upgrade the MCU via I2C to update the fault diagnosis model in the MCU.

[0037] This AI rack server uses a centralized power supply method. Power is supplied to the VR chip on the fan power board via the Power Shelf and Power Clip. The VR (Voltage Regulator) chip is a voltage converter chip used to convert the voltage input to the board into the voltage required by the board's components. The CPLD on this board can monitor the output voltage of the VR chip through the ADC module. The CPLD obtains the output voltage of the VR chip through the GPIO interface, processes it after internal ADC (Analog-to-Digital Converter) module, and performs logical judgment. When the voltage fluctuation exceeds the set threshold, an abnormal state is recorded. At the same time, the CPLD on this board can control the ENABLE signal of the VR chip and read its POWER GOOD signal through the GPIO interface. When the ENABLE signal of the VR chip is high and the POWER GOOD signal is high, the CPLD will detect the abnormal state. When the GOOD signal goes low, the CPLD records its abnormal state. In addition, the fan is one of the main components on the fan power supply board, and controlling and monitoring the fan's operating status is one of the main functions of this board. The CPLD on this board can directly or indirectly control the fan and obtain fan speed information. When the fan speed fluctuates greatly, the difference between the controlled speed and the read speed is large, or the fan stops abnormally, the CPLD records the abnormal state. The board is equipped with a temperature sensor to monitor the outlet temperature, which can be used as a basis for fan speed adjustment.

[0038] The GPU board serves as the carrier of the OAM module. Key components on this board include the OAM module, PHY Retimer chip, VR chip, and clock chip. The CPLD on the GPU board can obtain OAM's operational status information via the JTAG interface. After data processing, the CPLD extracts relevant log information, and records any errors or abnormal states. Simultaneously, the CPLD can obtain the PHY Retimer chip's operational status information via the UART interface. After data processing, the CPLD extracts relevant log information, and records any errors or abnormal states. Furthermore, the VR chip on the GPU board converts the 54V power input to the board into the voltage required by the onboard components. The CPLD monitors and records the VR chip's ENABLE signal, POWER GOOD signal, and voltage values, following the same monitoring and recording scheme as described above. When an abnormal state occurs, the CPLD also records it.

[0039] As a key board in the AI ​​server architecture, the switching board in this invention not only houses the switching chip and connects the motherboard and GPU board, but also incorporates a fault handling module. The CPLD on the switching board monitors the status information of critical components on the board (such as the VR chip, with monitoring logic as described above; and the SW switching chip, whose operating status information can be obtained via UART, processed by the CPLD, and relevant log information extracted; if errors are found, their abnormal states are recorded). Furthermore, it acts as an I2C Master, collecting status information from the GPU board CPLD and the fan power supply board CPLD. This information is then reported to the motherboard BMC via I2C and to the fault handling module's MCU via SPI.

[0040] The BMC on the motherboard obtains the CPU core temperature directly or indirectly through the PECI bus. At the same time, the BMC can obtain the board status information monitored by the motherboard CPLD through I2C. The CPLD on the motherboard can monitor the output voltage of the VR chip on the board and the status of its ENABLE and POWER GOOD signals through the ADC module. Its monitoring logic is the same as described above. The motherboard CPLD can also obtain information such as THERMTRIP, MEMOHT, THROTTLE, PWRGD, and RESET of the CPU and DIMM components through the GPIO interface. When the obtained information is inconsistent with the preset status, the CPLD can report the relevant information to the motherboard BMC through I2C and to the MCU of the fault handling module through SPI.

[0041] 2. Implementation of software information monitoring function The software information monitoring module is used to monitor the running status of server system software. Its main function is to monitor the status of local service processes and system resource usage in real time, and then send the monitored information to the fault diagnosis module for diagnosis. The monitored software information includes: CPU usage, memory usage, network traffic, core system service status, important application service status, and the status of processes that need to be monitored as configured by the user.

[0042] Figure 4 As shown, the BMC can obtain the monitored software operating status through the RedFish API. Therefore, the MCU of the fault diagnosis module only needs to obtain the above information from the BMC via I2C. The RedFish API is a hardware management standard based on RESTful API, using HTTP / HTTPS protocol and JSON data format. The BMC, as the RedFish server, exposes system information through a predefined resource model.

[0043] 3. Implementation of fault diagnosis function based on AI large model like Figure 5As shown, the chip-cooperative architecture clearly defines the roles of the MCU and BMC chips: the Renesas RA8 series Arm Cortex-M55 core MCU chip serves as the core processing unit, paired with an Arm Cortex-A55 based BMC chip. Collaboration between heterogeneous chips is achieved through I2C bus communication. The BMC chip handles some management tasks, while the MCU focuses on core algorithm computation, improving overall operating efficiency. A TF card is used to connect to the MCU via the SDMMC bus to store lightweight large models after quantization fine-tuning. This method flexibly expands storage capacity, facilitates model updates and maintenance, and enables the system to easily load different lightweight models to meet diverse application needs. Large models, after quantization fine-tuning, use internal algorithms to predict the remaining hard drive lifespan. Weibull distribution fitting is used: it can run efficiently on the Cortex-M55 using C language without hardware acceleration. It can quickly and accurately fit data distributions, providing a foundation for subsequent analysis, and is suitable for various data scenarios, especially for resource-constrained embedded environments. Transformer timing analysis: efficient operation is achieved through multi-dimensional optimization. In terms of model size compression, INT8 quantization is used instead of FP32 / FP16 to reduce memory usage and computational load, while Helium vector extension is used to accelerate computation. Regarding algorithm optimization, self-attention calculation is simplified (replacing global attention with "local attention"), redundant modules are removed, and fixed-point arithmetic is adopted, significantly reducing computational power requirements. In terms of hardware utilization, ML acceleration libraries such as CMSIS-NN for Cortex-M55 are called to optimize matrix operations and improve inference speed. The overall advantages are threefold: 1. Efficient resource utilization: From chip collaboration to algorithm optimization, the system achieves efficient operation of complex models and algorithms under limited hardware resources through reasonable architecture design and algorithm improvements, reducing hardware resource requirements while ensuring performance. 2. Application flexibility: Modular design allows for independent optimization and replacement of each part. Whether it's updating lightweight models or improving algorithm modules, it can be easily implemented, improving the system's adaptability and flexibility in different application scenarios.

[0044] like Figure 6 The diagram shown is a topology diagram of the fault diagnosis function based on the AI ​​large model of the present invention. The implementation of the function includes a data preprocessing layer for cleaning raw data after information collection, feature engineering and standardization, an AI analysis layer for multi-dimensional anomaly detection, root cause reasoning and prediction, and a decision layer for generating executable disposal strategies and risk control.

[0045] The technical implementation methods of the data preprocessing layer include data cleaning, feature engineering, data standardization, and multimodal fusion. This invention employs corresponding data preprocessing methods based on the different characteristics of the acquired data. For example, collected temperature information and fan speed information may contain jitter noise, so denoising processing is performed. The technical implementation methods of the AI ​​analysis layer include anomaly detection modules, root cause analysis modules, prediction modules, and multimodal fusion architectures. The anomaly detection algorithms used in hardware and software differ. For hardware anomalies, an LSTM-NDT (nonlinear dynamic threshold) model can be used to detect sensor data that deviates from the baseline. Running LSTM-NDT on an MCU chip requires reasonable optimization of the LSTM-NDT model. This involves reducing computation and memory usage through quantization (converting floating-point parameters to fixed-point numbers), model pruning to reduce size, and code optimization, striking a balance between computational speed and model accuracy. For software anomalies, LogBERT-based log pattern recognition can be used to identify new anomaly patterns. Similarly, LogBERT quantization is required: converting model parameters from FP32 to INT8 low-precision data type significantly reduces memory usage and computation. Unimportant connections or parameters are removed from the model. Pruning reduces the number of model parameters and computational complexity. The model structure is simplified by reducing the number of Transformer layers and the number of heads in the multi-head self-attention mechanism. Simultaneously, the computational process is optimized by using a more efficient matrix operation library, such as the CMSIS-NN library for Cortex-M55, to accelerate model inference. The root cause analysis module employs an interpretable multi-hop inference model based on knowledge graph embedding. Entities and relationships from the knowledge graph are embedded into a low-dimensional vector space, and multi-hop inference is performed through vector computation and model learning of the latent features of entities and relationships. The prediction module's main function is to predict the remaining lifespan of devices and fault propagation, such as using Weibull distribution fitting + Transformer time series analysis to predict the remaining lifespan of hard drives. The decision layer introduces input features, decision logic, and output actions to build a hierarchical decision engine. Based on different input features and the established decision logic, reasonable diagnostic results can be output.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A topology for monitoring the hardware and software status of an AI server, comprising an AI server and a fault diagnosis module based on a large AI model, characterized in that: The AI ​​server includes hardware boards and server software. The hardware boards include a motherboard, a switching board, a GPU board, and a fan power supply board. The fault diagnosis module is set on the switching board. The switching board is connected to the motherboard uplink and the GPU board downlink. The fan power supply board is connected to the switching board. The motherboard, switching board, GPU board, and fan power supply board are all equipped with complex programmable logic devices (CPLDs). The server software is connected to a software information monitoring module. The fault diagnosis module contains an MCU chip and a TF card. The TF card and the MCU chip transmit information through the SDMMC bus. The motherboard is equipped with a controller BMC, which is connected to the MCU chip via an I2C bus. VR chips are located on the motherboard, GPU board, and fan power board.

2. The topology for monitoring the hardware and software status of an AI server as described in claim 1, characterized in that: The GPU board also integrates an OAM module and a PHY Retimer chip. The OAM module is connected to the complex programmable logic device (CPLD) on the GPU board via a JTAG interface, and the PHY Retimer chip is connected to the complex programmable logic device (CPLD) on the GPU board via a UART interface.

3. The topology for monitoring the hardware and software status of an AI server as described in claim 1, characterized in that: The switching board carries an SW switching chip, which is connected to a complex programmable logic device (CPLD) on the switching board via a UART interface. The SW switching chip is also connected to CPLDs on the GPU board and the fan power supply board via an I2C bus.

4. The topology for monitoring the hardware and software status of an AI server as described in claim 1, characterized in that: The BMC controller connects to the MCU chip via the I2C bus and interacts with the server software via HTTP / HTTPS protocol and JSON data format.

5. A method for monitoring and diagnosing hardware and software information and faults in AI servers, characterized in that: Includes the following steps: S1. The complex programmable logic device (CPLD) set on each hardware card board performs real-time monitoring and collection of the key hardware information of the corresponding hardware card, and the software information monitoring module performs real-time monitoring and collection of the main operating status data of the system software. S2. Process the data collected in S1; S3. Based on the data processed in S2, the fault diagnosis module based on the AI ​​large model mounted on the switching board diagnoses hardware and software faults. S4. Generate fault diagnosis and fault handling suggestions, and quickly locate the fault location.

6. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: The server's hardware and software operation logs obtained by the MCU chip are stored via a TF card; the BMC controller upgrades the MCU chip via the I2C bus and updates the fault diagnosis model in the MCU chip.

7. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: The status of each hardware card is determined by high level, low level and voltage value; Specifically, the ENABLE signal, POWER GOOD signal, and voltage value of the VR chip are monitored and recorded by a complex programmable logic device (CPLD). When the voltage value fluctuates beyond the set threshold, the CPLD records the abnormal state. When the ENABLE signal of the VR chip is high and the POWER GOOD signal becomes low, the CPLD records the abnormal state.

8. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: For the GPU board, the Complex Programmable Logic Device (CPLD) on it obtains the operating status information of the OAM module through the JTAG interface. The CPLD processes the data, extracts relevant log information, and records any errors. The CPLD on it also obtains the operating status information of the PHY Retimer chip through the UART interface. The CPLD processes the data, extracts relevant log information, and records any errors.

9. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: For the motherboard, the controller BMC obtains CPU temperature information through the PECI bus and obtains the board status information monitored by the complex programmable logic device (CPLD) on the motherboard through the I2C bus. When the obtained information is inconsistent with the preset status, the complex programmable logic device (CPLD) will report the relevant information to the motherboard controller BMC through the I2C bus and report the information to the MCU of the fault handling module through the peripheral interface SPI.

10. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: For the fan power supply board, the complex programmable logic device (CPLD) on the board can control the fan and obtain fan speed information. When the fan stops running abnormally, the CPLD records the abnormal state.

11. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: For the switching board, the complex programmable logic device (CPLD) on the switching board can monitor the status information of important devices on the board. It also acts as the I2C Master to collect the status information of the GPU board's CPLD and the fan power supply board's CPLD. This information is then reported to the motherboard controller (BMC) via the I2C bus and to the microcontroller controller (MCU) of the fault handling module via the peripheral interface (SPI).

12. The method for monitoring and diagnosing hardware and software information and faults in an AI server as described in claim 5, characterized in that: The software information monitoring module monitors the status of local service processes and system resource usage in real time, and then sends the monitored information to the fault diagnosis module for diagnosis. Specifically, the controller BMC obtains software information through the RedFish API, and the microprogram controller MCU of the fault diagnosis module obtains the above software information from the controller BMC through the I2C bus and performs fault diagnosis.

Images