Server link fault location system, method, electronic equipment and storage medium

Abstract

This application discloses a server link fault location system, method, electronic device, and storage medium, relating to the field of server link testing technology. It collects physical layer and protocol layer data from data acquisition points using a data acquisition component, and controls a testing component to perform server link testing. The testing component includes a control module and an analysis module. The control module sends test commands to the server link. When a server link fault is detected, the analysis module obtains physical layer and protocol layer data from each data acquisition point from the acquisition component. Based on the acquired data, it determines the fault location and / or fault type of the server link, achieving automated link quality assessment and determining the fault location and fault type. This solves the problem that related technologies struggle to accurately locate fault locations and distinguish between hardware link faults and software errors through single stress tests, requiring manual fault location and resulting in low troubleshooting efficiency. Therefore, it achieves the technical effect of improving troubleshooting efficiency.

Description

Technical Field

[0001] This application relates to the field of server link testing technology, and in particular to a server link fault location system, method, electronic device and storage medium. Background Technology

[0002] AI (Artificial Intelligence) servers, as core devices for high-performance computing, are widely used in fields such as artificial intelligence, scientific computing, and graphics rendering. They consist of modules such as GPUs (Graphics Processing Units), CPUs (Central Processing Units), memory, storage, and networks, relying on the collaborative work of these components to achieve efficient parallel computing. Among these, the stability of the GPU link directly affects computational efficiency, distributed training stability, and data I / O (Input / Output) performance, and is crucial for ensuring business continuity. Therefore, the quality of the AI ​​server link plays a pivotal role in business operations.

[0003] However, traditional GPU fault location methods rely on a single stress test result, which cannot distinguish between hardware link failures and software errors, nor can they distinguish between abnormal links in the CPU-server motherboard-SW (Switch Board)-GPU board. Manual replacement of hardware segment by segment is required for troubleshooting, resulting in low troubleshooting efficiency and a high risk of misjudgment, making it difficult to meet the operation and maintenance needs of high-reliability AI server systems. Summary of the Invention

[0004] This application provides a server link fault location system, method, electronic device, and storage medium to at least solve the technical problems in the related art where it is difficult to accurately locate the fault location and distinguish between hardware link faults and software errors through a single stress test, and where manual fault location is required, resulting in low troubleshooting efficiency.

[0005] This application provides a server link fault location system. The server link includes a central processing unit, a server motherboard, a switching component, and a graphics processing component. Multiple data acquisition points are set on the server link. The system includes: an acquisition component, which includes multiple probe arrays located at the data acquisition points. The probe arrays include physical layer and protocol layer data. The physical layer acquires physical layer data from the data acquisition points, and the protocol layer acquires protocol layer data from the data acquisition points. A testing component, which includes a control module and an analysis module. The control module sends test commands to the server link. When a server link fault is detected, the analysis module obtains the physical layer and protocol layer data of each data acquisition point from the acquisition component. Based on the physical layer and protocol layer data of each data acquisition point, the system determines the fault location and / or fault type of the server link.

[0006] This application also provides a server link fault location method, which is applied to the test component of the above-mentioned server link fault location system. The method includes: issuing test instructions to the server link; when a server link fault is detected, obtaining physical layer data and protocol layer data of each data acquisition point from the acquisition component; and determining the fault location and / or fault type of the server link based on the target physical layer data and target protocol layer data of each data acquisition point.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described server link fault location method.

[0008] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described server link fault location method.

[0009] This application enables the acquisition of physical layer and protocol layer data from data acquisition points via an acquisition component, and the control of a testing component to perform server link testing. The testing component includes a control module and an analysis module. The control module sends test commands to the server link. When a server link failure is detected, the analysis module obtains physical layer and protocol layer data from each data acquisition point from the acquisition component. Based on the obtained data, it determines the location and / or type of the server link failure, thus achieving automated link quality assessment and determining the location and type of failure. Therefore, it can solve the problem that related technologies cannot accurately locate the failure location or distinguish between hardware link failures and software errors through a single stress test, and the low troubleshooting efficiency caused by the need for manual fault location, thereby achieving the technical effect of improving troubleshooting efficiency. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the structure of a server link fault location system provided in an embodiment of this application;

[0012] Figure 2 A structural diagram of the main hardware links of a server provided in one embodiment of this application;

[0013] Figure 3 This is a structural diagram of a server link fault location system provided in one embodiment of this application;

[0014] Figure 4 A flowchart illustrating the processing procedure of a server link fault location system provided in one embodiment of this application;

[0015] Figure 5 A flowchart illustrating a server link fault location method provided in an embodiment of this application;

[0016] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1This is a schematic diagram of the server link fault location system provided in an embodiment of this application. Figure 1 As shown, the system 10 includes: a data acquisition component 101, multiple probe arrays 1011, a testing component 102, a control module 1021, and an analysis module 1022; the server link 20 includes a central processing unit 201, a server motherboard 202, a switching component 203, and a graphics processing component 204.

[0021] First, the server link 20, the object of this application embodiment, will be specifically described. In the server link 20, the central processing unit 201 is the processing core of the server, and its main functions are task scheduling and management, logic processing, data preprocessing and post-processing, load sharing and hybrid computing, and fault tolerance. The server motherboard 202 is the core platform connecting all hardware components. Its design directly affects the system's performance, scalability, and stability, and needs to be optimized for high-performance computing, massive parallel processing, and multi-accelerator collaboration. The switching component 203 is the SW board, which is used to realize high-speed interconnection and communication, and is responsible for managing data flow, optimizing communication efficiency, and reducing latency. The graphics processing component 204 is the GPU board, which is the core hardware module that carries the graphics processor and is responsible for executing massive parallel computing tasks. The server link 20 includes the links connecting the central processing unit 201, the server motherboard 202, the switching component 203, and the graphics processing component 204. The modules are connected according to actual needs through standards such as PCIe (Peripheral Component Interconnect Express) and OCulink (Optical Copper Link).

[0022] Next, the embodiments of this application will be described in detail. In this embodiment, the acquisition component 101 includes multiple probe arrays 1011. The probe arrays 1011 are set at the data acquisition points. The probe arrays 1011 include a physical layer and a protocol layer. The physical layer acquires the physical layer data of the data acquisition points, and the protocol layer acquires the protocol layer data of the data acquisition points. The test component 102 includes a control module 1021 and an analysis module 1022. The control module 1021 sends test commands to the server link 20. When a fault is detected in the server link 20, the analysis module 1022 obtains the physical layer data and protocol layer data of each data acquisition point from the acquisition component 101. Based on the physical layer data and protocol layer data of each data acquisition point, the fault location and / or fault type of the server link 20 are determined.

[0023] The physical layer data of probe array 1011 and data acquisition points will be described in detail below and will not be repeated here. The protocol layer data of data acquisition points includes packet type (such as TLP (Transaction Layer Packet) and DLLP (Data Link Layer Packet)), packet sequence number, retransmission count, etc. The test command issued by control module 1021 is a graphics processor stress test command. After receiving the test command, server link 20 triggers a stress test on the graphics processor carried on graphics processing component 204, such as 3DMark (a widely used benchmark software that can be used to evaluate the performance of graphics processors, including various test scenarios to simulate different types of workloads) stress test mode or NVIDIA-SIM (a command-line utility that provides management and monitoring functions for graphics processors) stress test command. The fault location of server link 20 refers to the location of the faulty link segment of server link 20. The fault types of server link 20 include hardware link faults and software errors.

[0024] It is understood that the acquisition component 101 in this embodiment includes multiple probe arrays 1011, which are set at different data acquisition points. Each probe array 1011 consists of a physical layer and a protocol layer. The physical layer is used to acquire physical layer data of the corresponding data acquisition point, and the protocol layer is used to acquire protocol layer data of the corresponding data acquisition point. The test component 102 includes a control module 1021 and an analysis module 1022. When the server link 20 needs to be tested, the control module 1021 sends a test command to the server link 20. The test command is a graphics processor stress test command, such as the 3DMark stress test mode or the NVIDIA-SMI stress test command. After receiving the test command, the server link 20 triggers a stress test on the graphics processor. The control module 1021 continuously monitors the server link 20. When a fault is detected in the server link 20, the analysis module 1022 obtains the physical layer data and protocol layer data of each data acquisition point from the acquisition component 101, and determines the specific location and / or fault type of the faulty link segment of the server link 20 based on these data.

[0025] In this embodiment of the application, the data acquisition points include multiple acquisition points of the link between the central processing unit 201 and the server motherboard 202, the link between the server motherboard 202 and the switching component 203, the link between the switching component 203 and the graphics processing component 204, and the acquisition point at the power supply port of the graphics processing component 204.

[0026] The data acquisition points can be set in the PCIe slot of the server motherboard 202, the input / output ports of the switching component 203, the gold fingers of the graphics processing component 204, and the power supply detection points of the graphics processing component 204. The gold fingers of the graphics processing component 204 are a series of contact points on the edge of the graphics processing component 204 circuit board plated with a layer of conductive material. These contact points are designed to be inserted into the expansion slots (such as PCIe slots) on the server motherboard 202, thereby realizing data transmission and power supply between the graphics processing component 204 and other parts of the computer.

[0027] It is understood that the data acquisition points in this application embodiment include multiple locations, specifically: acquisition points of the link between the central processing unit 201 and the server motherboard 202, acquisition points of the link between the server motherboard 202 and the switching component 203, acquisition points of the link between the switching component 203 and the graphics processing component 204, and acquisition points at the power supply port of the graphics processing component 204, etc. These data acquisition points can be respectively set at the PCIe slot of the server motherboard 202, the input / output port of the switching component 203, the gold fingers of the graphics processing component 204, and the power supply detection point of the graphics processing component 204.

[0028] In this embodiment, the physical layer of the probe array 1011 includes a differential probe, an impedance test probe, and a power noise probe. The high-speed differential probe is connected to the acquisition points of the link between the central processing unit 201 and the server motherboard 202, the link between the server motherboard 202 and the switching component 203, and the link between the switching component 203 and the graphics processing component 204, to measure the first bandwidth data of each link segment of the server link 20. The impedance test probe is connected to the acquisition points of the link between the server motherboard 202 and the switching component 203 and the link between the switching component 203 and the graphics processing component 204 via an RF coaxial connector to measure the time-domain reflectometer impedance resolution data. The power noise probe is connected to the acquisition point at the power supply port of the graphics processing component 204 to measure the second bandwidth data and the noise floor data.

[0029] Among them, a high-speed differential probe is a test device used to measure high-speed signals, which can support a very high frequency range and can accurately capture first bandwidth data through its high bandwidth capability; an impedance test probe is a tool designed to be connected to a circuit or device and can be used to detect its impedance characteristics; an RF coaxial connector is a connection device used to transmit RF signals, which has good shielding performance and can effectively reduce the influence of external electromagnetic interference. It is widely used in occasions that require processing high-frequency signals. The RF coaxial connector in this application embodiment can use an SMA connector (Sub-Miniature version A RF coaxial connector); the time domain reflectometry impedance resolution data, i.e., TDR (Time Domain Reflectometry) impedance resolution, can distinguish the minimum distance or minimum impedance change between two adjacent impedance abrupt change points; the power supply noise probe is connected to the acquisition point at the power supply port of the graphics processing component 204 in parallel with the power supply port of the graphics processing component 204.

[0030] It is understood that the physical layer of the probe array 1011 in this embodiment includes a high-speed differential probe, an impedance test probe, and a power noise probe. The high-speed differential probe is connected to the acquisition points of the link between the central processing unit 201 and the server motherboard 202, the link between the server motherboard 202 and the switching component 203, and the link between the switching component 203 and the graphics processing component 204, and is used to measure the first bandwidth data of each link segment of the server link 20. The impedance test probe is connected to the acquisition points of the link between the server motherboard 202 and the switching component 203 and the link between the switching component 203 and the graphics processing component 204 through an RF coaxial connector, and is used to measure the time domain reflectometer impedance resolution data, which represents the minimum distance or minimum impedance change that can distinguish two adjacent impedance abrupt changes. The power noise probe is connected to the acquisition point at the power supply port of the graphics processing component 204 in parallel with the power supply port, and is used to measure the second bandwidth data and the noise floor data. The first bandwidth data, the time domain reflectometer impedance resolution data, the second bandwidth data, and the noise floor data constitute the required physical layer data.

[0031] In this embodiment, a time synchronization clock is provided on the physical layer and the protocol layer to align the data acquisition time of the physical layer with that of the protocol layer.

[0032] After aligning the physical layer data with the protocol layer data acquisition time based on the time synchronization clock, it is also necessary to check whether the time error between the aligned physical layer data and the protocol layer data acquisition time is less than or equal to the preset time error threshold. If it is greater than the preset time error threshold, time alignment needs to be performed again. The time error threshold is set according to the actual needs and is not specifically limited here.

[0033] It is understood that both the physical layer and the protocol layer in this application embodiment are equipped with time synchronization clocks to align the data acquisition time of the physical layer data with that of the protocol layer. Furthermore, after time alignment is completed based on the time synchronization clock, it is necessary to further check whether the time error between the aligned physical layer data and the protocol layer data is less than or equal to a preset time error threshold (e.g., 100ns). If the time error is greater than the time error threshold, the time alignment operation needs to be re-executed to ensure that the physical layer and protocol layer data remain highly synchronized in the time dimension.

[0034] In this embodiment, the control module 1021 sends a fault command to the acquisition component 101; the acquisition component 101 responds to the fault command and acquires physical layer data and protocol layer data within a target time period, wherein the target time period is the time period before the fault is sent.

[0035] The fault command is issued when the control module 1021 detects a fault in the server link 20. The target time period is selected by setting a target time period length. The target time period length is set according to actual needs and is not specifically limited here. For example, if the target time period length is 30 seconds, when the acquisition component 101 responds to the fault command and selects physical layer data and protocol layer data, then the target time period is 30 seconds back from the time the fault was sent.

[0036] It should be noted that the control module 1021 monitors server link 20. When it detects a failure of the graphics processor in the fixed slot, it locates the abnormal bus identifier of the device in server link 20. The bus identifier is the bus ID (Identifier). When the bus identifier is abnormal, it is determined that server link 20 is faulty. The abnormal bus identifier includes: bus identifier jump, upstream node loss, device type code change, etc. The bus identifier jump means that the originally stable bus identifier suddenly changes, indicating that there may be a hardware failure, link instability or configuration error. The upstream node loss means that the device cannot communicate through its directly connected superior node. The device type code change means that the control module 1021 has identified a different type or function of the device, even if no actual hardware has been modified. It is usually caused by firmware corruption, accidental modification of configuration registers or driver errors.

[0037] It is understood that when the control module 1021 detects an abnormality in the device bus identifier in the server link 20, it determines that the server link 20 has failed and sends a fault command to the acquisition component 101. After the acquisition component 101 responds to the fault command, it extracts a time period of the preset target time period length before the time of the fault as the target time period, for example, the preset target time period length is 30 seconds, that is, it counts back 30 seconds from the time of the fault as the target time period. At the same time, it acquires the physical layer data and protocol layer data within the target time period, and uses the physical layer data and protocol layer data within the target time period as the basis for analysis by the analysis module 1022.

[0038] In this embodiment of the application, the test component 102 further includes a display, on which an interactive interface is provided, displaying the topology diagram and eye diagram of the server link 20; the analysis module 1022 renders the topology diagram and eye diagram of the server link 20 based on the physical layer data and protocol layer data of the data acquisition points, and annotates the topology diagram and eye diagram of the server link 20 based on the fault location and fault type.

[0039] The topology diagram of server link 20 depicts the relationship between all hardware devices and their connections from the central processing unit 201 to the graphics processing unit 204 in server link 20; the eye diagram is obtained through physical layer data and is formed by superimposing multiple cycles of signal waveforms on a graph to form an "eye" shape pattern, thereby providing information about signal quality; the topology diagram and eye diagram of server link 20 are marked based on fault location and fault type, which means that the server link segment at the fault location is marked with a different color on the topology diagram. The different colors are different from the colors displayed in the topology diagram when the link is in normal condition, thereby realizing the differentiation of faulty link segments.

[0040] It is understood that the test component 102 in this embodiment of the application also includes a display with an interactive interface for displaying the topology diagram and eye diagram of the server link 20. The topology diagram and eye diagram are obtained by the analysis module 1022 analyzing the physical layer data and protocol layer data of the data acquisition points. The analysis module 1022 annotates the topology diagram of the server link 20 based on the location and type of the fault, and finally displays it intuitively on the display. Specifically, the topology diagram of the server link 20 depicts in detail the relationship between all hardware devices and their connection methods from the central processing unit 201 to the graphics processing component 204; the eye diagram is generated by superimposing multiple cycles of signal waveforms to provide information about signal quality. When a fault is detected, the analysis module 1022 highlights the server link segment at the fault location on the topology diagram with a color different from that of the normal link, so that testers can quickly identify the problematic link segment.

[0041] According to the embodiments of this application, physical layer data and protocol layer data of data acquisition points can be collected by the acquisition component, and the test component can be controlled to perform server link testing. The test component includes a control module and an analysis module. The control module sends test commands to the server link. When a server link failure is detected, the analysis module obtains physical layer data and protocol layer data of each data acquisition point from the acquisition component. Based on the obtained data, the fault location and / or fault type of the server link are determined, realizing automated link quality assessment and determining the fault location and fault type, thus achieving the technical effect of improving troubleshooting efficiency.

[0042] The server link fault location system will be further described below through a specific embodiment.

[0043] First, the main hardware components of the server in this embodiment will be described in detail, such as... Figure 2 As shown, the server in this embodiment is an AI server, and its main hardware components include a central processing unit (CPU), a server motherboard, a switching board (SW), and a graphics processing unit (GPU). Specifically:

[0044] The central processing unit (CPU) is the brain of a server, and its main functions include task scheduling and management, logical processing, data preprocessing and post-processing, load sharing and hybrid computing, and fault tolerance.

[0045] The server motherboard is the core platform connecting all hardware components, and its design directly impacts the system's performance, scalability, and stability. Compared to ordinary server motherboards, AI server motherboards need to be optimized for HPC (High-Performance Computing), massively parallel processing, and multi-accelerator collaboration. As the core carrier of high-performance computing, the AI ​​server motherboard must strike a balance between multi-accelerator support, high-bandwidth interconnects, reliability, and scalability.

[0046] The switching component, or SW board, is a key component for enabling high-speed interconnection and communication, especially in multi-GPU / multi-accelerator scenarios. It is responsible for managing data flow, optimizing communication efficiency, and reducing latency. The SW board is a crucial hub for high-performance computing in AI servers, and its design directly impacts the collaborative efficiency of multiple GPUs / multi-accelerators.

[0047] The graphics processing unit, or GPU board, is the core hardware module that houses the graphics processing unit (GPU) or a dedicated AI accelerator. It is responsible for performing large-scale parallel computing tasks (such as deep learning training / inference, scientific computing, etc.). The core functions of the GPU board mainly include parallel computing acceleration, multi-GPU collaborative expansion, high-bandwidth memory support, and energy efficiency and thermal management. The GPU board is the core carrier of AI computing power, and its design revolves around high-parallel computing, high-bandwidth interconnects, and reliable operation.

[0048] PCIe is a high-speed serial computer expansion bus standard. The CPU's built-in PCIe controller manages device enumeration and address allocation. Similarly, PCIe connects various boards for communication services; for example, in this embodiment, the SW board and GPU board are connected using a high-speed PCIe interface. PCIe is the backbone bus connecting computing, storage, and networking in AI servers, and its bandwidth and latency directly affect the efficiency of multi-GPU collaboration. Figure 2 In this context, PCIe ×16 is a PCIe link width configuration, indicating that the PCIe link consists of 16 physical channels. PCIe Gen4 is the fourth generation peripheral component interconnect fast bus (PCI Express Generation 4).

[0049] OCulink is a high-speed external expansion interface standard based on the PCIe protocol, connected via copper or fiber optic cables. It is primarily used to address the limitations of traditional interfaces (such as USB) in terms of bandwidth and latency.

[0050] The main components of this embodiment are as follows: Figure 3 As shown, it includes:

[0051] Control module: Its main function is to trigger GPU stress tests, such as 3DMark stress test mode or NVIDIA-SIM stress test, and to monitor the server topology model, i.e., the bus ID-physical link mapping table.

[0052] Acquisition components: multi-level link probes, including motherboard PCIe slot probes / SW board input / output port probes / GPU board gold finger probes and GPU board power supply test points, etc. The main component is a PCIe bus probe array, including physical layer and protocol layer. The physical layer measurement probes mainly include: (1) high-speed differential probes for measuring bandwidth, soldered to PCIe Lane test points; (2) impedance test probes for measuring TDR resolution, connected via SMA connectors (between motherboard, SW board, and GPU); (3) power supply noise probes for measuring bandwidth and noise floor limits, connected in parallel to the GPU board 12V power supply line. The physical layer measures impedance / eye diagram, and the protocol layer parses TLP packets, including link training failure, DLLP verification error, and transaction layer delay and other abnormal types. At the same time, the physical layer and protocol layer set a time synchronization clock to ensure that the time alignment error is less than 100ns. The above collected data are summarized into link quality data at each level.

[0053] Analysis module: When an anomaly occurs, it mainly performs bus ID topology analysis and link quality analysis at all levels. It can also perform machine learning prediction. Machine learning is an artificial intelligence technology that enables computers to learn from data and improve their performance without explicit programming. It can be mainly divided into supervised learning, unsupervised learning, and semi-supervised learning.

[0054] Display: A web-based visualization interface. It allows viewing of real-time bus topology diagrams and dynamically rendered eye diagrams, including an interactive interface. It allows configuration of some CLI (Command-Line Interface) console functions, and is used for highlighting faulty links in red, generating diagnostic reports, and providing relevant maintenance suggestions.

[0055] as follows Figure 4 This is a flowchart illustrating the specific implementation and processing procedures of this embodiment.

[0056] (1) The control module triggers GPU stress testing (such as dcgmi stress –memtest, where dcgmi is a toolset for managing and monitoring GPUs in the data center, and dcgmi stress –memtest is the instruction for dcgmi to control stress testing; or NVIDIA-SMI stress test instruction), monitors the complete bus ID topology and transmits signals to the display and data acquisition components.

[0057] (2) When a fixed-slot GPU failure is detected, the current device tree is generated by using lspci -tvnn (lspci is a command-line tool for displaying detailed information of all PCI (Peripheral Component Interconnect) devices. The lspci -tvnn command provides a tree view that shows the hierarchical relationship between devices) and the abnormal bus ID is located by comparing it with the baseline topology. The link quality data (bandwidth information, impedance deviation rate, eye diagram, and PCIe TLP packet retransmission rate, etc.) of each link segment are automatically backtracked for 30 seconds before the failure.

[0058] (3) The analysis module uses the fault correlation engine to perform topology and link quality analysis based on the data collected by the acquisition components. This includes comparing the topology with the baseline topology during anomalies, analyzing the link quality quantification model, and predicting link quality through machine learning methods, as detailed below:

[0059] Define a quality score Q for each link segment:

[0060]

[0061] in Impedance deviation rate refers to the percentage fluctuation of the measured link impedance relative to the standard impedance. Eye diagram signal jitter refers to the deviation of the eye diagram signal edge from the ideal position, which can be obtained by analyzing the eye diagram using a jitter analyzer. Power supply fluctuation refers to the percentage fluctuation of the measured power supply port voltage relative to the standard voltage. The lower the quality score Q, the worse the quality of that link segment. (Values ​​used in this embodiment) ).

[0062] The main parameters for link monitoring are shown in Table 1, which is a table of the main parameters for link monitoring and their corresponding fault threshold ranges.

[0063] Table 1

[0064]

[0065] In Table 1, the PCIe channel width negotiation status refers to the process by which PCIe devices negotiate the optimal communication channel width (e.g., x1, x2, x4, x8, x16, where xm represents the use of m physical channels for communication) through link training. This ensures that both ends of the device can transmit data with optimal bandwidth. The differential signal eye diagram aperture area can usually be obtained from the first bandwidth data using an oscilloscope. The smaller the aperture area in the differential signal eye diagram compared to the standard eye diagram area, the greater the signal distortion; conversely, the larger the aperture area in the differential signal eye diagram compared to the standard eye diagram area, the less interference or attenuation the signal experiences.

[0066] Each parameter is compared with its corresponding fault threshold range. If any parameter falls within the fault threshold range, a fault is detected in the link. The fault threshold range is set according to actual needs and is not specifically limited here. In Table 1, the fault threshold range is set as follows: if the PCIe channel width negotiation state is in non-x16 mode for more than 5 seconds, a link fault is detected; if the differential signal eye diagram opening area is less than 60% of the standard eye diagram area, a link fault is detected; if the absolute value of the impedance deviation rate is greater than 5%, a link fault is detected; if the absolute value of the power supply fluctuation is greater than 5%, a link fault is detected.

[0067] (4) Output a diagnostic report and generate a fault probability distribution map based on the quality score of each link. For example: CPU-server motherboard link: 12%; server motherboard-SW board link: 83%; then mark the server motherboard-SW board link as a faulty segment; SW board-GPU board link: 5%; GPU body: 0%; and output maintenance suggestions.

[0068] For example, in an 8-GPU server, a sustained stress test on the GPUs fails. Traditional solutions require replacing all GPU boards for troubleshooting. However, with the system in this embodiment, it is not necessary to replace all GPU boards. Specifically:

[0069] (1) The faulty GPU bus ID was found to be 04:00.0 (normally it should be 03:00.0) by using lspci.

[0070] (2) Trace the upstream bus ID 02:00.1 to port 4 of the SW board;

[0071] (3) The impedance of the link was measured by an impedance test probe. Under normal test conditions, the curve was 85Ω±2Ω, and under abnormal conditions, it was 112Ω (exceeding the standard). At the same time, there was an abnormal delay in the protocol layer signal of this link segment, which further confirmed the abnormal link.

[0072] (4) It was confirmed that the PCB trace of the 4th channel of the SW board was broken.

[0073] Next, the server link fault location method provided in the embodiments of this application is described with reference to the accompanying drawings. Figure 5 This is a flowchart illustrating the server link fault location method provided in this application embodiment. The method is applied to the test component of the aforementioned server link fault location system, such as... Figure 5 As shown, the method includes the following steps:

[0074] In step S301, a test command is sent to the server link.

[0075] The test command is issued through the control module of this application embodiment; the test command is a stress test command for the graphics processor.

[0076] It is understood that in this embodiment of the application, the control module first sends a test command to the server link. After receiving the command, the server link triggers a stress test on the graphics processor, and the control module monitors the server link in real time.

[0077] In step S302, when a server link failure is detected, physical layer data and protocol layer data of each data acquisition point are obtained from the acquisition component.

[0078] Among them, detecting a server link failure means that the control module detects a failure of the graphics processor in the fixed slot of the server link. At this time, the control module locates the abnormal bus identifier of the device in the server link and sends a fault command to the acquisition component. Obtaining physical layer data and protocol layer data of each data acquisition point from the acquisition component means that the analysis module of this application embodiment obtains the physical layer data and protocol layer data of each data acquisition point obtained by the acquisition component. The obtained data is the physical layer data and protocol layer data within the target time period obtained after the acquisition component responds to the fault command and extracts a time period of the preset target time period length before the time of the fault occurrence as the target time period.

[0079] It is understood that when the control module of this application embodiment detects a failure of the graphics processor in the fixed slot of the server link, it locates the abnormal bus identifier of the device in the server link and sends a fault command to the acquisition component. The analysis module obtains the physical layer data and protocol layer data of each data acquisition point within the target time period from the acquisition component. For example, after the acquisition component responds to the fault command, the preset target time period length is 30 seconds, that is, 30 seconds are counted back from the time of the fault as the target time period, and the physical layer data and protocol layer data within the target time period are obtained. The physical layer data and protocol layer data within the target time period are used as the basis for the analysis of the analysis module.

[0080] In step S303, the fault location and / or fault type of the server link are determined based on the physical layer data and protocol layer data of each data acquisition point.

[0081] The fault location of a server link refers to the location of the faulty link segment; the types of server link faults include hardware link faults and software errors.

[0082] It is understood that the embodiments of this application can analyze and locate the location of the faulty link segment of the server link through the physical layer data and protocol layer data of each data collection point, and can distinguish the type of server link fault. It does not require replacing all graphics processing components for troubleshooting, thus achieving the technical effect of improving troubleshooting efficiency.

[0083] In this embodiment of the application, the fault location and / or fault type of the server link is determined based on the physical layer data and protocol layer data of each data collection point, including: calculating the monitoring parameters of each link segment of the server link through the physical layer data; determining the faulty link of the server link based on the monitoring parameters of each link segment and the corresponding fault threshold; locating the faulty link based on the identifier of the data collection point; and determining the fault type of the server link based on the protocol layer data.

[0084] The monitoring parameters for each link segment include PCIe channel width negotiation status, differential signal eye diagram closure, impedance deviation rate, and power supply fluctuation. These parameters are calculated using physical layer data from each data acquisition point. Specifically, the PCIe channel width negotiation status refers to the process by which PCIe devices negotiate the optimal communication channel width (e.g., x1, x2, x4, x8, x16) through link training, ensuring that both ends of the device can transmit data with optimal bandwidth. The differential signal eye diagram can typically be obtained from the first bandwidth data using an oscilloscope. The wider the "eye" in the differential signal eye diagram, i.e., the larger the opening area, the less signal distortion; conversely, the smaller the opening area, the greater the signal interference or attenuation. The impedance deviation rate refers to the percentage fluctuation of the measured link impedance compared to the standard impedance. Power supply fluctuation refers to the percentage fluctuation of the measured power supply port voltage compared to the standard voltage. The fault thresholds corresponding to the monitoring parameters of each link segment are set according to actual needs and are not specifically limited here. For example, PCIe... The fault threshold for channel width negotiation is set to a duration greater than 5 seconds in non-x16 mode, indicating a fault. The fault threshold for differential signal eye diagrams is set to a differential signal eye diagram opening area less than 60% of the standard eye diagram area, indicating a fault. The fault threshold for impedance deviation rate is set to an absolute value greater than 5%, indicating a fault. The fault threshold for power supply fluctuation is set to an absolute value greater than 5%, indicating a fault. The identifier of the data acquisition point is the bus identifier where the data acquisition point is located. The fault type of the server link is determined based on the protocol layer data. By analyzing the protocol layer data, for example, if there are too many retransmissions, TLP packet loss, or format errors, it is determined that there is a problem with the software.

[0085] It is understood that, by collecting and analyzing the physical layer and protocol layer data of each data acquisition point in the server link, the monitoring parameters of each link segment, such as PCIe channel width, eye diagram quality, bit error rate, and power supply fluctuation, can be calculated and compared with preset fault thresholds to determine whether a faulty link exists. If a link is determined to be faulty, the specific location of the faulty link can be located by combining the bus identifier at the data acquisition point. At the same time, by analyzing protocol layer data (such as TLP packet loss, retransmission count, etc.), it is possible to further identify whether the link fault is caused by a software error, thereby distinguishing the fault type of the server link and realizing efficient fault detection and diagnosis of the server link.

[0086] In this embodiment of the application, determining the fault location and / or fault type of the server link based on the physical layer data and protocol layer data of each data acquisition point further includes: calculating the quality parameters of each segment of the server link using the target physical layer data; calculating the quality score of each segment of the server link using a quality scoring formula based on the quality parameters of each segment of the server link; analyzing the quality of each segment of the server link based on the quality score; and generating a fault probability distribution map based on the quality of each segment of the server link.

[0087] The formula for calculating the quality parameters of each segment of the server link is as follows:

[0088]

[0089] in , The value is set according to actual needs and is not specifically limited here; the lower the quality score Q, the worse the quality of the link segment.

[0090] It is understood that the embodiments of this application collect and analyze the physical layer data of each segment in the server link, extract key parameters reflecting communication quality, such as impedance deviation rate, eye diagram quality, power supply fluctuation, etc., and use the set quality scoring formula to evaluate each link segment to obtain its link quality status. On this basis, a fault probability distribution map of each link segment is further drawn to visually show which link segments have a high risk of failure, helping to achieve refined monitoring and fault early warning of the server link.

[0091] According to the server link fault location method provided in the embodiments of this application, the control module sends test instructions to the server link. When a server link fault is detected, the analysis module obtains physical layer data and protocol layer data of each data acquisition point from the acquisition component. Based on the acquired data, the fault location and / or fault type of the server link are determined, realizing automated link quality assessment and being able to determine the fault location and fault type, thus achieving the technical effect of improving troubleshooting efficiency.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0093] For a description of the features in the embodiments of the server link fault location method, please refer to the relevant descriptions in the embodiments of the server link fault location system, which will not be repeated here.

[0094] Embodiments of this application also provide an electronic device, such as... Figure 6As shown, it includes a memory 401 and a processor 402. The memory 401 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in the above-described server link fault location method embodiment.

[0095] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the above-described server link fault location method embodiments when running.

[0096] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0097] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] The foregoing has provided a detailed description of a server link fault location system, method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A server link fault location system, characterized in that, The server link includes a central processing unit, a server motherboard, a switching component, and a graphics processing component. Multiple data acquisition points are set up on the server link. The system includes: The acquisition component includes multiple probe arrays disposed at the data acquisition points. Each probe array includes a physical layer and a protocol layer. The physical layer acquires physical layer data from the data acquisition points, and the protocol layer acquires protocol layer data from the data acquisition points. The data acquisition points include multiple acquisition points for the link between the central processing unit and the server motherboard, the link between the server motherboard and the switching component, the link between the switching component and the graphics processing unit, and the acquisition point at the power supply port of the graphics processing unit. The testing component includes a control module and an analysis module. The control module sends test commands to the server link. When a server link failure is detected, the analysis module obtains physical layer data and protocol layer data for each data acquisition point from the acquisition component. Based on the physical layer data and protocol layer data for each data acquisition point, the analysis module determines the fault location and / or fault type of the server link, including: calculating the quality parameters of each segment of the server link using the target physical layer data; calculating the quality score of each segment of the server link using a quality scoring formula based on the quality parameters of each segment; analyzing the quality of each segment of the server link based on the quality score; and generating a fault probability distribution map based on the quality of each segment of the server link.

2. The server link fault location system according to claim 1, characterized in that, The physical layer of the probe array includes a high-speed differential probe, an impedance test probe, and a power supply noise probe, wherein... The high-speed differential probe is connected to the acquisition point of the link between the central processing unit and the server motherboard, the acquisition point of the link between the server motherboard and the switching component, and the acquisition point of the link between the switching component and the graphics processing component to measure the first bandwidth data of each link segment of the server link. The impedance test probe is connected to the acquisition point of the link between the server motherboard and the switching component and the acquisition point of the link between the switching component and the graphics processing component via an RF coaxial connector to measure the impedance resolution data of the time domain reflectometer. The power noise probe is connected to the acquisition point at the power supply port of the graphics processing component to measure the second bandwidth data and the noise limit data.

3. The server link fault location system according to claim 1, characterized in that, The physical layer and the protocol layer are equipped with a time synchronization clock, which is used to align the data acquisition time of the physical layer with that of the protocol layer.

4. The server link fault location system according to claim 1, characterized in that, The control module sends a fault command to the acquisition component; The acquisition component responds to the fault command and acquires the physical layer data and the protocol layer data within a target time period, wherein the target time period is the time period prior to the fault transmission time.

5. The server link fault location system according to claim 1, characterized in that, The testing component also includes a display with an interactive interface that displays the topology and eye diagram of the server link. The analysis module renders the topology and eye diagram of the server link based on the physical layer data and protocol layer data of the data acquisition points, and annotates the topology and eye diagram of the server link based on the fault location and fault type.

6. A method for locating server link faults, characterized in that, The method is applied to the test component of the server link fault location system according to any one of claims 1-5, wherein the method includes: Send test commands to the server link; When a server link failure is detected, physical layer data and protocol layer data of each data acquisition point are obtained from the acquisition component. The data acquisition points include multiple acquisition points of the link between the central processing unit and the server motherboard, the link between the server motherboard and the switching component, the link between the switching component and the graphics processing component, and the acquisition point at the power supply port of the graphics processing component. Based on the physical layer data and protocol layer data of each data acquisition point, determine the fault location and / or fault type of the server link, including: calculating the quality parameters of each segment of the server link using the target physical layer data; calculating the quality score of each segment of the server link using a quality scoring formula based on the quality parameters of each segment; analyzing the quality of each segment of the server link based on the quality score; and generating a fault probability distribution map based on the quality of each segment of the server link.

7. The server link fault location method according to claim 6, characterized in that, The step of determining the fault location and / or fault type of the server link based on the physical layer data and protocol layer data of each data acquisition point includes: The monitoring parameters of each link segment of the server link are calculated using the physical layer data; Based on the monitoring parameters and corresponding fault thresholds of each link segment, the faulty link of the server link is determined, and the location of the faulty link is located based on the identifier of the data collection point. The fault type of the server link is determined based on the protocol layer data.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the server link fault location method as described in any one of claims 6 to 7 when executing the computer program.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the server link fault location method as described in any one of claims 6 to 7.

Images