Fault detection method, device, equipment and medium
By identifying the target link based on fault reporting information in the graphics processor interconnect system and detecting the link segments step by step, the problem of difficult fault location in GPU module interconnect in the prior art is solved, and the efficiency of fault detection and maintenance is improved.
Patent Information
- Application Number
- CN202610098148.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-23
- Publication Date
- 2026-04-14
AI Technical Summary
In existing technologies, when GPU module interconnect failures occur in graphics processor interconnect system servers, fault location is difficult, requiring the disassembly and reassembly of complex machines, resulting in low maintenance efficiency.
The target link is determined based on the fault reporting information, and the link segments in the target link are detected. The faulty link segment is accurately located by gradually detecting the components connected by internal and external cables.
It improves the fault detection efficiency of graphics processor interconnect systems, enables efficient and accurate fault location, and reduces maintenance time and the possibility of misoperation.
Smart Images
Figure CN121864556A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of server technology, and in particular to a fault detection method, apparatus, device, and medium. Background Technology
[0002] Currently, in order to improve computing power, GPU (Graphics Processing Unit) interconnect system servers that support scale-out mode have emerged. However, when the server of this system encounters GPU module interconnect failure, production or data center maintenance personnel need to disassemble and reassemble complex machines to locate the fault. At the same time, due to the inability to accurately locate the fault, a large number of key components need to be tested and replaced for repair verification, which is time-consuming and has low repair efficiency.
[0003] Therefore, how to improve the fault detection efficiency of graphics processor interconnect systems is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] The purpose of this invention is to provide a fault detection method, apparatus, device, and medium that can improve the fault detection efficiency of graphics processor interconnect systems. The specific solution is as follows:
[0005] In a first aspect, the present invention provides a fault detection method, comprising:
[0006] Based on the fault reporting information, a target link with a fault in the graphics processor interconnect system is identified. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line.
[0007] The first link segment in the target link is detected to obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module.
[0008] If the detection result of the first link segment indicates that the first link segment does not meet the requirements, the first link segment is segmented for detection. If the detection result of the first link segment indicates that the first link segment meets the requirements, the faulty link segment is determined based on the second link segment, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
[0009] Optionally, if the detection result of the first link segment indicates that the first link segment does not meet the requirements, segment detection is performed on the first link segment, including:
[0010] The third link segment is detected to obtain the detection result of the third link segment, wherein the third link segment is a link segment connected by internal cables or a link segment connected by external cables;
[0011] If the detection result of the third link segment indicates that the third link segment does not meet the requirements, then the third link segment is determined to be a faulty link segment;
[0012] If the detection result of the third link segment indicates that the third link segment meets the requirements, then the fourth link segment is determined to be a faulty link segment, wherein the fourth link segment is the remaining link segment in the first link segment excluding the third link segment.
[0013] Optionally, before detecting the first link segment in the target link and obtaining the detection result of the first link segment, the method further includes:
[0014] If the first accelerator module is the faulty end in the fault reporting information, then the link segment between the first accelerator module and the second timer is determined as the first link segment;
[0015] If the second accelerator module is identified as the faulty end in the fault reporting information, then the link segment between the first timer and the second accelerator module is determined as the first link segment.
[0016] Optionally, both the first accelerator module and the second accelerator module are modules equipped with a graphics processor. The faulty target link in the graphics processor interconnect system is determined based on fault reporting information, including:
[0017] The faulty graphics processor identifier is determined based on the fault report information;
[0018] Based on the accelerator module corresponding to the faulty graphics processor identifier, the target link in the graphics processor interconnect system with the fault is determined.
[0019] Optionally, determining the target link with a fault in the graphics processor interconnect system based on the accelerator module corresponding to the faulty graphics processor identifier includes:
[0020] Based on a pre-defined correspondence, the accelerator module corresponding to the faulty graphics processor identifier is determined, and the target link with the fault in the graphics processor interconnect system is determined based on the accelerator module.
[0021] Optional, also includes:
[0022] Identify the driver version corresponding to the graphics processor;
[0023] The preset correspondence is obtained based on the driver version.
[0024] Optionally, the first link segment in the target link is detected to obtain the first link segment detection result, including:
[0025] The communication bit error rate of the first link segment in the target link is detected to obtain the first link segment detection result. If the first link segment detection result is greater than the preset bit error rate threshold, it indicates that the first link segment does not meet the requirements; otherwise, it indicates that the first link segment meets the requirements.
[0026] Secondly, the present invention provides a fault detection device, comprising:
[0027] The target link determination module is used to determine the target link with a fault in the graphics processor interconnect system based on the fault reporting information. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line.
[0028] The first detection module is used to detect the first link segment in the target link and obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module.
[0029] The second detection module is used to perform segment detection on the first link segment when the detection result of the first link segment indicates that the first link segment does not meet the requirements, and to determine the faulty link segment based on the second link segment when the detection result of the first link segment indicates that the first link segment meets the requirements, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
[0030] Thirdly, the present invention provides an electronic device, comprising:
[0031] Memory, used to store computer programs;
[0032] A processor is used to execute the computer program to implement the steps of the aforementioned fault detection method.
[0033] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned fault detection method.
[0034] Fifthly, the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed fault detection method.
[0035] As can be seen from the above scheme, the present invention provides a fault detection method, including: determining a target link with a fault in the graphics processor interconnect system based on fault reporting information, wherein the target link is the link between a first accelerator module in a first tail section and a second accelerator module in a second tail section, wherein in the first tail section, the first accelerator module and a first timer are connected via a first internal cable, and in the second tail section, the second accelerator module and the second timer are connected via a second internal cable, and the first timer and the second timer are connected via an external line; detecting a first link segment in the target link to obtain a first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module; if the first link segment detection result indicates that the first link segment does not meet the requirements, performing segment detection on the first link segment; if the first link segment detection result indicates that the first link segment meets the requirements, determining a faulty link segment based on a second link segment, wherein the second link segment is the remaining link segment in the target link excluding the first link segment.
[0036] As can be seen, the beneficial effects of the present invention are as follows: Based on fault reporting information, a target link with a fault in the graphics processor interconnect system is determined. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. The components inside the tail section are connected by internal cables, and different tail sections are connected by external cables. First, the first link segment in the target link is detected to obtain the first link segment detection result. If the first link segment does not meet the requirements, the first link segment is further segmented for detection. If the first link segment meets the requirements, the faulty link segment is determined based on the second link segment. In this way, by gradually segmenting the detection on the target link, the faulty link segment can be located efficiently and accurately, thereby improving the fault detection efficiency of the graphics processor interconnect system.
[0037] Correspondingly, the fault detection device, equipment and medium provided by the present invention also have the above-mentioned technical effects. Attached Figure Description
[0038] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart of a fault detection method provided in an embodiment of the present invention;
[0040] Figure 2 This is a schematic diagram of a graphics processor interconnect system topology provided in an embodiment of the present invention;
[0041] Figure 3 A schematic diagram of a fault detection topology for a GPU interconnect system supporting Scale-out mode is provided in an embodiment of the present invention.
[0042] Figure 4 A fault detection flowchart provided in an embodiment of the present invention;
[0043] Figure 5 This is a schematic diagram of the structure of a fault detection device provided in an embodiment of the present invention;
[0044] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.
[0046] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.
[0047] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] With the increasing demand for large-scale AI (Artificial Intelligence) models, traditional CPU-centric servers can no longer meet the needs of parallel computing. The mainstream architecture has shifted to a heterogeneous computing architecture combining CPUs with acceleration chips (such as GPUs, FPGAs, and ASICs) to provide higher computing power and energy efficiency. As the scale of large model parameters continues to expand, traditional computing architectures are struggling to meet the demands. The development paradigm of AI infrastructure is being redefined, with computing power requirements accelerating from single-point to system-level integration. Scale-up (vertical scaling) and scale-out have become the two core dimensions of computing power expansion. Scale-up enhances the computing power of single nodes, ushering in the era of supernodes; scale-out is primarily used for horizontal interconnection between a large number of nodes in ultra-large-scale AI clusters.
[0049] When a GPU module scale-out interconnect failure occurs in a graphics processor interconnect system during production or after delivery to a data center, production or data center maintenance personnel need to disassemble complex machines to locate the faulty interconnect. Furthermore, due to the inability to accurately pinpoint the fault, numerous critical components need to be inspected and replaced for repair verification. This process can easily take one to two hours to locate and repair a single faulty system, resulting in significant time consumption and low repair efficiency. Therefore, this invention provides a fault detection scheme that can efficiently and accurately locate faulty link segments, thereby improving the fault detection efficiency of graphics processor interconnect systems.
[0050] Next, a fault detection method provided by an embodiment of the present invention will be described in detail. Figure 1 This is a flowchart of a fault detection method provided in an embodiment of the present invention. The fault detection method includes:
[0051] Step S11: Based on the fault reporting information, determine the target link in the graphics processor interconnect system that has a fault, wherein the target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line.
[0052] The graphics processor interconnect system may include a front and rear structure. The front structure houses the CPU, and the rear structure may include a first rear and a second rear, each rear containing multiple accelerator modules. Each accelerator module houses a graphics processor module. The accelerator modules inside the rear are connected to retimers via internal cables, and the rear modules are connected to each other via external cables, specifically connecting two end retimers.
[0053] This invention provides a server supporting a scale-out mode GPU interconnect system, considering AI cluster design. This system comprises one compute node head unit and two GPU Box tail units. Each GPU Box tail unit supports eight GPU cards. The eight GPU cards in tail unit 1 (first tail unit) are connected to eight GPU cards in tail unit 2 (second tail unit) via external cables, forming a scale-out mode GPU interconnect system. This server supporting scale-out mode GPU interconnect system, because it includes one compute node head unit and two GPU Box tail units, with the eight GPUs in each tail unit interconnected to form a scale-out mode, involves cross-chassis GPU interconnection. It adds a retimer signal to improve driving capability and uses internal and external cables for networking interconnection. This model has a large number of interconnection cables and components, resulting in high system complexity.
[0054] See Figure 2 As shown, Figure 2This is a schematic diagram of a graphics processor interconnect system topology provided in an embodiment of the present invention, supporting scale-out mode. This composite model includes a computer head and two GPU Box tails. The computer head supports mainstream platform CPUs. The PCIe (Peripheral Component Interconnect Express) signals in the CPU chip support network cards to achieve data communication between the computer head and the data center. Simultaneously, the CPU's PCIe signals can be connected to the two GPU Box tails via external cables. The four sets of PCIe x16 signals on the computer head are extended via PCIe Switch chips supported by the GPU Box tails, and the extended PCIe links connect to eight GPU modules. The GPU modules are in OAM (Open Accelerator Module) form, thus realizing the PCIe computing link communication design. A management controller (such as a BMC (Baseboard Management Controller, whose main function is to manage the entire server) chip) is connected to the data center's management switch via management network signals and can receive remote machine management commands. The eight OAM modules (accelerator modules) at the tail section have their signal driving capability increased through a retimer. The scale-out interconnect signals of the OAM modules are connected to an external I / O (Input / Output) board via cables, and then connected to another OAM module at the tail section via an external cable. The other OAM module uses the same design. The external ports can be labeled 1-1~1-4, 2-1~2-4, and are connected to the BMC via IPMB (Intelligent Platform Management Bus) 1, and OAM modules 0~7.
[0055] Further, see Figure 3 As shown, Figure 3This is a schematic diagram of a fault detection topology for a GPU interconnect system supporting scale-out mode, provided in an embodiment of the present invention. The eight OAM modules of tail unit 1 are connected to a Retimer board (re-timer) via internal cables. The Retimer board integrates eight external interconnect I / O ports, which are then connected to the I / O ports of tail unit 2 via external cables. Tail unit 2 uses the same Retimer design. This interconnection topology enables interconnection from OAM0 of tail unit 1 to OAM1 of tail unit 2, and from OAM1 of tail unit 1 to OAM0 of tail unit 2… This dual-tail OAM interconnection topology is the automated segmented fault detection topology for OAM modules of the present invention. Tail unit 1 and tail unit 2 each contain internal cables 0 to 7, and external cables 0 to 7 connect tail unit 1 and tail unit 2, TX (Transmit) and RX (Receive).
[0056] The fault reporting information can be fault information reported by the tail structure of the graphics processor interconnect system. It can carry information such as fault type and fault end. Fault type can include link interruption, data transmission error, etc.
[0057] In an optional implementation, determining the target link in the graphics processor interconnect system that has a fault based on the fault reporting information may include: determining the faulty graphics processor identifier based on the fault reporting information; and determining the target link in the graphics processor interconnect system that has a fault based on the accelerator module corresponding to the faulty graphics processor identifier.
[0058] In other words, fault reporting information can carry the identifier of the faulty graphics processor.
[0059] In an optional implementation, determining the target link in the graphics processor interconnect system that has a fault based on the accelerator module corresponding to the faulty graphics processor identifier includes: determining the accelerator module corresponding to the faulty graphics processor identifier based on a preset correspondence, and determining the target link in the graphics processor interconnect system that has a fault based on the accelerator module.
[0060] This embodiment of the invention can determine the accelerator module corresponding to the faulty graphics processor identifier based on a preset correspondence. The preset correspondence is the relationship between the graphics processor identifier and the accelerator module, and may further include the retimer identifier and I / O port corresponding to the graphics processor identifier. Furthermore, the preset correspondence includes the correspondence between the first tail and the second tail. This embodiment can store the preset correspondence in a designated register to accurately identify and precisely locate the identifier. Table 1 shows the correspondence between GPU IDs (i.e., identifiers) and the IDs of each component in the link of a GPU interconnect system supporting Scale-out mode. In this GPU cluster system supporting Scale-out mode, due to the complexity of the GPU Scale-out interconnect link, which includes OAM modules, retimers, I / O boards, internal cables, and external cables, this embodiment independently encodes key components on the link and maps them one-to-one with the GPU IDs under the OS (Operating System). As shown in Table 1, GPU IDs 0-15 are mapped one-to-one with OAM slots 0-7, Retimers 1-7, and IO boards 1-7 of tail 1, and OAM slots 0-7, Retimers 1-7, and IO boards 1-7 of tail 2, respectively. This ID pair is then entered into a register of the management controller (e.g., BMC) and simultaneously uploaded to the system OS for ID identification in automated segmented fault detection, enabling precise location.
[0061] Table 1
[0062]
[0063] In an optional implementation, this embodiment may further include: identifying the driver version corresponding to the graphics processor; and obtaining the preset correspondence based on the driver version.
[0064] During the production process, different driver versions may correspond to different preset correspondences. This embodiment can determine the preset correspondences of driver versions to avoid errors.
[0065] Step S12: Detect the first link segment in the target link to obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module.
[0066] In other words, the embodiments of the present invention can first detect long link segments to determine whether there are any faults.
[0067] In an optional implementation, before detecting the first link segment in the target link and obtaining the first link segment detection result, the method may further include: if the first accelerator module is a faulty end in the fault reporting information, then the link segment between the first accelerator module and the second timer is determined as the first link segment; if the second accelerator module is a faulty end in the fault reporting information, then the link segment between the first timer and the second accelerator module is determined as the first link segment.
[0068] It should be noted that in some cases, detecting the link where the faulty end is located first can help pinpoint the problem more quickly.
[0069] In an optional implementation, detecting the first link segment in the target link to obtain a first link segment detection result may include: detecting the communication bit error rate of the first link segment in the target link to obtain a first link segment detection result; if the first link segment detection result is greater than a preset bit error rate threshold, it indicates that the first link segment does not meet the requirements; otherwise, it indicates that the first link segment meets the requirements.
[0070] This embodiment can issue detection commands, and based on these commands, perform data transmission and reception on the first link to obtain the bit error rate (BER) of the first link. For example, data is sent from the first accelerator module to the second timer, and the first accelerator module then receives data returned by the second timer, thereby calculating the BER. The detection commands can be PRBS (Pseudo-Random Binary Sequence) related commands, used for testing the communication system, such as BER testing.
[0071] Step S13: If the detection result of the first link segment indicates that the first link segment does not meet the requirements, the first link segment is segmented for detection. If the detection result of the first link segment indicates that the first link segment meets the requirements, the faulty link segment is determined based on the second link segment, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
[0072] Specifically, the first link segment can be segmented for testing into two parts: an internal link and an external link. Furthermore, if the first link segment's testing results indicate that it meets the requirements, the second link segment can be tested to identify faulty links. If the second link segment does not meet the bit error rate requirements, then the second link segment has a problem.
[0073] In an optional implementation, if the detection result of the first link segment indicates that the first link segment does not meet the requirements, segment detection is performed on the first link segment, including: detecting a third link segment to obtain a third link segment detection result, wherein the third link segment is a link segment connected by internal cables or a link segment connected by external cables; if the detection result of the third link segment indicates that the third link segment does not meet the requirements, then the third link segment is determined to be a faulty link segment; if the detection result of the third link segment indicates that the third link segment meets the requirements, then the fourth link segment is determined to be a faulty link segment, wherein the fourth link segment is the remaining link segment in the first link segment excluding the third link segment. For example, the third link segment is the link segment between the first accelerator module and the first timer, and the fourth link segment is the link segment between the first timer and the second timer.
[0074] In this embodiment, link detection can be performed by sending commands to detect bit error rate. If the detection result is greater than the preset bit error rate threshold, the characterization does not meet the requirements; otherwise, the characterization meets the requirements.
[0075] Furthermore, this invention provides an automated segmented fault detection scheme for GPU interconnect systems in Scale-out mode. Based on the interconnect topology of the system and the automated segmented fault detection topology of the GPU interconnect, a correspondence between GPU IDs and component IDs in the Scale-out mode GPU interconnect system is constructed. This correspondence is then entered into a register in the BMC and uploaded to the system OS for ID identification in automated segmented fault detection, achieving precise location. The proposed automated segmented fault detection method for GPU interconnect performs automated segmented fault detection on three interconnect links of the GPU interconnect, achieving accurate fault reporting for the three links: OAM0 (i.e., the first accelerator module) to Retimer0 (i.e., the first timer), Retimer0 to Retimer1 (i.e., the second timer), and OAM1 (i.e., the second accelerator module) to Retimer1 (i.e., the second timer).
[0076] See Figure 4 As shown, Figure 4This invention provides a fault detection flowchart for automated segmented fault detection of GPU module interconnects. When servers supporting Scale-out mode GPU cluster systems encounter faults in the Scale-out dual-tail GPU interconnect during production or data center delivery, automated segmented fault detection can be performed according to the following process. Since different driver versions result in different GPU IDs, the driver version of the GPU module is first identified to ensure the correct correspondence between the driver version and the preset GPU ID and the IDs of each component in the link. Then, the reported fault type is identified, and the faulty end and the peer end are identified, such as a fault between OAM0 of tail 1 and OAM1 of tail 2. The head OS issues PRBS-related commands to automatically detect large loops in the interconnect link, i.e., the OAM0-Retimer0-Retimer1 link, obtains the communication error code corresponding to the returned OAM data of tail 1, and determines whether the actual bit error rate meets the requirements according to the designed bit error rate standard. If the bit error rate of the large loopback link meets the requirements, it indicates that the link at this end is not faulty. Further verification of the other end of the link is then performed. The head unit OS issues PRBS commands to perform small loopback 1 verification, i.e., verifying the OAM1-Retimer1 link. If it does not meet the requirements, it indicates a problem with the OAM1-Retimer1 link, and a link fault from OAM1 to Retimer1 is reported for repair. If the large loopback does not meet the judgment criteria, the verification is further refined. The head unit OS issues PRBS commands to perform small loopback 2 verification, verifying the OAM0-Retimer0 link. The bit error rate returned by small loopback 2 is used for judgment. If it meets the requirements, it indicates that the link is not faulty, and a link fault from Retimer0 to Retimer1 is reported for repair; if it does not meet the requirements, it indicates a problem with the OAM0-Retimer0 link, and a link fault from OAM0 to Retimer0 is reported for repair. In summary, automated segmented fault detection and accurate repair reporting for the three-segment link interconnection of GPU modules supporting Scale-out mode can be achieved, improving maintainability. Fail indicates that the requirements are not met, while Pass indicates that the requirements are met.
[0077] This embodiment pre-establishes a mapping relationship between GPU IDs and component IDs in the Scale-out mode GPU interconnect system, and records this mapping relationship in a table in a register of the BMC, while simultaneously uploading it to the system OS for ID identification in automated segmented fault detection, achieving precise location. When a GPU module Scale-out interconnect failure is encountered during the production of this Scale-out supported GPU interconnect system, or after delivery to the data center, a GPU module Scale-out interconnect failure is detected. The automated segmented fault detection scheme for GPU interconnect provided by this invention can be remotely executed through the OS to automatically detect segmented faults in the three interconnect links of the GPU interconnect, achieving precise fault reporting for the three links: OAM0 to Retimer0, Retimer0 to Retimer1, and OAM1 to Retimer1. In other words, this invention provides an automated segmented fault detection scheme for GPU interconnect systems supporting Scale-out mode, which can detect GPU module Scale-out interconnect failures when encountered during the production of this Scale-out supported GPU interconnect system, or after delivery to the data center. An automated segmented fault detection scheme can be remotely executed via the OS to automatically detect faults in three segments of the GPU interconnect link. This enables accurate fault reporting for the three links: OAM0 to Retimer0, Retimer0 to Retimer1, and OAM1 to Retimer1. This reduces the on-site location time for production or data center maintenance engineers, minimizes the risk of operational errors, improves the maintainability of this complex system, and increases maintenance efficiency. This invention can be applied to any server product interconnected with a host and a Box.
[0078] See Figure 5 As shown, Figure 5 This is a schematic diagram of a fault detection device provided in an embodiment of the present invention. The fault detection device includes:
[0079] The target link determination module 51 is used to determine the target link with a fault in the graphics processor interconnect system based on the fault reporting information. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line.
[0080] The first detection module 52 is used to detect the first link segment in the target link and obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module.
[0081] The second detection module 53 is used to perform segment detection on the first link segment when the detection result of the first link segment indicates that the first link segment does not meet the requirements, and to determine the faulty link segment based on the second link segment when the detection result of the first link segment indicates that the first link segment meets the requirements, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
[0082] In an optional implementation, the second detection module 53 may be specifically used to: detect the third link segment and obtain a third link segment detection result, wherein the third link segment is a link segment connected by internal cables or a link segment connected by external cables; if the third link segment detection result indicates that the third link segment does not meet the requirements, then the third link segment is determined to be a faulty link segment; if the third link segment detection result indicates that the third link segment meets the requirements, then the fourth link segment is determined to be a faulty link segment, wherein the fourth link segment is the remaining link segment in the first link segment excluding the third link segment.
[0083] Furthermore, the device can also be used for:
[0084] If the first accelerator module is the faulty end in the fault reporting information, then the link segment between the first accelerator module and the second timer is determined as the first link segment;
[0085] If the second accelerator module is identified as the faulty end in the fault reporting information, then the link segment between the first timer and the second accelerator module is determined as the first link segment.
[0086] Wherein, both the first accelerator module and the second accelerator module are modules equipped with graphics processors, and the target link determination module 51 may specifically include:
[0087] The identifier determination submodule is used to determine the faulty graphics processor identifier based on the fault reporting information;
[0088] The link determination submodule is used to determine the target link in the graphics processor interconnect system that has a fault, based on the accelerator module corresponding to the faulty graphics processor identifier.
[0089] The link determination submodule can be specifically used to: determine the accelerator module corresponding to the faulty graphics processor identifier based on a preset correspondence, and determine the target link with a fault in the graphics processor interconnect system based on the accelerator module.
[0090] Furthermore, the device also includes a correspondence determination submodule, used to identify the driver version corresponding to the graphics processor; and to obtain the preset correspondence based on the driver version.
[0091] In an optional implementation, the first detection module 52 can be used to: detect the communication bit error rate of a first link segment in the target link, obtain the first link segment detection result, and if the first link segment detection result is greater than a preset bit error rate threshold, then the first link segment does not meet the requirements; otherwise, the first link segment meets the requirements.
[0092] As can be seen, this embodiment of the invention determines the target link with a fault in the graphics processor interconnect system based on fault reporting information. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. The components inside the tail section are connected by internal cables, and different tail sections are connected by external cables. First, the first link segment in the target link is detected to obtain the first link segment detection result. If the first link segment does not meet the requirements, the first link segment is further segmented for detection. If the first link segment meets the requirements, the faulty link segment is determined based on the second link segment. In this way, by gradually segmenting the detection on the target link, the faulty link segment can be located efficiently and accurately, thereby improving the fault detection efficiency of the graphics processor interconnect system.
[0093] Figure 5 The description of the features in the corresponding embodiments can be found in [reference needed]. Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here.
[0094] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, the electronic device includes: a memory 60 for storing computer programs;
[0095] The processor 61 is used to implement the steps of the fault detection method as described in the above embodiments when executing a computer program.
[0096] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.
[0097] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the fault detection method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, configuration data.
[0098] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.
[0099] Those skilled in the art will understand that Figure 6 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.
[0100] It is understood that if the fault detection method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, and other media capable of storing program code.
[0101] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the fault detection method described above.
[0102] Furthermore, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned disclosed fault detection method.
[0103] The foregoing has provided a detailed description of a fault detection method, apparatus, device, and medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0104] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0105] The foregoing has provided a detailed description of the fault detection method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.
Claims
1. A fault detection method, characterized in that, include: Based on the fault reporting information, a target link with a fault in the graphics processor interconnect system is identified. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line. The first link segment in the target link is detected to obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module. If the detection result of the first link segment indicates that the first link segment does not meet the requirements, the first link segment is segmented for detection. If the detection result of the first link segment indicates that the first link segment meets the requirements, the faulty link segment is determined based on the second link segment, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
2. The fault detection method according to claim 1, characterized in that, If the detection result of the first link segment indicates that the first link segment does not meet the requirements, segment detection is performed on the first link segment, including: The third link segment is detected to obtain the detection result of the third link segment, wherein the third link segment is a link segment connected by internal cables or a link segment connected by external cables; If the detection result of the third link segment indicates that the third link segment does not meet the requirements, then the third link segment is determined to be a faulty link segment; If the detection result of the third link segment indicates that the third link segment meets the requirements, then the fourth link segment is determined to be a faulty link segment, wherein the fourth link segment is the remaining link segment in the first link segment excluding the third link segment.
3. The fault detection method according to claim 1, characterized in that, Before detecting the first link segment in the target link and obtaining the detection result of the first link segment, the method further includes: If the first accelerator module is the faulty end in the fault reporting information, then the link segment between the first accelerator module and the second timer is determined as the first link segment; If the second accelerator module is identified as the faulty end in the fault reporting information, then the link segment between the first timer and the second accelerator module is determined as the first link segment.
4. The fault detection method according to claim 1, characterized in that, Both the first accelerator module and the second accelerator module are modules equipped with graphics processors. Based on fault reporting information, the target link in the graphics processor interconnect system with a fault is determined, including: The faulty graphics processor identifier is determined based on the fault report information; Based on the accelerator module corresponding to the faulty graphics processor identifier, the target link in the graphics processor interconnect system with the fault is determined.
5. The fault detection method according to claim 4, characterized in that, The step of determining the target link in the graphics processor interconnect system that has a fault based on the accelerator module corresponding to the faulty graphics processor identifier includes: Based on a pre-defined correspondence, the accelerator module corresponding to the faulty graphics processor identifier is determined, and the target link with the fault in the graphics processor interconnect system is determined based on the accelerator module.
6. The fault detection method according to claim 5, characterized in that, Also includes: Identify the driver version corresponding to the graphics processor; The preset correspondence is obtained based on the driver version.
7. The fault detection method according to any one of claims 1 to 6, characterized in that, The first link segment in the target link is detected to obtain the first link segment detection result, including: The communication bit error rate of the first link segment in the target link is detected to obtain the first link segment detection result. If the first link segment detection result is greater than the preset bit error rate threshold, it indicates that the first link segment does not meet the requirements; otherwise, it indicates that the first link segment meets the requirements.
8. A fault detection device, characterized in that, include: The target link determination module is used to determine the target link with a fault in the graphics processor interconnect system based on the fault reporting information. The target link is the link between the first accelerator module in the first tail section and the second accelerator module in the second tail section. In the first tail section, the first accelerator module and the first timer are connected through a first internal cable. In the second tail section, the second accelerator module and the second timer are connected through a second internal cable. The first timer and the second timer are connected through an external line. The first detection module is used to detect the first link segment in the target link and obtain the first link segment detection result, wherein the first link segment is the link segment between the first accelerator module and the second timer, or the link segment between the first timer and the second accelerator module. The second detection module is used to perform segment detection on the first link segment when the detection result of the first link segment indicates that the first link segment does not meet the requirements, and to determine the faulty link segment based on the second link segment when the detection result of the first link segment indicates that the first link segment meets the requirements, wherein the second link segment is the remaining link segment in the target link other than the first link segment.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the fault detection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the fault detection method as described in any one of claims 1 to 7.