Link fault detection method and apparatus, control device, and readable storage medium

Abstract

The application discloses a link fault detection method and device, a control device and a readable storage medium, and belongs to the technical field of artificial intelligence. The detection method comprises the following steps: determining the total number of GPU cards related to each communication link in m communication links, and the number of first GPU cards related to each communication link; the m communication links are communication links between switches associated with a GPU cluster, and the first GPU cards are GPU cards with communication failure; estimating a first probability that each communication link is in a normal state according to the total number and the number of first GPU cards related to each communication link; obtaining an average value of a probability that a link connected by all GPU cards related to the m communication links is in a normal state according to the first probability; and detecting whether each communication link has a fault according to the average value and the first probability. Thus, a link fault caused by GPU cluster flashover can be detected in time.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a link fault detection method, device, control equipment, and readable storage medium. Background Technology

[0002] In related technologies, communication link fault analysis methods for graphics processing unit (GPU) clusters used for large model training mainly rely on monitoring indicators such as switch port traffic, central processing unit (CPU) resource utilization, and GPU resource utilization to analyze and determine whether there are faults in the corresponding communication links. However, if there are intermittent outages in the GPU cluster, such as network interruptions lasting microseconds, milliseconds, or seconds, this fault analysis method is difficult to detect the link faults in a timely manner. Summary of the Invention

[0003] The purpose of this application is to provide a link failure detection method, apparatus, control device, and readable storage medium to solve the problem in related technologies that it is difficult to detect link failures caused by GPU cluster outages in a timely manner.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] Firstly, a link fault detection method is provided, applied to control equipment, including:

[0006] Determine the total number of GPU cards associated with each of the m communication links, and the number of first GPU cards associated with each communication link; wherein, m is an integer greater than 1, the m communication links are communication links between switches associated with the GPU cluster, and the first GPU card is a GPU card that has failed to communicate;

[0007] Based on the total number and the number of first GPU cards associated with each communication link, estimate the first probability that each communication link is in a normal state;

[0008] Based on the first probability that each communication link is in a normal state, the average probability that all GPU card-connected links related to the m communication links are in a normal state is obtained.

[0009] Based on the average value and the first probability that each communication link is in a normal state, the system detects whether each communication link is faulty.

[0010] Secondly, a link fault detection device is provided, applied to control equipment, including:

[0011] The determination module is used to determine the total number of GPU cards associated with each of the m communication links, and the number of first GPU cards associated with each communication link; wherein, m is an integer greater than 1, the m communication links are communication links between switches associated with the GPU cluster, and the first GPU card is a GPU card that has failed to communicate.

[0012] An estimation module is used to estimate a first probability that each communication link is in a normal state based on the total number and the number of first GPU cards associated with each communication link;

[0013] The analysis module is used to analyze and obtain the average probability that all GPU card-connected links related to the m communication links are in a normal state, based on the first probability that each communication link is in a normal state.

[0014] The detection module is used to detect whether each communication link is faulty based on the average value and the first probability that each communication link is in a normal state.

[0015] Thirdly, a control device is provided, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0016] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] In this embodiment, by determining the total number of GPU cards associated with each of the m communication links and the number of first GPU cards associated with each communication link, where the m communication links are communication links between switches associated with the GPU cluster and the first GPU cards are GPU cards that have failed to communicate, a first probability that each communication link is in a normal state is estimated based on the total number and the number of first GPU cards associated with each communication link. Based on the first probability that each communication link is in a normal state, the average probability that all links connected to the GPU cards associated with the m communication links are in a normal state is analyzed. Based on the average value and the first probability that each communication link is in a normal state, the presence of a fault in each communication link is detected. GPU cluster intermittent fault location can be performed based on the GPU cluster topology map, thereby timely detecting link faults caused by GPU cluster intermittent faults. Attached Figure Description

[0018] Figure 1 This is a flowchart of a link failure detection method provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a GPU cluster constructed according to a two-layer switch mode in an embodiment of this application;

[0020] Figure 3A This is a schematic diagram of the model data saving process in an embodiment of this application;

[0021] Figure 3B This is a schematic diagram of the model data recovery process in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of a link fault detection device provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The link fault detection method, apparatus, control device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0027] Please see Figure 1 , Figure 1 This is a flowchart of a link failure detection method provided in an embodiment of this application. The method is applied to a control device, which may be, but is not limited to, a controller, a server, etc. Figure 1 As shown, the method includes the following steps:

[0028] Step 11: Determine the total number of GPU cards associated with each of the m communication links, and the number of first GPU cards associated with each communication link; where m is an integer greater than 1, the m communication links are communication links between switches associated with the GPU cluster, and the first GPU card is the GPU card that failed to communicate.

[0029] Step 12: Based on the total number and the number of first GPU cards associated with each communication link, estimate the first probability that each communication link is in a normal state;

[0030] Step 13: Based on the first probability that each communication link is in a normal state, analyze and obtain the average probability that all GPU card-connected links related to the m communication links are in a normal state;

[0031] Step 14: Based on the average value and the first probability that each communication link is in a normal state, detect whether there is a fault in each communication link.

[0032] In this embodiment, the GPU cluster may be associated with a two-layer or more-layer switch. When the GPU cluster is associated with a two-layer switch, the m communication links are specifically communication links between the two-layer switches associated with the GPU cluster. When the GPU cluster is associated with a three-layer switch, the m communication links may be communication links between the three-layer switches associated with the GPU cluster.

[0033] The GPU cluster can be used for training large models, such as Large Language Models (LLMs).

[0034] The solution in this application can locate GPU cluster intermittent failures based on the GPU cluster topology map, thereby promptly detecting link failures caused by GPU cluster intermittent failures.

[0035] Furthermore, when GPU clusters are used for large model training, the amount of training data is large, typically in the trillions of terabytes, and the number of model parameters is in the hundreds of billions. Therefore, the requirements for communication quality are high. If a network interruption occurs, it will affect the training efficiency of the large model. The solution in this application can detect link failures caused by GPU cluster interruptions in a timely manner, thereby ensuring the training efficiency of the large model and realizing intelligent operation and maintenance of GPU clusters used for large model training.

[0036] In this embodiment of the application, the NVidia Collective multi-GPU Communication Library (NCCL) can be used to locate intermittent network outages in the GPU cluster.

[0037] Prior to step 11 above, the link failure detection method in this embodiment may further include:

[0038] By utilizing the NVIDIA Unified Communication Library, each GPU card in the GPU cluster is detected to identify those experiencing communication failures. This allows for efficient detection of GPU cards with communication failures, enabling the localization of intermittent network outages in the GPU cluster.

[0039] The NVIDIA Collective Communication Library (NCCL) is a library for communication between GPUs, enabling multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and networks. NCCL provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, point-to-point send, and receive, all optimized for high-speed interconnects via Peripheral Component Interconnect Express (PCIe) and NVLink within nodes. NCCL establishes a communication endpoint for each GPU card. During each communication process in collective communication, the response is checked; if a response times out (checkTimeout = True) or the response result is not equal to ncclSuccess, the communication for that GPU card is considered to have failed.

[0040] Optionally, the process of estimating the first probability that each communication link is in a normal state based on the total number and the number of first GPU cards associated with each communication link may include: first, calculating the ratio of the number of first GPU cards associated with each communication link to the total number; then, subtracting the ratio from 1 to obtain the first probability that each communication link is in a normal state. This first probability can be understood as the probability that each communication link is in a normal state estimated based on GPU cards that have failed in communication.

[0041] In this embodiment, the Least Absolute Shrinkage and Selection Operator (LASSO) algorithm can be used to analyze link faults. The basic idea of ​​LASSO is to minimize the sum of squared residuals under the constraint that the sum of the absolute values ​​of the regression coefficients is less than a constant, thereby generating some regression coefficients that are strictly equal to 0.

[0042] Optionally, step 13 above may include:

[0043] Based on the first probability that each communication link is in a normal state, the probability that the links between all GPU cards and their associated switches related to the m communication links are in a normal state is analyzed using the LASSO (Least Absolute Shrinkage and Selection) algorithm.

[0044] The average probability of the links between all GPU cards associated with the m communication links and their associated switches being in a normal state is obtained by taking the average value.

[0045] Optionally, the process of detecting whether each communication link is faulty based on the average value and the first probability that each communication link is in a normal state may include:

[0046] If the first probability of the first communication link among the m communication links being in a normal state is greater than or equal to the average value, it is determined that the first communication link is not faulty; and / or, if the first probability of the second communication link among the m communication links being in a normal state is less than the average value, it is determined that the second communication link is faulty.

[0047] In some embodiments, after obtaining the average probability that all GPU card connections associated with the m communication links are in a normal state, hypothesis testing can be used to locate intermittent network outages in the GPU cluster. For example, firstly, the probability that each communication link is in a normal state is set to be equal to the average value. Then, if the first probability that a first communication link among the m communication links is in a normal state is greater than or equal to the average value, it is determined that the first communication link is not faulty; and / or, if the first probability that a second communication link among the m communication links is in a normal state is less than the average value, it is determined that the second communication link is faulty.

[0048] By using hypothesis testing, we can quickly locate network outages in GPU clusters.

[0049] The following is combined with Figure 2 The communication link fault detection process in the embodiments of this application will be described.

[0050] like Figure 2 As shown, taking a GPU cluster (or GPU server cluster) built in a two-layer switch mode as an example, these two switches are Layer 0 and Layer 1. Each GPU card is connected to two switches to ensure bandwidth and reliability. For example, GPU card (0,0) is connected to switch Layer 1. 1,0 and Layer 1,1 GPU card (0,1) uplink switch layer 1,0 and Layer1,1 GPU card (0,2) uplink switch Layer 1,2 and Layer 1,3 GPU card (0,3) uplink switch layer 1,2 and Layer 1,3 ,etc.

[0051] against Figure 2 Each GPU card in the system uses NCCL to detect its communication status. If the response times out (checkTimeout=True) or the response result is not equal to ncclSuccess, the communication of the corresponding GPU card is considered to have failed.

[0052] For the communication link path between two layer switches i The probability of it being in a normal state can be set as follows: This formula can be further expanded to, i.e. Among them, e i Indicates the communication link path i The number of GPU cards with related communication failures, n i Indicates the communication link path i Total number of related GPU cards, ports j Indicates the communication link path i The associated switch downlink port connects to the GPU cards in the GPU cluster, such as... Figure 2 It connects to two GPU cards.

[0053] Taking the logarithm of the above formula, we get:

[0054]

[0055] Further simplification yields: in, Indicates the communication link path i Logarithmic transformation of the probability of being in a normal state; x ij The value is 0 or 1, indicating whether such a link exists; N represents the number of GPU cards connected to the downstream port of the switch; ε i This indicates measurement noise.

[0056] according to Figure 2 The network architecture shown, taking the links connecting the first four GPU cards as an example, the probability that communication links path1, path2, path3, and path4 are in a normal state can be expressed as:

[0057] y1=β Layer1,0,0,0 +β Layer1,0,0,1 +βLayer1,0,1,0 +β Layer1,0,1,1 +ε1

[0058] y2=β Layer1,1,0,0 +β Layer1,1,0,1 +β Layer1,1,1,0 +β Layer1,1,1,1 +ε2

[0059] y3=β Layer1,2,0,2 +β Layer1,2,0,3 +β Layer1,2,1,2 +β Layer1,2,1,3 +ε3

[0060] y4=β Layer1,3,0,2 +β Layer1,3,0,3 +β Layer1,3,1,2 +β Layer1,3,1,3 +ε4

[0061] y1, y2, y3, and y4 are known quantities. For y1, β Layer1,0,0,0 This represents the GPU card (0,0) and its associated switch layer. 1,0 The probability that the link between them is in a normal state, β Layer1,0,0,1 This represents the GPU card (0,1) and its associated switch layer. 1,0 The probability that the link between them is in a normal state, β Layer1,0,1,0 This represents the GPU card (1,0) and its associated switch layer. 1,0 The probability that the link between them is in a normal state, β Layer1,0,1,1 This represents the GPU card (1,1) and its associated switch layer. 1,0 The probability that the link between them is in a normal state, where ε1 represents the measurement noise of path1; for y2, β Layer1,1,0,0 This represents the GPU card (0,0) and its associated switch layer. 1,1 The probability that the link between them is in a normal state, β Layer1,1,0,1 This represents the GPU card (0,1) and its associated switch layer. 1,1 The probability that the link between them is in a normal state, β Layer1,1,1,0 This represents the GPU card (1,0) and its associated switch layer. 1,1 The probability that the link between them is in a normal state, β Layer1,1,1,1 This represents the GPU card (1,1) and its associated switch layer. 1,1 The probability that the link between them is in a normal state, ε2 represents the measurement noise of path2; for y3, β Layer1,2,0,2 This represents the GPU card (0,2) and its associated switch layer. 1,2 The probability that the link between them is in a normal state, β Layer1,2,0,3This represents the GPU card (0,3) and its associated switch layer. 1,2 The probability that the link between them is in a normal state, β Layer1,2,1,2 This represents the GPU card (1,2) and its associated switch layer. 1,2 The probability that the link between them is in a normal state, β Layer1,2,1,3 This represents the GPU card (1,3) and its associated switch layer. 1,2 The probability that the link between them is in a normal state, ε3 represents the measurement noise of path3; for y4, β Layer1,3,0,2 This represents the GPU card (0,2) and its associated switch layer. 1,3 The probability that the link between them is in a normal state, β Layer1,3,0,3 This represents the GPU card (0,3) and its associated switch layer. 1,3 The probability that the link between them is in a normal state, β Layer1,3,1,2 This represents the GPU card (1,2) and its associated switch layer. 1,3 The probability that the link between them is in a normal state, β Layer1,3,1,3 This represents the GPU card (1,3) and its associated switch layer. 1,3 The probability that the link between them is in a normal state, ε4 represents the measurement noise of path4.

[0062] Following the above method, potential faults in the GPU cluster are analyzed using the LASSO method, and the objective function is calculated: Where argmin represents the transformation that minimizes the objective function; y represents the probability that the relevant communication link is in a normal state, which is a known quantity; β represents the probability that the link between the relevant GPU card and its associated switch is in a normal state, with x taking the value 0 or 1, indicating whether such a link exists; ρ represents the weighting coefficient, which can be set based on implementation requirements; and |||1 represents the absolute value. Since each GPU card only connects to a limited number of links (such as...), ... Figure 2 Each GPU card connects to two different switch ports, therefore the objective function consists of a sparse matrix. By solving the objective function, the probability β of the links between all GPU cards and their associated switches being in a normal state can be obtained, and thus the average value can be derived.

[0063] The probability that a certain communication link is in a normal state If so, the communication link is considered to have failed; if a communication link is in a normal state, the probability is... If the communication link fails, it is considered to be non-existent.

[0064] In this way, based on the NCCL response results and the GPU cluster topology map, Lasso regression analysis can be used to locate GPU cluster network outages, thereby achieving rapid fault location, reducing reliance on manual experience, and improving operation and maintenance efficiency.

[0065] The above detection methods can effectively identify potential link faults, but they cannot accurately determine whether a network outage is intermittent or a persistent fault. To further determine whether a communication link has a persistent fault, a determination can be made based on the number of measurements.

[0066] Optionally, when a fault is detected in the third communication link among the m communication links, the link fault detection method may further include:

[0067] Acquire measurement data from the third communication link; such measurement data may be, for example, network telemetry data.

[0068] Based on the measurement data, it is determined whether the third communication link has a persistent fault.

[0069] In this way, the measurement data can be combined to further determine whether there is a persistent fault in the communication link.

[0070] Optionally, after determining that a communication link has a persistent fault based on statistical analysis of network telemetry data, the faulty link can be deleted and the GPU cluster topology can be updated.

[0071] For example, network telemetry data can be acquired based on the telemetry function provided by the Ethernet switching platform. A fault snapshot (WJH) can be used to quickly inspect packets on all ports, achieving speeds far exceeding traditional packet inspection schemes and enabling microsecond-level packet inspection. For instance, packet inspection can primarily target messages such as Auto Negotiation Failure (MES0), Signal Integrity (MES2), Symbol Error Counter (MES3), and CRC Error Counter (MES4).

[0072] In the performance measurement data of each link mentioned above, it can be assumed that if the Symbol error counter is greater than Sth (e.g., Sth = 10) and the CRC error counter is greater than Cth (e.g., Cth = 10), then the corresponding link communication is considered to have failed.

[0073] Optionally, the performance data acquisition period can be in the microsecond range, such as 1µs or 10µs. The statistical analysis time window is T (let's assume T = 100µs), and the above performance data is sum(MES). i >Mth iWhen i∈[0,6], it is determined that the current link is faulty, and the faulty link is deleted from the NCCL topology graph.

[0074] In this embodiment, when using a GPU cluster for large model training, to avoid training interruptions due to network failures, model data and training-related data can be temporarily stored in two ways: epoch and batch. This reduces the impact of network interruptions, enabling continuous training of large models without interruption in the event of network interruptions or failures, and allowing for rapid training recovery. An epoch represents the process of training all training samples once, and a batch represents dividing the training data into several parts, each with a size of Batch_Size. Depending on the size of the temporarily stored data, at the end of each epoch, if only neural network / model parameters and training scheduling data are considered and training data is not required, the data can be temporarily stored on local GPU memory. After each batch, if storing training data is considered, the relevant data can be temporarily stored on external storage. The process of saving and restoring training data and model-related data is as follows: Figure 3A and Figure 3B As shown.

[0075] It should be noted that the link fault detection method provided in this application can be executed by a link fault detection device or a control module within that device for executing the link fault detection method. This application uses the execution of the link fault detection method by a link fault detection device as an example to illustrate the link fault detection device provided in this application.

[0076] Please see Figure 4 , Figure 4 This is a schematic diagram of a link fault detection device provided in an embodiment of this application. This device is applied to control equipment, such as... Figure 4 As shown, the link fault detection device 40 includes:

[0077] The determination module 41 is used to determine the total number of GPU cards associated with each of the m communication links, and the number of first GPU cards associated with each communication link; where m is an integer greater than 1, the m communication links are communication links between switches associated with the GPU cluster, and the first GPU card is a GPU card that has failed to communicate.

[0078] The estimation module 42 is used to estimate the first probability that each communication link is in a normal state based on the total number and the number of first GPU cards associated with each communication link;

[0079] Analysis module 43 is used to analyze and obtain the average probability of all GPU card-connected links related to the m communication links being in a normal state based on the first probability that each communication link is in a normal state.

[0080] The detection module 44 is used to detect whether each communication link is faulty based on the average value and the first probability that each communication link is in a normal state.

[0081] Optionally, the estimation module 42 is specifically used to: calculate the ratio of the number of first GPU cards associated with each communication link to the total number; and subtract the ratio from 1 to obtain the first probability that each communication link is in a normal state.

[0082] Optionally, the analysis module 43 includes:

[0083] The analysis unit is used to analyze the probability that the links between all GPU cards associated with the m communication links and their associated switches are in a normal state, based on the first probability that each communication link is in a normal state, using the Least Absolute Shrinkage and Selection Algorithm (LASSO).

[0084] The calculation unit is used to average the probability that the links between all GPU cards associated with the m communication links and their associated switches are in a normal state, and to obtain the average probability that the links connected to all GPU cards associated with the m communication links are in a normal state.

[0085] Optionally, the detection module 44 is specifically used to: determine that the first communication link is not faulty when the first probability of the first communication link in the m communication links being in a normal state is greater than or equal to the average value; and / or determine that the second communication link is faulty when the first probability of the second communication link in the m communication links being in a normal state is less than the average value.

[0086] Optionally, the link failure detection device 40 also includes:

[0087] The processing module is used to detect each GPU card in the GPU cluster using the NVIDIA Collective Communication Library to identify GPU cards in the GPU cluster that have failed to communicate.

[0088] Optionally, the link failure detection device 40 also includes:

[0089] The acquisition module is used to acquire measurement data of the third communication link when a fault is detected in the third communication link among the m communication links;

[0090] The determination module is used to determine whether there is a persistent fault in the third communication link based on the measurement data.

[0091] The link fault detection device 40 of this application embodiment can implement each process of the above-described link fault detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0092] Optional, such as Figure 5 As shown, this application embodiment also provides a control device 50, including a processor 51, a memory 52, and a program or instructions stored in the memory 52 and executable on the processor 51. When the program or instructions are executed by the processor 51, they implement the various processes of the above-described link fault detection method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0093] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they can implement the various processes of the above-described link fault detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0094] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0095] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0096] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a service classification device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A link fault detection method applied to a control device, characterized in that, The method comprises: determining the total number of graphics processing unit (GPU) cards connected by each of m communication links and the number of first GPU cards connected by each of the m communication links, wherein m is an integer greater than 1, the m communication links are communication links between switches associated with a GPU cluster, and the first GPU cards are GPU cards with communication failures; estimating a first probability that each of the m communication links is in a normal state according to the total number and the number of first GPU cards connected by each of the m communication links; obtaining an average value of a probability that links connected by all GPU cards associated with the m communication links are in a normal state by analysis according to the first probability that each of the m communication links is in a normal state; detecting whether each of the m communication links has a fault according to the average value and the first probability that each of the m communication links is in a normal state.

2. The method of claim 1, wherein, The method of estimating the first probability that each of the m communication links is in a normal state according to the total number and the number of first GPU cards connected by each of the m communication links comprises: calculating a ratio of the number of first GPU cards connected by each of the m communication links to the total number; and obtaining the first probability that each of the m communication links is in a normal state by subtracting the ratio from 1.

3. The method of claim 1, wherein, The method of obtaining the average value of the probability that links connected by all GPU cards associated with the m communication links are in a normal state by analysis according to the first probability that each of the m communication links is in a normal state comprises: obtaining the probability that links between all GPU cards associated with the m communication links and switches associated with the GPU cards are in a normal state by analysis according to the first probability that each of the m communication links is in a normal state using a least absolute shrinkage and selection operator (LASSO) algorithm; and obtaining the average value of the probability that links connected by all GPU cards associated with the m communication links are in a normal state by averaging the probability that links between all GPU cards associated with the m communication links and switches associated with the GPU cards are in a normal state.

4. The method according to any one of claims 1 to 3, characterized in that, The method of detecting whether each of the m communication links has a fault according to the average value and the first probability that each of the m communication links is in a normal state comprises: determining that a first communication link of the m communication links does not have a fault when the first probability that the first communication link is in a normal state is greater than or equal to the average value; and / or determining that a second communication link of the m communication links has a fault when the first probability that the second communication link is in a normal state is less than the average value.

5. The method of claim 1, wherein, Before the method of determining the total number of GPU cards connected by each of m communication links and the number of first GPU cards connected by each of the m communication links, the method further comprises: detecting each GPU card in a GPU cluster using an NVIDIA Collective Communications Library (NVCC) to obtain GPU cards with communication failures in the GPU cluster.

6. The method of claim 1, wherein, When it is detected that a third communication link of the m communication links has a fault, the method further comprises: obtaining measurement data of the third communication link; and determining whether the third communication link has a persistent fault according to the measurement data.

7. A link fault detection apparatus characterized by comprising: The method comprises: determining the total number of GPU cards associated with each of m communication links and the number of first GPU cards associated with each of the m communication links, wherein m is an integer greater than 1, the m communication links are communication links between switches associated with a GPU cluster, and the first GPU cards are GPU cards with failed communication; estimating a first probability that each of the m communication links is in a normal state according to the total number and the number of first GPU cards associated with each of the m communication links; analyzing an average value of a probability that all links connected by GPU cards associated with the m communication links are in a normal state according to the first probability that each of the m communication links is in a normal state; detecting whether each of the m communication links has a fault according to the average value and the first probability that each of the m communication links is in a normal state.

8. The apparatus of claim 7, wherein, The analyzing module comprises: an analyzing unit configured to analyze, according to the first probability that each of the m communication links is in a normal state, a probability that all links between switches associated with all GPU cards associated with the m communication links are in a normal state by using a least absolute shrinkage and selection algorithm (LASSO); a calculating unit configured to average the probability that all links between switches associated with all GPU cards associated with the m communication links are in a normal state to obtain the average value of the probability that all links connected by GPU cards associated with the m communication links are in a normal state.

9. A control device, characterized by The method comprises a processor, a memory, and a program or instruction stored on the memory and executable on the processor, and the program or instruction is executed by the processor to implement the steps of the link fault detection method according to any one of claims 1 to 6.

10. A readable storage medium, characterized by, The readable storage medium stores a program or instruction, and the program or instruction is executed by the processor to implement the steps of the link fault detection method according to any one of claims 1 to 6.

Images