Link failure classification method, apparatus, device, medium, and program product

Abstract

The application provides a link fault grading method and device, equipment, medium and program product, relates to the technical field of communication, and is used for improving the processing efficiency of link exception. The specific technical scheme is: obtaining transmission parameters of communication links between at least two hardware; determining N communication links satisfying a link exception condition from the communication links between the at least two hardware based on the transmission parameters; and determining a fault level of each communication link based on first information of each communication link in the N communication links. The application is applied to the scene of detecting and responding to the communication links in the intelligent computing cluster.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a link fault classification method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence and large model training, distributed intelligent computing tasks are demanding more and more computing power. In order to meet the computing power requirements of intelligent computing tasks, the application of intelligent computing clusters is becoming more and more widespread. As a result, the requirements for fault detection of data exchange between different hardware (CPU / memory / GPU, etc.) in the computing power server of the intelligent computing cluster are becoming more and more stringent.

[0003] In existing technologies, for the communication links of intelligent computing clusters, the round-trip time of probe packets between servers is measured at the software layer, and the physical network status is obtained based on the round-trip time. Then, the connectivity and congestion of the communication link are evaluated through the network status.

[0004] However, existing technologies only detect the network status of the communication link and determine whether there is an anomaly in the communication link. When an anomaly occurs, the technology only reports the anomaly, resulting in low efficiency in handling link anomalies. Summary of the Invention

[0005] This application provides a link fault classification method, apparatus, device, medium, and program product to improve the detection efficiency of link anomalies.

[0006] In a first aspect, embodiments of this application provide a link fault classification method, the method comprising: acquiring transmission parameters of a communication link between at least two hardware nodes, the transmission parameters including real-time throughput, link bandwidth and link latency; based on the transmission parameters, determining N communication links from the communication links between the at least two hardware nodes that meet the link anomaly conditions, where N is a positive integer; and based on first information of each of the N communication links, determining the fault level of each communication link, the first information including transmission parameters and link anomaly information.

[0007] The technical solution provided in this application brings at least the following beneficial effects: obtaining transmission parameters of the communication link between at least two hardware nodes; determining N communication links that meet the link anomaly conditions from the communication links between the at least two hardware nodes based on the transmission parameters; and determining the fault level of each communication link based on the first information of each of the N communication links. In this solution, the transmission status of the communication link can be obtained more accurately through the transmission parameters. Therefore, based on the transmission status and link anomaly conditions of the communication link, the communication links that meet the link anomaly conditions can be obtained. Then, based on the anomaly status information of the communication link, the impact of the anomaly can be evaluated, and the fault level of the communication link can be obtained based on the impact of the anomaly. This achieves fault level classification based on the actual transmission status of the communication link, realizing refined management of communication link faults. This allows the operation and maintenance end to more clearly obtain the degree of anomaly of the communication link, improving the efficiency of handling link anomalies.

[0008] One possible implementation involves determining N communication links that meet the link anomaly conditions from the communication links between the at least two hardware nodes based on the aforementioned transmission parameters. This includes: determining M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links, where M is an integer greater than or equal to N; and determining the N communication links from the M communication links based on the link bandwidth and link delay of each of the M communication links, where the link bandwidth of the N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

[0009] Another possible implementation involves determining M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes. This includes: when it is detected that the real-time throughput of any communication link between the at least two hardware nodes is less than the real-time throughput at a first time point than at a second time point, determining that any communication link is a suspected abnormal link, where the first time point is the time point following the second time point; and determining all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0010] Another possible implementation involves determining M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes. This includes: calculating the link utilization of the communication links between the at least two hardware nodes based on the real-time throughput and theoretical bandwidth of the communication links between the at least two hardware nodes; determining any communication link as a suspected abnormal link when an abnormal link utilization is detected in any of the communication links between the at least two hardware nodes; and determining all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0011] Another possible implementation involves obtaining the transmission parameters of the communication link between at least two hardware nodes, including: collecting data communication information between at least two hardware nodes during the operation of the intelligent computing task, wherein the data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time; and based on the data communication information, obtaining the real-time throughput, link bandwidth, and link latency of the communication link between the at least two hardware nodes.

[0012] Another possible implementation is that the link anomaly information in the first information mentioned above includes: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

[0013] Another possible implementation involves determining the fault level of each of the N communication links based on the first information of each of the N communication links, including: determining the first communication link as having a first fault level when it meets a first condition and a second condition; determining the first communication link as having a second fault level when it meets the first condition but does not meet the second condition; and determining the first communication link as having a third fault level when it does not meet the first condition but meets the third condition. The first condition is that the link bandwidth is lower than a bandwidth threshold or the link latency is higher than a latency threshold. The second condition includes at least one of the following: the duration of the abnormal link exceeds a duration threshold, the proportion of abnormal link time exceeds a time proportion threshold, or the proportion of abnormal links exceeds a link proportion threshold. The third condition is that the real-time throughput at the third time point is less than the real-time throughput at the fourth time point, where the third time point is the time following the fourth time point. The first fault level is higher than the second fault level, and the second fault level is higher than the third fault level.

[0014] Another possible implementation method further includes: when the first communication link is at the first fault level or the second fault level, reporting the fault event of the first communication link to the intelligent computing center network management system for processing; when the first communication link is at the third fault level, feeding back the fault-related information of the first communication link to the application-side hardware, so that the application-side hardware can perform task optimization and performance tuning of the first communication link based on the fault-related information.

[0015] Secondly, embodiments of this application provide a link fault classification device, comprising: an acquisition module; the acquisition module being configured to acquire transmission parameters of a communication link between at least two hardware nodes, the transmission parameters including real-time throughput, link bandwidth, and link latency; the acquisition module being configured to determine, based on the transmission parameters, N communication links from the communication links between the at least two hardware nodes that satisfy link anomaly conditions, where N is a positive integer; the acquisition module being further configured to determine the fault level of each communication link based on first information of each of the N communication links, the first information including transmission parameters and link anomaly information.

[0016] One possible implementation is that the acquisition module is specifically used to: determine M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes, where M is an integer greater than or equal to N; and determine N communication links from the M communication links based on the link bandwidth and link delay of each of the M communication links, where the link bandwidth of the N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

[0017] Another possible implementation, the acquisition module described above, is specifically used to: when it is detected that the real-time throughput of any communication link between the at least two hardware nodes is less than the real-time throughput at the first moment, determine that any communication link is a suspected abnormal link, and the first moment is the next moment after the second moment; determine all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0018] In another possible implementation, the above apparatus further includes: a calculation module; the calculation module is used to calculate the link utilization of the communication link between the at least two hardware nodes based on the real-time throughput and theoretical bandwidth of the communication link between the at least two hardware nodes; the acquisition module is specifically used to determine any communication link as a suspected abnormal link when an abnormal link utilization is detected in any of the communication links between the at least two hardware nodes; and to determine all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0019] Another possible implementation is that the acquisition module described above is specifically used to: collect data communication information between at least two hardware nodes during the operation of the intelligent computing task, wherein the data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time; and based on the data communication information, obtain the real-time throughput, link bandwidth, and link latency of the communication link between the at least two hardware nodes.

[0020] Another possible implementation is that the link anomaly information in the first information mentioned above includes: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

[0021] Another possible implementation is that the acquisition module described above is specifically used to: determine the first communication link as having a first fault level when the first communication link among the N communication links meets the first condition and the second condition; determine the first communication link as having a second fault level when the first communication link among the N communication links meets the first condition but does not meet the second condition; determine the first communication link as having a third fault level when the first communication link among the N communication links does not meet the first condition but meets the third condition; wherein the first condition is that the link bandwidth is lower than the bandwidth threshold or the link latency is higher than the latency threshold; the second condition includes at least one of the following: the duration of the abnormal link exceeds the duration threshold, the proportion of the abnormal link time exceeds the time proportion threshold, and the proportion of abnormal links exceeds the link proportion threshold; the third condition is that the real-time throughput at the third time is less than the real-time throughput at the fourth time, and the third time is the time after the fourth time; the first fault level is higher than the second fault level, and the second fault level is higher than the third fault level.

[0022] In another possible implementation, the above apparatus further includes: a sending module; the sending module is configured to report the fault event of the first communication link to the data center network management system for processing when the first communication link is at the first fault level or the second fault level; the sending module is also configured to feed back the fault-related information of the first communication link to the application-side device when the first communication link is at the third fault level, so that the application-side device can perform task optimization and performance tuning of the first communication link based on the fault-related information.

[0023] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.

[0024] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.

[0025] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.

[0026] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0027] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of a network architecture for applying a link fault classification method according to an embodiment of this application;

[0029] Figure 2 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0030] Figure 3 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0031] Figure 4 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0032] Figure 5 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0033] Figure 6 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0034] Figure 7 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0035] Figure 8 A hardware interconnection topology diagram for a link fault classification method provided in this application embodiment;

[0036] Figure 9 A flowchart illustrating a link fault classification method provided in an embodiment of this application;

[0037] Figure 10 A flowchart illustrating another link fault classification method provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the structure of a link fault classification device provided in an embodiment of this application;

[0039] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The link fault classification method, apparatus, equipment, media, and program products provided in this application will be described in detail below with reference to the accompanying drawings.

[0041] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0042] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0043] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0044] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0045] The present application provides a link fault classification method, apparatus, device, medium, and program product that can be applied to scenarios for link anomaly identification.

[0046] With the exponential growth in demand for distributed deep learning (DML) computing power, hyperscale intelligent computing centers have become the core infrastructure supporting modern AI services. To optimize energy efficiency and total cost of ownership while pursuing ultimate performance, intelligent computing clusters are shifting from traditional single-architecture designs to heterogeneous computing environments comprised of GPUs, NPUs, and dedicated ASICs. However, the introduction of this diverse computing platform has led to increasing differences in driver interfaces and interconnect architectures among different vendors, posing unprecedented challenges to the unified operation and maintenance of these clusters.

[0047] Because distributed training tasks inherently suffer from the "bottleneck effect," the throughput of the entire cluster is often limited by the worst-performing node. This makes DML services extremely sensitive to even minor fluctuations in the underlying infrastructure. Most critically, faults in intelligent computing environments often manifest as "grayscale faults," such as a hidden bandwidth decrease in PCIe links due to hardware aging or driver defects. These anomalies do not trigger explicit system crashes but can lead to a continuous degradation in task execution efficiency. In such cases, application-layer logs (such as RDMA communication timeouts) are often highly misleading, making it difficult for operations personnel to determine from ambiguous error codes whether the root cause stems from accelerator card hardware, system drivers, or complex network congestion.

[0048] To address the above issues, existing technical solutions can be broadly categorized into the following approaches:

[0049] 1) Host-based detection mechanism based on vendor-driven coupling.

[0050] This method primarily utilizes vendor-provided APIs or hardware counters for intra-host link testing. Its technical principle is based on loopback testing, which identifies bottlenecks by measuring the bandwidth and latency of data transmission between hardware components within the host. However, the detection modules and analysis logic of such solutions are typically tightly coupled to specific hardware ecosystems (such as the CUDA driver stack), resulting in extremely high migration costs on heterogeneous platforms. Furthermore, this approach usually only provides end-to-end round-trip metrics, making it difficult to accurately break down the performance of unidirectional links and lacking the ability to identify business interference.

[0051] 2) End-to-end network monitoring mechanism based on service blindness.

[0052] This method focuses on monitoring the physical network status from a cluster perspective through RRT probing between servers. Its technical implementation typically measures the round-trip time of probe packets at the software layer to assess link connectivity and congestion. However, this approach often fails to monitor specific business operations, cannot define the actual impact of network anomalies on large-scale business flows, and its measured software-layer metrics are highly susceptible to fluctuations in host CPU load, making it difficult to accurately separate host processing latency from actual network latency.

[0053] In summary, the existing approach, lacking quantitative assessment and business awareness of fault severity, directly leads to operational decision-making dilemmas: operations personnel are often forced to make difficult trade-offs between "performance loss due to response latency" and "task interruptions caused by over-response (such as mistakenly isolating normal nodes or unnecessary server restarts)." Therefore, there is an urgent need for a mechanism that is cross-platform adaptable, business-aware, and based on severity grading for intelligent response and full-stack root cause tracing to ensure the performance stability of large-scale intelligent computing clusters under complex fault scenarios.

[0054] Existing mechanisms typically employ a uniform reporting strategy for all detected anomalies, failing to quantify the actual impact of bottlenecks within the host on service performance. This lack of evaluation dimensions often leads to over-intervention by operations personnel when faced with momentary spikes (such as mistakenly isolating nodes) or delayed responses when faced with hidden, continuous degradation (canary-scale failures).

[0055] In this solution, a severity grading assessment model (P0-P2) based on business impact is constructed. By jointly calculating the impact range and persistence of the fault through multi-dimensional indicators, it provides an accurate step-by-step response basis for the automated operation and maintenance system, avoids over-response, and improves diagnostic accuracy.

[0056] To address the aforementioned technical problems, this application provides a link fault classification method, apparatus, device, medium, and program product. By using transmission parameters, the transmission status of the communication link can be obtained more accurately. Then, based on the transmission status of the communication link and the link anomaly conditions, the communication links that meet the link anomaly conditions can be identified. Furthermore, based on the first information of the communication link, the impact of the anomaly in the communication link is evaluated, and the fault level of the communication link is obtained based on the impact of the communication link anomaly. This enables the classification of fault levels based on the actual transmission status of the communication link, achieving refined management of communication link faults. This allows the operation and maintenance end to more clearly obtain the degree of anomaly of the communication link, improving the efficiency of handling link anomalies.

[0057] The following description, in conjunction with the accompanying drawings, details the link fault classification method, apparatus, device, medium, and program products provided in the embodiments of this application.

[0058] Figure 1 This illustration shows a network architecture for applying a link failure classification method according to an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a link fault classification device 101 and a terminal device 102. The link fault classification device 101 and the terminal device 102 are interconnected.

[0059] In some embodiments, the link fault classification device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster consisting of multiple servers. It should be noted that this application embodiment does not limit the specific device form of the link fault classification device 101. Figure 1 The link failure classification device 101 is illustrated using a single server as an example.

[0060] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.

[0061] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0062] See Figure 2 This is a flowchart illustrating a link fault classification method provided in an embodiment of this application. Figure 2 As shown, the link fault classification method provided in this application embodiment can be implemented by the above-mentioned link fault classification device, specifically including the following steps 201 to 203.

[0063] Step 201: The link fault classification device obtains the transmission parameters of the communication link between at least two hardware nodes.

[0064] In some embodiments of this application, the above transmission parameters include real-time throughput, link bandwidth, and link latency.

[0065] In some embodiments of this application, the link fault classification device can deploy a detection module to dynamically detect the data communication interface in the AI ​​accelerator runtime library and collect data communication information between hardware nodes in the host during the operation of intelligent computing tasks.

[0066] In some embodiments of this application, the link fault classification device can calculate the real-time throughput of the communication path between different hardware nodes and form communication flow record data.

[0067] In some embodiments of this application, the link fault classification device can actively detect and measure the bandwidth and latency of relevant communication paths to obtain the actual performance indicators of the link in the current state, including path bandwidth and latency information.

[0068] For example, a detection system is deployed in an Artificial Intelligence (AI) training server. The server includes AI accelerators: 8 x 910A Neural Processing Units (NPUs). The server is interconnected via a high-speed serial computer extended bus standard (Peripheral Component Interconnect Express, PCIe) 4.0+ cache coherency system high-speed interconnect. The AI ​​runtime environment of the server is AscendCL. During the training task, the link fault classification device obtains the data communication behavior between AI accelerators through the AscendComputing Language (AscendCL) runtime interface aclrtMemcpy, and records the communication data size, sending device, receiving device, and communication time information.

[0069] In some embodiments, combined with Figure 2 ,like Figure 3 As shown, step 201 above can be implemented through steps 201a and 201b.

[0070] Step 201a: The link fault classification device collects data communication information between at least two hardware nodes during the operation of the intelligent computing task.

[0071] In some embodiments of this application, the aforementioned data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time.

[0072] Step 201b: The link fault classification device obtains the real-time throughput, link bandwidth, and link latency of the communication link between at least two hardware nodes based on data communication information.

[0073] In some embodiments of this application, the link fault classification device can further calculate the instantaneous throughput of each link within the host based on the collected communication flow information.

[0074] In some embodiments of this application, the instantaneous throughput described above can characterize the actual data transmission rate of the communication link.

[0075] In some embodiments of this application, the link fault classification device can obtain the link bandwidth of each communication link through active detection.

[0076] In some embodiments of this application, the link bandwidth described above can characterize the maximum data transmission rate of the communication link.

[0077] In some embodiments of this application, the link fault classification device can obtain the link delay of each communication link based on the network transmission protocol.

[0078] In some embodiments of this application, the link delay described above is used to characterize the time spent transmitting data on the communication link.

[0079] In this way, by collecting multi-dimensional data, a more comprehensive evaluation of the communication link can be conducted, thereby enabling more accurate identification of abnormal information in the communication link.

[0080] It is understandable that a comprehensive evaluation of bottleneck links within the host can be conducted using multi-dimensional performance indicators, including: link bandwidth, link latency, duration of anomalies, proportion of anomalies occurring at different times, proportion of abnormal links, and changes in the throughput of intelligent computing tasks. By combining multiple indicators, a unified evaluation of communication link anomalies within the host can be performed, avoiding misjudgments caused by relying on only a single performance indicator.

[0081] Step 202: The link fault classification device determines N communication links that meet the link fault conditions from the communication links between at least two hardware nodes based on the transmission parameters.

[0082] In some embodiments of this application, N is a positive integer.

[0083] In some embodiments of this application, the link fault classification device can evaluate the communication link based on the transmission parameters of each communication link.

[0084] In some embodiments of this application, the link fault classification device can obtain abnormal conditions of the communication link based on the changes in throughput or abnormal changes in link utilization of each communication link.

[0085] In some embodiments, combined with Figure 2 ,like Figure 4 As shown, step 202 above can be implemented through steps 202a and 202b.

[0086] Step 202a: The link fault classification device determines M communication links from the communication links between at least two hardware nodes based on the real-time throughput of the communication links between at least two hardware nodes.

[0087] In some embodiments of this application, M is an integer greater than or equal to N.

[0088] In some embodiments of this application, the link fault classification device can calculate the link utilization rate by combining the theoretical bandwidth of the link to identify communication links that may have abnormal performance. When a decrease in link throughput or abnormal link utilization is detected, the link is marked as a suspected abnormal link.

[0089] In some embodiments of this application, the link fault classification device can locate the specific link with performance abnormalities by analyzing the correlation between different communication paths after obtaining communication flow monitoring data and link detection data.

[0090] For example, the link fault classification device can calculate the communication throughput between different hardware based on the collected communication data. For instance, if the following communication path is detected during a certain training phase: NPU3→Mem NUMA 3, the bandwidth of this path is about 200Gbps under normal circumstances, but it is found that the bandwidth of this path drops to 100Gbps. At the same time, multiple communication flows passing through the same PCIe switching path also show a similar drop, then the communication link is considered to be a suspected abnormal link.

[0091] In some embodiments, combined with Figure 4 ,like Figure 5 As shown, step 202a above can be implemented through the following steps 202a1 and 202a2.

[0092] Step 202a1: When the link fault classification device detects that the real-time throughput of any communication link between at least two hardware nodes is less than the real-time throughput at the first moment, it determines that any communication link is a suspected abnormal link.

[0093] In some embodiments of this application, the first moment mentioned above is the moment following the second moment mentioned above.

[0094] In some embodiments of this application, the link fault classification device can collect real-time throughput data of the communication link, and when it is found that the real-time throughput of the communication link shows a downward trend, the communication link is identified as a suspected abnormal link.

[0095] In some embodiments of this application, the aforementioned decreasing trend in real-time throughput can be that the values ​​of a consecutive real-time throughputs are decreasing.

[0096] Step 202a2: The link fault classification device identifies all suspected abnormal links from the communication links between at least two hardware nodes to obtain M communication links.

[0097] In some embodiments of this application, the link fault classification device can acquire all suspected abnormal links to obtain the aforementioned M communication links.

[0098] In some embodiments, combined with Figure 4 ,like Figure 6 As shown, step 202a above can be implemented through steps 202a3 to 202a5.

[0099] Step 202a3: The link fault classification device calculates the link utilization rate of the communication link between at least two hardware nodes based on the real-time throughput and theoretical bandwidth of the communication link between at least two hardware nodes.

[0100] In some embodiments of this application, the link fault classification device can determine the link utilization rate as a percentage of the ratio of real-time throughput to theoretical bandwidth.

[0101] Step 202a4: When the link fault classification device detects that the link utilization of any communication link between at least two hardware nodes is abnormal, it determines that any communication link is a suspected abnormal link.

[0102] In some embodiments of this application, the link fault classification device can detect whether the link utilization of any communication link is abnormal by means of a preset threshold.

[0103] In some embodiments of this application, the link fault classification device can identify communication links with abnormal link utilization as suspected abnormal links.

[0104] Step 202a5: The link fault classification device identifies all suspected abnormal links from the communication links between at least two hardware nodes to obtain M communication links.

[0105] In some embodiments of this application, the link fault classification device can acquire all suspected abnormal links to obtain the aforementioned M communication links.

[0106] Step 202b: The link fault classification device determines N communication links from the M communication links based on the link bandwidth and link delay of each of the M communication links.

[0107] In some embodiments of this application, the link bandwidth of N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

[0108] In some embodiments of this application, the link fault classification device can determine the above-mentioned N communication links by comparing and analyzing the measurement results of different paths, and use the relevant measurement results as data input for subsequent severity assessment.

[0109] In some embodiments of this application, the link fault classification device can determine that the communication link is a bottleneck link when the communication link meets any of the following conditions: the link bandwidth is lower than a preset bandwidth threshold; the link delay is higher than a preset delay threshold.

[0110] In some embodiments of this application, the link fault classification device can acquire all bottleneck links to obtain the aforementioned N communication links.

[0111] For example, the link fault classification device can trigger link performance detection to measure the bandwidth and latency of the communication path from NPU3 to other hardware. The detection results show that the bandwidth of multiple paths passing through the PCIe link is lower than the threshold, while the performance of paths not passing through the link is normal. Therefore, the PCIe link is identified as a bottleneck link within the host.

[0112] Step 203: The link fault classification device determines the fault level of each communication link based on the first information of each of the N communication links.

[0113] In some embodiments of this application, the first information mentioned above includes transmission parameters and link anomaly information.

[0114] In some embodiments of this application, the above transmission parameters may include link bandwidth, link latency, and real-time throughput.

[0115] In some embodiments of this application, the aforementioned real-time throughput may include: deep learning throughput and intelligent computing task throughput.

[0116] In some embodiments of this application, the above-mentioned link anomaly information may include: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

[0117] In some embodiments of this application, the link fault classification device can evaluate the impact of the first information of the communication link using multi-dimensional indicators and classify the fault according to the evaluation results.

[0118] In some embodiments of this application, the link fault classification device can preset a threshold for each item in the first information and compare each data item in the first information of the communication link with the preset threshold to obtain the degree of abnormality of the communication link.

[0119] In some embodiments of this application, the link fault classification device can determine the abnormal situation of the communication link based on whether the overall throughput of the intelligent computing task of the communication link has abnormally decreased.

[0120] In some embodiments of this application, the transmission parameters in the first information mentioned above include: link bandwidth, link latency, and intelligent computing task throughput.

[0121] In some embodiments of this application, the above-mentioned intelligent computing task throughput represents the throughput of the communication link during deep learning.

[0122] In some embodiments of this application, the link anomaly information in the first information mentioned above includes: link anomaly duration, link anomaly occurrence time ratio, and anomaly link ratio.

[0123] For example, combined Figure 2 ,like Figure 7 As shown, step 203 can be implemented by any one of steps 203a to 203c.

[0124] Step 203a: When the first communication link among the N communication links meets the first condition and the second condition, the link fault classification device determines the first communication link as the first fault level.

[0125] In some embodiments of this application, the first communication link mentioned above is any one of N communication links.

[0126] In some embodiments of this application, the first condition is that the link bandwidth is lower than the bandwidth threshold or the link latency is higher than the latency threshold.

[0127] In some embodiments of this application, the second condition includes at least one of the following: the duration of the abnormal link exceeds the duration threshold, the proportion of the abnormal link time exceeds the time proportion threshold, and the proportion of abnormal links exceeds the link proportion threshold.

[0128] In some embodiments of this application, when the link fault classification device obtains that the link bandwidth of the communication link is lower than the bandwidth threshold or the link delay is higher than the delay threshold, and the abnormal duration is greater than the preset abnormal duration threshold or the abnormal time ratio is greater than the abnormal time ratio threshold or the abnormal link ratio is greater than the abnormal link ratio threshold, the communication link is determined to be of the first fault level (also known as P0 level severe fault).

[0129] In some embodiments of this application, the first fault level mentioned above indicates that there is a persistent performance bottleneck in the communication link, which may seriously affect the execution efficiency of the intelligent computing task or cause the training task to fail.

[0130] For example, the link fault classification device can perform a severity assessment on any of the above N communication links. If it finds that the link bandwidth of the communication link is greater than the bandwidth threshold, the abnormal duration of the communication link is greater than the abnormal duration threshold, and the abnormal link ratio of the communication link is greater than the abnormal link ratio threshold, then the communication link is determined to be of the first fault level.

[0131] Step 203b: When the first communication link among the N communication links meets the first condition but does not meet the second condition, the link fault classification device determines the first communication link as the second fault level.

[0132] In some embodiments of this application, the first fault level is higher than the second fault level.

[0133] In some embodiments of this application, when the link fault classification device detects that the link bandwidth or link delay of the communication link is abnormal, but the duration or proportion of the abnormality does not reach the severe fault threshold, the communication link is determined to be of the second fault level.

[0134] In some embodiments of this application, the aforementioned second fault level (also known as P1 level fault) indicates that there is a performance abnormality in the communication link, but it has not yet caused a persistent and serious impact.

[0135] Step 203c: When the first communication link among the N communication links does not meet the first condition but meets the third condition, the link fault classification device determines the first communication link as the third fault level.

[0136] In some embodiments of this application, the third condition is that the throughput of the intelligent computing task at the third time point is less than the throughput of the intelligent computing task at the fourth time point, and the third time point is the time point following the fourth time point.

[0137] In some embodiments of this application, the second fault level is higher than the third fault level.

[0138] In some embodiments of this application, when the link fault classification device obtains that the link bandwidth and link latency of the communication link are not abnormal, but detects that the throughput of the intelligent computing task has dropped significantly, the communication link is determined to be at the third fault level (also known as the P2 level performance warning).

[0139] In some embodiments of this application, the aforementioned third fault level is typically caused by application layer factors such as task scheduling strategies, parallel configurations, or changes in communication modes, rather than by hardware link failures within the host.

[0140] For example, such as Figure 8 The diagram shows a schematic of the network topology within the host machine. It includes four Central Processing Units (CPUs), eight NPUs, and four DDR (Double Data Rate) memory modules. Each NPU is equipped with a high-speed network interface (QSFP56 package, 200Gbps). As shown, the CPUs and NPUs are interconnected via PCIe links, NPUs are connected via NPU interconnect links, CPUs are connected via CPU interconnect links, and the CPUs and DDRs are connected via memory channels. The NPUs are connected to their high-speed network interfaces via RDMA optical modules.

[0141] In this way, by classifying the fault levels of communication links, we can make differentiated judgments on abnormal conditions of the links, enabling more refined fault management and operation and maintenance decisions.

[0142] Understandably, after identifying bottleneck links within the host, the link fault classification device can further propose a classification assessment method based on the degree of link performance anomalies and their impact on intelligent computing tasks. This method classifies link anomalies into multiple severity levels, including: P0 level severe fault: abnormal link bandwidth or latency, and the duration or range of the anomaly exceeds a preset threshold; P1 level important fault: abnormal link bandwidth or latency, but the duration or range of the anomaly does not reach the severe fault threshold; and P2 level performance warning: normal link performance indicators, but an abnormal decrease in intelligent computing task throughput. This classification mechanism can differentiate the judgment based on the degree of impact of link anomalies on services, thereby achieving more refined fault management and operation and maintenance decisions.

[0143] The link fault classification method provided in this application can more accurately obtain the transmission status of the communication link through transmission parameters. Then, based on the transmission status of the communication link and the abnormal conditions of the link, the communication link that meets the abnormal conditions can be obtained. Then, based on the first information of the communication link, the impact of the abnormality of the communication link is evaluated, and the fault level of the communication link is obtained based on the impact of the abnormality of the communication link. This realizes the classification of fault levels based on the actual transmission status of the communication link, achieving refined management of communication link faults. This allows the operation and maintenance end to more clearly obtain the degree of abnormality of the communication link and improves the efficiency of handling link abnormalities.

[0144] In some embodiments, combined with Figure 2 ,like Figure 9 As shown, after step 203 above, the link fault classification method provided in this application embodiment may further include the following steps 301 or 302.

[0145] Step 301: When the first communication link is at the first or second fault level, the link fault classification device reports the fault event of the first communication link to the intelligent computing center network management system for processing.

[0146] In some embodiments of this application, the link fault classification device can generate corresponding fault event information based on the fault classification results of the communication link.

[0147] In some embodiments of this application, the link fault classification device can report the fault event to the data center network management system for unified processing when the first communication link is at the first fault level or the second fault level.

[0148] Step 302: When the first communication link is at the third fault level, the link fault classification device feeds back the fault-related information of the first communication link to the application-side device, so that the application-side device can optimize the task and tune the performance of the first communication link based on the fault-related information.

[0149] In some embodiments of this application, the link fault classification device can feed back relevant information to the application side when the first communication link is at the third fault level, for task optimization and performance tuning.

[0150] In this way, by implementing different response strategies for communication lines with different fault levels, refined handling of faulty communication lines is achieved, thereby improving the efficiency of communication line operation and maintenance.

[0151] Understandably, different response strategies are executed based on the severity level of the fault events. These strategies include: uniformly reporting severe and important faults and triggering the data center network management system for further analysis and processing; and feeding back performance warning issues to the application side for task optimization. Through this hierarchical response mechanism, intelligent handling of communication faults within the host can be achieved, thereby improving the operational efficiency of the AI ​​training system.

[0152] The link fault classification method of this application will be described below through specific embodiments.

[0153] The implementation process of the link fault classification method provided in this application includes the following S1 to S4:

[0154] This solution analyzes the performance status of communication links within the host and, in conjunction with the characteristics of link anomalies, duration, and impact on intelligent computing tasks, classifies and assesses faults, thereby achieving intelligent diagnosis and graded response to bottleneck links within the host.

[0155] For example, such as Figure 10 The diagram illustrates a flowchart of intelligent response and root cause tracing based on severity grading. The link fault grading device can collect performance data of communication paths within the host. By recording flow data, identifying suspected links, and performing active probing, it completes the collection of performance data of communication paths within the host. Then, based on the performance data of communication paths within the host, it identifies bottleneck links within the host, determining links with excessively low bandwidth, excessively high latency, and persistently abnormal throughput as bottleneck links. Subsequently, based on the above bottleneck links, it performs a severity grading assessment of the bottleneck links within the host. When abnormal indicators are detected, the bottleneck links are graded according to the impact range of the abnormal indicators and the actual operating effect, obtaining the grading assessment results. Finally, based on the grading assessment results of the links, it generates graded fault times and executes intelligent response.

[0156] The specific steps are as follows:

[0157] Step S1: The link fault classification device collects communication path performance data within the host and performs performance detection on suspected abnormal links.

[0158] A monitoring module is deployed in the host system to dynamically monitor the data communication interface in the AI ​​accelerator runtime library. It collects data communication information between various hardware nodes in the host during the operation of intelligent computing tasks, including but not limited to information such as data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time. It also calculates the real-time throughput of the communication path between different hardware nodes and forms communication flow record data.

[0159] Based on the collected communication flow information, the system further analyzes the instantaneous throughput of each link within the host and calculates the link utilization rate in conjunction with the theoretical bandwidth of the links to identify communication links that may have performance anomalies. When a decrease in link throughput or abnormal link utilization is detected, the link is marked as a suspected abnormal link.

[0160] For suspected abnormal links, the system actively probes the relevant communication paths to measure bandwidth and latency, obtaining the link's true performance indicators under its current state, including path bandwidth and latency information. By comparing and analyzing the measurement results of different paths, the system determines whether there are bottleneck links within the host communication network, and uses the relevant measurement results as data input for subsequent severity assessment.

[0161] Step S2: The link fault classification device identifies the bottleneck link within the host.

[0162] In some embodiments of this application, after obtaining communication flow monitoring data and link detection data, the link fault classification device locates the specific link location with performance abnormality by analyzing the correlation between different communication paths.

[0163] In some embodiments of this application, when a link meets any of the following conditions, the link fault classification device determines that the communication link is a bottleneck link within the host: the link bandwidth is lower than a preset bandwidth threshold; the link latency is higher than a preset latency threshold.

[0164] In some embodiments of this application, the link fault classification device further enters the severity assessment process for the identified bottleneck links.

[0165] Step S3: The link fault classification device performs a severity classification assessment of the bottleneck link within the host.

[0166] In some embodiments of this application, the link fault classification device assesses the impact of identified bottleneck links within a host using multi-dimensional indicators and classifies the faults based on the assessment results. The assessment indicators include: whether the link bandwidth is lower than a bandwidth threshold. Is the link latency higher than the latency threshold? Does the abnormal duration exceed the duration threshold? Does the proportion of time when an anomaly occurs exceed the time proportion threshold? Does the proportion of abnormal links within the host exceed the link proportion threshold? ; Has the overall throughput of intelligent computing tasks experienced an abnormal decline?

[0167] In some embodiments of this application, the link fault classification device constructs a severity classification mechanism, including: a first fault level (i.e., P0 level severe fault), a second fault level (i.e., P1 level important fault), and a third fault level (i.e., P2 level performance warning).

[0168] In some embodiments of this application, when a bottleneck link simultaneously meets the following conditions, the link fault classification device determines that the bottleneck link is a P0 level severe fault: the link bandwidth is lower than the bandwidth threshold or the link delay is higher than the delay threshold; and the abnormal duration, abnormal time ratio or abnormal link ratio meets at least one threshold condition.

[0169] In some embodiments of this application, a P0-level fault indicates a persistent performance bottleneck in the critical communication link within the host, which may severely affect the execution efficiency of intelligent computing tasks or cause training tasks to fail.

[0170] In some embodiments of this application, when a bottleneck link meets the following conditions, the link fault classification device determines that the bottleneck link is a P1 level critical fault: the link bandwidth or link latency is abnormal; but the duration or proportion of the abnormality does not reach the critical fault threshold.

[0171] In some embodiments of this application, a P1 level fault indicates that there is a performance abnormality in the intra-host link, but it has not yet caused a persistent and serious impact.

[0172] In some embodiments of this application, when neither the link bandwidth nor the link latency is abnormal, but the link fault classification device detects a significant decrease in the throughput of the intelligent computing task, the link fault classification device determines that the bottleneck link is a P2 level performance warning.

[0173] In some embodiments of this application, the faults corresponding to P2 level performance warnings are usually caused by application layer factors such as task scheduling strategies, parallel configurations, or changes in communication modes, rather than hardware link faults within the host.

[0174] Step S4: The link fault classification device generates classified fault events and executes intelligent responses.

[0175] In some embodiments of this application, the link fault classification device can generate corresponding fault event information and execute different levels of response strategies based on the fault classification results of the communication link.

[0176] In some embodiments of this application, the link fault classification device can report the fault event to the data center network management system for unified processing when it detects that the communication link is a P0 level fault or a P1 level fault.

[0177] In some embodiments of this application, the link fault classification device can detect a P2 level warning for the communication link and feed back relevant information to the application side for task optimization and performance tuning.

[0178] For example, a system is deployed in an AI training server with the following configuration: AI accelerator: 910 ANPU × 8; Interconnection method: PCIe 4.0 + cache coherence system high-speed interconnect; AI runtime environment: AscendCL. During the training task, the link fault classification device detects the data communication behavior between AI accelerators by monitoring the AscendCL runtime interface aclrtMemcpy, and records the communication data size, sending device, receiving device, and communication time information. The link fault classification device calculates the communication throughput between different hardware based on the collected communication data. For example, during a certain training phase, the following communication path is detected: NPU3 → MemNUMA3. Under normal circumstances, the bandwidth of this path is about 200Gbps, but during monitoring, it is found that the bandwidth of this path drops to 100Gbps, and multiple communication flows passing through the same PCIe switching path also show a similar decrease. The link fault classification device then triggers link performance detection to measure the bandwidth and latency of the communication path from NPU3 to other hardware. The detection results show that the bandwidth of multiple paths passing through this PCIe link is below the threshold, while the performance of paths not passing through this link is normal; therefore, the link fault classification device identifies this PCIe link as a bottleneck link within the host.

[0179] Subsequently, the link fault classification device performs a severity assessment: the link bandwidth is lower than the bandwidth threshold, the abnormal duration exceeds the preset threshold, and the proportion of affected communication paths exceeds the system threshold. According to the severity assessment rules, the link fault classification device determines the fault to be a P0 level severe fault, generates a corresponding fault event, and reports it to the data center network management system for unified processing.

[0180] In this way, the above mechanism can intelligently identify communication bottlenecks within the host and implement hierarchical management based on the degree of impact of the fault, thereby improving the stability and maintainability of large-scale AI training environments.

[0181] It should be noted that the descriptions of each step S1 to S4 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0182] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0183] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] This application embodiment can divide the link fault classification device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0185] In some embodiments, this application also provides a link fault classification device. This link fault classification device may include one or more functional modules for implementing the link fault classification method of the above method embodiments.

[0186] For example, Figure 11 This is a schematic diagram of a link fault classification device provided in an embodiment of this application. Figure 11 As shown, the link fault classification device 900 includes: an acquisition module 901; wherein, the acquisition module 901 is used to acquire transmission parameters of the communication link between at least two hardware nodes, the transmission parameters including real-time throughput, link bandwidth and link latency; the acquisition module 901 is used to determine N communication links that meet the link anomaly conditions from the communication links between the at least two hardware nodes based on the transmission parameters, where N is a positive integer; the acquisition module 901 is also used to determine the fault level of each communication link based on first information of each of the N communication links, the first information including transmission parameters and link anomaly information.

[0187] In some embodiments, the acquisition module 901 is specifically used to: determine M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes, where M is an integer greater than or equal to N; and determine N communication links from the M communication links based on the link bandwidth and link delay of each of the M communication links, where the link bandwidth of the N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

[0188] In other embodiments, the acquisition module 901 is specifically used to: when it is detected that the real-time throughput of any communication link between the at least two hardware nodes is less than the real-time throughput at the first moment, determine that the communication link is a suspected abnormal link, and the first moment is the next moment after the second moment; determine all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0189] In some other embodiments, the apparatus further includes: a calculation module; the calculation module is configured to calculate the link utilization of the communication link between the at least two hardware nodes based on the real-time throughput and theoretical bandwidth of the communication link between the at least two hardware nodes; the acquisition module 901 is specifically configured to determine any communication link as a suspected abnormal link when an abnormal link utilization is detected in any of the communication links between the at least two hardware nodes; and to determine all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

[0190] In some other embodiments, the acquisition module 901 is specifically used to: collect data communication information between at least two hardware nodes during the operation of the intelligent computing task, wherein the data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time; and based on the data communication information, acquire the real-time throughput, link bandwidth, and link latency of the communication link between the at least two hardware nodes.

[0191] In some other embodiments, the link anomaly information in the first information mentioned above includes: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

[0192] In some other embodiments, the acquisition module 901 is specifically configured to: determine the first communication link as a first fault level when the first communication link among the N communication links meets the first condition and the second condition; determine the first communication link as a second fault level when the first communication link among the N communication links meets the first condition but does not meet the second condition; determine the first communication link as a third fault level when the first communication link among the N communication links does not meet the first condition but meets the third condition; wherein the first condition is that the link bandwidth is lower than the bandwidth threshold or the link latency is higher than the latency threshold; the second condition includes at least one of the following: the duration of the abnormal link exceeds the duration threshold, the proportion of the abnormal link time exceeds the time proportion threshold, and the proportion of abnormal links exceeds the link proportion threshold; the third condition is that the real-time throughput at the third time is less than the real-time throughput at the fourth time, and the third time is the time after the fourth time; the first fault level is higher than the second fault level, and the second fault level is higher than the third fault level.

[0193] In some other embodiments, the above-described apparatus further includes: a sending module; the sending module is configured to report the fault event of the first communication link to the data center network management system for processing when the first communication link is at the first fault level or the second fault level; the sending module is further configured to feed back the fault-related information of the first communication link to the application-side device when the first communication link is at the third fault level, so that the application-side device can perform task optimization and performance tuning of the first communication link based on the fault-related information.

[0194] In the link fault classification device provided in this application, the transmission status of the communication link can be obtained more accurately through transmission parameters. Then, based on the transmission status of the communication link and the abnormal conditions of the link, the communication links that meet the abnormal conditions can be obtained. Then, based on the first information of the communication link, the impact of the abnormality of the communication link is evaluated, and the fault level of the communication link is obtained based on the impact of the abnormality of the communication link. This realizes the classification of fault levels based on the actual transmission status of the communication link, achieving refined management of communication link faults. This allows the operation and maintenance end to more clearly obtain the degree of abnormality of the communication link, improving the efficiency of handling link abnormalities.

[0195] It should be noted that the link fault classification device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0196] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 12 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.

[0197] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0198] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0199] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0200] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the link fault classification method provided in the embodiments of this application.

[0201] In another possible implementation, memory 91 can also be integrated with processor 92.

[0202] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0203] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0204] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described link fault classification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0205] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0206] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the link fault classification method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a pluggable hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0207] This application also provides a computer program product, which is stored in a storage medium and implements the link fault classification method provided in the above embodiments when the computer program product is executed by a computer.

[0208] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0210] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A link fault classification method, characterized in that, include: Obtain the transmission parameters of the communication link between at least two hardware nodes, including real-time throughput, link bandwidth, and link latency; Based on the transmission parameters, N communication links that meet the link anomaly conditions are determined from the communication links between the at least two hardware nodes, where N is a positive integer; Based on the first information of each of the N communication links, the fault level of each communication link is determined, wherein the first information includes transmission parameters and link anomaly information.

2. The link fault classification method according to claim 1, characterized in that, The step of determining N communication links that meet the link anomaly conditions from the communication links between the at least two hardware nodes based on the transmission parameters includes: Based on the real-time throughput of the communication links between the at least two hardware nodes, M communication links are determined from the communication links between the at least two hardware nodes, where M is an integer greater than or equal to N. Based on the link bandwidth and link delay of each of the M communication links, N communication links are determined from the M communication links, wherein the link bandwidth of the N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

3. The link fault classification method according to claim 2, characterized in that, The step of determining M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes includes: When it is detected that the real-time throughput of any communication link between the at least two hardware nodes is less than the real-time throughput at the second time in the first time, the communication link is determined to be a suspected abnormal link, and the first time is the time after the second time. All suspected abnormal links are identified from the communication links between the at least two hardware nodes to obtain M communication links.

4. The link fault classification method according to claim 2, characterized in that, The step of determining M communication links from the communication links between the at least two hardware nodes based on the real-time throughput of the communication links between the at least two hardware nodes includes: Based on the real-time throughput and theoretical bandwidth of the communication link between the at least two hardware nodes, calculate the link utilization of the communication link between the at least two hardware nodes. When abnormal link utilization is detected in any communication link between the at least two hardware nodes, the communication link is determined to be a suspected abnormal link. All suspected abnormal links are identified from the communication links between the at least two hardware nodes to obtain M communication links.

5. The link fault classification method according to any one of claims 1 to 4, characterized in that, The acquisition of transmission parameters for the communication link between at least two hardware nodes includes: Collect data communication information between at least two hardware nodes during the operation of the intelligent computing task. The data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time. Based on the data communication information, the real-time throughput, link bandwidth, and link latency of the communication link between the at least two hardware nodes are obtained.

6. The link fault classification method according to claim 1, characterized in that, The link anomaly information in the first information includes: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

7. The link fault classification method according to claim 6, characterized in that, The step of determining the fault level of each communication link based on the first information of each of the N communication links includes: When the first communication link among the N communication links meets the first condition and the second condition, the first communication link is determined to be of the first fault level. When the first communication link among the N communication links meets the first condition but does not meet the second condition, the first communication link is determined to be of the second fault level. When the first communication link among the N communication links does not meet the first condition but meets the third condition, the first communication link is determined to be at the third fault level. The first condition is that the link bandwidth is lower than the bandwidth threshold or the link latency is higher than the latency threshold; the second condition includes at least one of the following: the duration of the abnormal link exceeds the duration threshold, the proportion of the abnormal link time exceeds the time proportion threshold, and the proportion of abnormal links exceeds the link proportion threshold; the third condition is that the real-time throughput at the third time is less than the real-time throughput at the fourth time, and the third time is the time after the fourth time. The first fault level is higher than the second fault level, and the second fault level is higher than the third fault level.

8. The link fault classification method according to claim 7, characterized in that, The method further includes: If the first communication link is at the first fault level or the second fault level, the fault event of the first communication link shall be reported to the intelligent computing center network management system for processing. When the first communication link is at the third fault level, the fault-related information of the first communication link is fed back to the application-side hardware, so that the application-side hardware can perform task optimization and performance tuning of the first communication link based on the fault-related information.

9. A link fault classification device, characterized in that, include: Get the module; The acquisition module is used to acquire the transmission parameters of the communication link between at least two nodes, including real-time throughput, link bandwidth and link latency. The acquisition module is used to determine, based on the transmission parameters, N communication links that satisfy the link anomaly conditions from the communication links between the at least two nodes, where N is a positive integer; The acquisition module is further configured to determine the fault level of each communication link based on the first information of each of the N communication links, wherein the first information includes transmission parameters and link anomaly information.

10. The link fault classification device according to claim 9, characterized in that, The acquisition module is specifically used for: Based on the real-time throughput of the communication links between the at least two hardware nodes, M communication links are determined from the communication links between the at least two hardware nodes, where M is an integer greater than or equal to N. Based on the link bandwidth and link delay of each of the M communication links, N communication links are determined from the M communication links, wherein the link bandwidth of the N communication links is less than a preset bandwidth threshold or the link delay is greater than a preset delay threshold.

11. The link fault classification device according to claim 10, characterized in that, The acquisition module is specifically used for: When it is detected that the real-time throughput of any communication link between the at least two hardware nodes is less than the real-time throughput at the second time in the first time, the communication link is determined to be a suspected abnormal link, and the first time is the time after the second time. All suspected abnormal links are identified from the communication links between the at least two hardware nodes to obtain M communication links.

12. The link fault classification device according to claim 10, characterized in that, The device further includes: a computing module; The calculation module is used to calculate the link utilization of the communication link between the at least two hardware nodes based on the real-time throughput and theoretical bandwidth of the communication link between the at least two hardware nodes. The acquisition module is specifically configured to determine any communication link as a suspected abnormal link when an abnormal link utilization is detected in any communication link between the at least two hardware nodes; and to determine all suspected abnormal links from the communication links between the at least two hardware nodes to obtain M communication links.

13. The link fault classification device according to any one of claims 9 to 12, characterized in that, The acquisition module is specifically used for: Collect data communication information between at least two hardware nodes during the operation of the intelligent computing task. The data communication information includes at least one of the following: data sending node, data receiving node, data transmission size, data transmission start time, and data transmission end time. Based on the data communication information, the real-time throughput, link bandwidth, and link latency of the communication link between the at least two hardware nodes are obtained.

14. The link fault classification device according to claim 9, characterized in that, The link anomaly information in the first information includes: link anomaly duration, link anomaly occurrence time ratio, and abnormal link ratio.

15. The link fault classification device according to claim 14, characterized in that, The acquisition module is specifically used for: When the first communication link among the N communication links meets the first condition and the second condition, the first communication link is determined to be of the first fault level. When the first communication link among the N communication links meets the first condition but does not meet the second condition, the first communication link is determined to be of the second fault level. When the first communication link among the N communication links does not meet the first condition but meets the third condition, the first communication link is determined to be at the third fault level. The first condition is that the link bandwidth is lower than the bandwidth threshold or the link latency is higher than the latency threshold; the second condition includes at least one of the following: the duration of the abnormal link exceeds the duration threshold, the proportion of the abnormal link time exceeds the time proportion threshold, and the proportion of abnormal links exceeds the link proportion threshold; the third condition is that the real-time throughput at the third time is less than the real-time throughput at the fourth time, and the third time is the time after the fourth time. The first fault level is higher than the second fault level, and the second fault level is higher than the third fault level.

16. The link fault classification device according to claim 15, characterized in that, The device further includes: a transmitting module; The sending module is used to report the fault event of the first communication link to the intelligent computing center network management system for processing when the first communication link is at the first fault level or the second fault level. The sending module is further configured to, when the first communication link is at the third fault level, feed back the fault-related information of the first communication link to the application-side hardware, so that the application-side hardware can perform task optimization and performance tuning of the first communication link based on the fault-related information.

17. An electronic hardware device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the link fault classification method as described in any one of claims 1-8.

18. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the link fault classification method as described in any one of claims 1-8.

19. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the link fault classification method as described in any one of claims 1-8.

Images