Network testing method and device of computing power cluster

Abstract

The application relates to the technical field of computer network testing, and in particular to a network testing method and device for a computing power cluster, which comprises the following steps: constructing a weighted undirected graph structure according to obtained network information of the computing power cluster; generating a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge by using a preset graph theory algorithm; and testing the computing power cluster according to each test path included in the test path set. Through the innovative construction process of the graph structure and the test path generation strategy, the application can realize fast, accurate and intelligent health diagnosis of the network infrastructure of the computing power cluster, and ensure that the network can meet the strict requirements of extremely low delay, extremely high bandwidth and zero packet loss for large-scale distributed AI training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer network testing, and particularly relates to a network testing method and device for a computing power cluster. BACKGROUND

[0002] With the explosive growth of the demand for computing power for artificial intelligence training tasks, computing power clusters of the order of ten thousand cards or even one hundred thousand cards have become mainstream infrastructure. Such large-scale computing power clusters are highly dependent on high-performance, low-latency, and zero-packet-loss network interconnection to support collective communication operations such as All-Reduce.

[0003] The existing network testing method is that an operation and maintenance personnel logs in to a device through SSH and manually tests the connectivity and bandwidth of a computing power cluster node by node and link by link using basic network tools such as ping, traceroute, and iperf3. This results in that the traditional network testing method cannot meet the efficient, accurate, and lightweight operation and maintenance requirements of the computing power cluster.

[0004] Therefore, there is an urgent need for a network verification method for a super-large computing power cluster, so as to efficiently, accurately, and quickly test the network of the computing power cluster. SUMMARY

[0005] The present application provides a network testing method and device to solve the problem that the existing network testing method relies on manual operation, repeated link testing, and interference between test links, resulting in low testing efficiency and inaccurate testing results.

[0006] In a first aspect, an embodiment of the present application provides a network testing method for a computing power cluster, the method comprising:

[0007] According to the obtained network information of the computing power cluster, a weighted undirected graph structure is constructed, wherein the network information carries network configuration information of each computing node in the computing power cluster and communication link information of the computing power cluster; a node of the graph structure represents a computing node in the computing power cluster, and an edge of the graph structure represents that the computing nodes corresponding to two nodes have a communication link in the computing power cluster, and an initial weight of the edge is determined based on network configuration information of the computing nodes corresponding to the two nodes.

[0008] A preset graph algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0009] According to each test path included in the test path set, the computing power cluster is tested.

[0010] Further, the graph algorithm includes a single-source shortest path algorithm and a minimum spanning tree algorithm.

[0011] The preset graph algorithm is used to generate each test path according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0012] Each first computing node of the Spine layer in the computing cluster takes each first node in the graph structure as a source node, each second computing node of the Leaf layer in the computing cluster takes each second node in the graph structure as a destination node, and the single-source shortest path algorithm is used to generate a first test path from each first node to each second node according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0013] Each second node is taken as a source node, and each third computing node of the access layer in the computing cluster takes each third node in the graph structure as a destination node, and the single-source shortest path algorithm is used to generate a second test path from each second node to each third node according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0014] The minimum spanning tree algorithm is used to generate a third test path according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0015] The first test path, the second test path and the third test path are determined as the test path set.

[0016] Further, the method further comprises:

[0017] Obtaining a test result corresponding to each test path, the test result being an actual performance index value of the test path.

[0018] According to the relationship between the actual performance index value corresponding to each test path and the preset performance index threshold, the weight of each edge corresponding to each test path in the graph structure is updated.

[0019] The graph algorithm and the updated graph structure are used to update the test path set, and the updated test path set is used for testing until a preset test completion condition is met.

[0020] Further, the updating of the weight of each edge corresponding to each test path in the graph structure according to the relationship between the actual performance index value corresponding to each test path and the preset performance index threshold comprises:

[0021] For each link in each test path, if the actual performance indicator value of the link meets the requirement of the preset performance indicator threshold, the weight of the edge corresponding to the link in the graph structure is increased by a preset penalty value; if the actual performance indicator value of the link does not meet the requirement of the preset performance indicator threshold, the weight of the edge corresponding to the link in the graph structure is updated according to a preset attenuation factor.

[0022] Further, the actual performance indicator value is a measured bandwidth and / or a packet loss rate.

[0023] Further, after the weight of the edge corresponding to the link in the graph structure is updated according to the preset attenuation factor, the method further comprises:

[0024] For each edge corresponding to each test path in the graph structure, if the updated weight of the edge is lower than a preset minimum weight, the weight of the edge is updated to the minimum weight.

[0025] Further, after the weight of the edge corresponding to the link in the graph structure is increased by the preset penalty value, the method further comprises:

[0026] updating the saved number of abnormalities corresponding to the link;

[0027] If the updated number of abnormalities reaches a preset number of abnormality threshold, passive monitoring data corresponding to the link is obtained, the passive monitoring data including sFlow / NetFlow flow sampling, switch port cyclic redundancy check (CRC) error count, forward error correction (FEC) statistics, optical module received optical power, and digital diagnostic monitoring (DDM) alarm information.

[0028] An abnormality detection model is used to detect abnormalities of the link according to the passive monitoring data, and determine a confidence score corresponding to the link.

[0029] If the confidence score exceeds a preset threshold, an alarm is given. Further, the testing of the computing power cluster according to each test path included in the test path set comprises:

[0030] According to the network corresponding to each test path included in the test path set, the test path set is divided into a plurality of test path subsets;

[0031] For each test path subset, each test path including a non-shared link in the test path subset is determined; a preset test command is used to schedule each test path of the non-shared link in parallel.

[0032] Further, the use of a preset test command to schedule each test path of the non-shared link in parallel comprises:

[0033] For each test path of the share-free link, at least one of the following is used for testing:

[0034] The test path is tested for delay and packet loss using a ping command or an hping3 command;

[0035] The test path is tested for bandwidth using an iperf3 command to perform a point-to-point bandwidth throughput test;

[0036] The test path is tested for load using an NVIDIA Collective Communication Library (NCCL).

[0037] In a second aspect, the embodiments of the present application also provide an electronic device, which includes a processor configured to implement the steps of the network testing method of the computing power cluster according to any of the above embodiments when executing a computer program stored in a memory.

[0038] In the embodiments of the present application, a weighted undirected graph structure is constructed according to the obtained network information of the computing power cluster, wherein the network information carries network configuration information of each computing node in the computing power cluster and communication link information of the computing power cluster; a node of the graph structure represents a computing node in the computing power cluster, an edge of the graph structure represents that the two nodes corresponding to the computing nodes connected by the edge have a communication link in the computing power cluster, and an initial weight of the edge is determined based on the network configuration information of the two nodes corresponding to the computing nodes; a preset graph algorithm is used to generate a test path set according to each edge of each node in the graph structure and the initial weight of each edge; and the computing power cluster is tested according to each test path included in the test path set. Through the innovative graph structure construction and test path generation strategy, the embodiments of the present application realize fast, accurate and intelligent health diagnosis of the network infrastructure of the computing power cluster, and ensure that the network can meet the strict requirements of extremely low delay, extremely high bandwidth and zero packet loss for large-scale distributed AI training. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0040] Figure 1 A network testing process diagram of a computing power cluster is provided for the embodiments of the present application;

[0041] Figure 2 A flowchart of dynamic weight adjustment and feedback closed loop is provided for the embodiments of the present application;

[0042] Figure 3 A system architecture schematic diagram is provided for the embodiments of the present application.

[0043] Figure 4 A network test overall flowchart is provided for the embodiments of the present application.

[0044] Figure 5 A network test device structure schematic diagram of a computing power cluster is provided for the embodiments of the present application.

[0045] Figure 6 An electronic device structure schematic diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. The embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily without conflict. Moreover, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0047] The terms “first” and “second” in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term “comprise” and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. “Multiple” in the present application can mean at least two, for example, can be two, three or more, and the embodiments of the present application are not limited.

[0048] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Specific details of the present embodiments are set forth for the purpose of providing a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the embodiments described herein can be practiced without many of the specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the present embodiments. It should be noted that, in the present embodiments, some software, components, models, etc. that are available in the industry can be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or must have used the scheme.

[0049] With the explosive growth of computing power demand for artificial intelligence training tasks, ten thousand card level or even one hundred thousand card level computing power clusters have become mainstream infrastructure. Such large-scale computing power clusters are highly dependent on high-performance, low-latency, zero-packet loss network interconnection to support collective communication operations such as All-Reduce.

[0050] The current network verification of AI computing power clusters mainly relies on three types of technical solutions:

[0051] 1. Traditional manual inspection and point-by-point testing:

[0052] The operation and maintenance personnel log in to the device through the Secure Shell (SSH) protocol, and use ping, traceroute, iperf3, and other basic network tools to manually test the connectivity and bandwidth of each node and link. In some scenarios, scripts are written to implement semi-automatic testing, but the essence is still in traversal or random sampling mode.

[0053] 2. Commercial network test platform:

[0054] Using professional test equipment such as Keysight Ixia and Spirent TestCenter, high-precision test traffic is generated between selected nodes through dedicated hardware or software agents to simulate various network load conditions and measure key performance indicators such as throughput, delay, and packet loss rate.

[0055] 3. Academic research verification framework:

[0056] High-performance computing (HPC) benchmarking tools such as Microsoft SuperBench use topology-aware technology and probabilistic models to quickly scan the cluster through the design of optimized test case sets to identify performance abnormal nodes or network areas.

[0057] The above existing technologies have the following significant defects:

[0058] 1. Traditional manual inspection and point-by-point testing: extremely inefficient and incomplete coverage. In a multi-card cluster, the number of nodes and links grows exponentially (complexity O(N²)). Point-by-point testing requires executing tens of millions of test tasks, taking days or even weeks, and consuming a large amount of network resources, severely impacting normal AI training operations. Manual operation is prone to misjudgment or omissions due to human factors, and cannot meet the AI ​​cluster's need for rapid fault response.

[0059] 2. Commercial Network Testing Platforms: Resource-intensive and Poorly Adaptable. While commercial testing tools offer comprehensive functionality, they require the deployment of numerous dedicated hardware or proxies, consuming valuable rack space and power resources. Their "full-mesh" testing mode generates massive traffic that severely disrupts production operations. More critically, their predefined traffic models differ from AI-trained real-world cluster communication patterns (such as NCCL's Ring and Tree algorithms), resulting in insufficient accuracy of test results and a lack of dynamic adjustment capabilities, making it difficult to adapt to rapid changes in cluster topology.

[0060] 3. Academic Research-Based Validation Framework: Limited Positioning Accuracy. While research-based tools offer improved efficiency, they primarily focus on quickly identifying abnormal areas, lacking in-depth diagnostic and precise positioning capabilities. Their testing strategies are typically based on fixed patterns, unable to intelligently adjust subsequent validation paths based on real-time test results. They also lack the ability to isolate and detour faulty links, resulting in a mean time between failures (MTTR) that remains as high as several hours.

[0061] In summary, none of the existing solutions have effectively resolved the contradiction between "testing efficiency" and "verification integrity," and they cannot achieve routine, lightweight, and automated network health monitoring. This results in AI clusters experiencing training task interruptions due to minor network faults (such as optical module contamination or single-channel failure), leading to a huge waste of computing resources.

[0062] Based on this, this application proposes a network testing method for a computing power cluster. A weighted undirected graph structure is constructed based on the obtained network information of the computing power cluster. The network information carries network configuration information for each computing node in the computing power cluster and communication link information of the computing power cluster. Nodes in the graph structure represent computing nodes in the computing power cluster, and edges in the graph structure represent two connected computing nodes whose corresponding computing nodes have a communication link in the computing power cluster. The initial weight of the edge is determined based on the network configuration information of the two connected computing nodes. A preset graph theory algorithm is used to generate a set of test paths based on each edge corresponding to each node in the graph structure and the initial weight of each edge. The computing power cluster is then tested based on each test path included in the set of test paths.

[0063] The application is particularly suitable for verifying the network topology connection correctness and bandwidth performance of a GPU / AI acceleration card cluster of a million or even ten million cards, and aims to solve the key pain points such as low efficiency, huge resource consumption and difficult problem positioning of the traditional network test method in such a large-scale, high-complexity and high-reliability AI training scene. Through the innovative graph modeling and dynamic feedback mechanism, the application realizes fast, accurate and intelligent health diagnosis of the AI computing cluster network infrastructure, and ensures that the network can meet the stringent requirements of extremely low delay, extremely high bandwidth and zero packet loss for large-scale distributed AI training.

[0064] Embodiment 1:

[0065] Figure 1 A network test process diagram of a computing cluster is provided for the embodiments of the application, and the process includes:

[0066] S101: According to the obtained network information of the computing cluster, a weighted undirected graph structure is constructed, wherein the network information carries the network configuration information of each computing node in the computing cluster and the communication link information of the computing cluster; the nodes of the graph structure represent the computing nodes in the computing cluster, and the edges of the graph structure represent that the corresponding computing nodes of the two nodes have a communication link in the computing cluster, and the initial weight of the edge is determined based on the network configuration information of the corresponding computing nodes of the two nodes.

[0067] The network test method of the computing cluster provided by the embodiments of the application is applied to a management server, the management server is configured with a control plane, and the management server runs outside the computing cluster, is responsible for path calculation, policy scheduling and result analysis, and needs to have a global topology view and strong computing power.

[0068] In the embodiments of the application, the computing cluster can be abstracted as a graph structure, which can be a weighted undirected graph , wherein V represents a node V in the graph structure, the nodes in the graph structure correspond one-to-one to the computing nodes in the computing cluster, the computing nodes in the computing cluster can be a representative server and a switch, E represents an edge in the graph structure, and the edges in the graph structure correspond one-to-one to the physical links in the computing cluster.

[0069] Specifically, the management server can obtain network information of the computing cluster, and the network information carries network configuration information of each computing node in the computing cluster and communication link information of the computing cluster. The management server constructs a graph structure according to the communication link information and each computing node contained in the computing cluster, and calculates the weight of each edge in the graph structure according to the network configuration information of the computing node.

[0070] In the graph structure, the two computing nodes corresponding to the nodes existing the communication link are connected through the edge.

[0071] In a possible real-time manner, a management server obtains the neighbor relationship and the interface state of the computing nodes included in a Link Layer Discovery Protocol (LLDP) / Cisco Discovery Protocol (CDP) through polling of a Management Information Base (MBI) library of a switch by using a Simple Network Management Protocol (SNMP); the management server obtains a global topology view through a Software-Defined Networking Controller (SDN) northbound Application Programming Interface (API); and the management server obtains local network configuration information of each computing node through a lightweight agent deployed on the computing node.

[0072] Specifically, in the embodiment of the application, the management server obtains a network topology graph of the computing power cluster from the SDN, and the network topology graph carries each computing node included in the computing power cluster and a physical link between each two computing nodes. The management server constructs an initial graph structure according to the network topology graph. The SDN can be SONiC, Arista EOS, or the like.

[0073] However, the network topology graph may have errors compared with the actual situation, and therefore, after constructing the initial graph structure, the management server obtains the neighbor relationship and the interface state of each computing node from the MBI library of the switch, wherein the neighbor relationship of each computing node is the information of the computing node directly connected to the computing node. The management server updates the edges in the initial graph structure according to the neighbor relationship of each computing node, to obtain an intermediate graph structure.

[0074] For example, for each two nodes, if there is an edge between the two nodes in the initial graph structure, but the management server obtains that the two nodes are not directly connected in the neighbor relationship of each computing node, the management server deletes the edge between the two nodes in the initial graph structure; if there is no edge between the two nodes in the initial graph structure, but the management server obtains that the two nodes are directly connected in the neighbor relationship of each computing node, the management server adds the edge between the two nodes in the initial graph structure.

[0075] In addition, the management server also acquires network configuration information of each computing node locally sent by a lightweight agent deployed on each computing node. After receiving the network configuration information, the management server determines a weight corresponding to each edge in the intermediate graph structure based on the network configuration information, adds the weight corresponding to each edge to the intermediate graph structure, and adds the network configuration information and interface state corresponding to each node to a node corresponding to each computing node in the graph structure, thereby obtaining the graph structure.

[0076] The interface state includes but is not limited to Up / Down, Cyclic Redundancy Check (CRC) error count, optical module receiving power, and Forward Error Correction (FEC) rate. The network configuration information includes but is not limited to device type, IP address, unique device ID, bandwidth, delay, and historical failure rate of each link corresponding to the computing node, and the like. The device type can be a GPU server, a leaf, and a spine.

[0077] In the embodiment of the application, the management server can determine the initial weight corresponding to each edge in the graph structure according to the information of the link corresponding to the edge. Specifically, the management server can determine the initial weight corresponding to each edge according to the bandwidth, delay, and historical failure rate of the link corresponding to the edge.

[0078] For example, the initial weight corresponding to each edge satisfies the following formula:

[0079]

[0080] wherein, represents the initial weight, , , is an empirical adjustment parameter, wherein, , represents the bandwidth, represents the delay, represents the historical failure rate; wherein the unit of the bandwidth used in the above formula is gigabit per second (Gbps), and the unit of the delay is millisecond (ms).

[0081] The calculation formula of the initial weight can set a lower initial weight for a link with high bandwidth, low delay, and stable history, and vice versa.

[0082] For example, the initial weight of the edge of the 100Gbps link is set to 1, and the initial weight of the edge of the 200Gbps link is set to 0.5.

[0083] S102: A preset graph theory algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0084] In the embodiments of the present application, after the management server constructs the graph structure, the management server can use a preset graph theory algorithm to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0085] In a possible implementation, the management server can take one or more nodes in the graph structure as source nodes and run a graph theory algorithm to calculate each shortest path from the source node to all other nodes according to each edge corresponding to each node in the graph structure and the initial weight of each edge, and the management server takes each shortest path as the test path set.

[0086] S103: The computing power cluster is tested according to each test path included in the test path set.

[0087] In the embodiments of the present application, the management server tests the computing power cluster according to each test path included in the test path set.

[0088] Specifically, the management server issues a test instruction carrying a test path to an agent configured in each computing node in the computing power cluster. After receiving the test instruction, the agent calls a series of mature network test tools to simulate different types of traffic and test the corresponding test path.

[0089] For example, for basic connectivity and delay testing, the ping or hping3 command can be used; for bandwidth throughput testing, the iperf3 command can be used; and in order to more realistically reflect the AI training load, the NVIDIA Collective Communications Library (NCCL) can also be called to perform collective communication operations, which can effectively expose the performance bottleneck of the network under real business scenarios. In order to maximize test efficiency and avoid impacting production business, the test execution adopts a highly parallel architecture.

[0090] In addition, in the embodiments of the present application, the management server can also intelligently divide the test task into multiple batches, and the test paths in each batch are non-shared links, that is, they do not compete for the bandwidth of the same communication link, so they can be safely executed in parallel.

[0091] For example, in a Fat-Tree topology, multiple server pairs in different Pods can be tested simultaneously. Meanwhile, the system strictly controls the total test traffic, for example, limits it within 10% of the total network capacity, and dynamically adjusts the batch size and concurrency according to the business busy degree of the cluster (such as daytime or night). The performance data collected during the test (such as throughput, delay, jitter, packet loss rate) will be reported to the result analysis module in real time through streaming, along with metadata such as timestamp, source / destination node, etc., to provide data support for subsequent dynamic feedback loop.

[0092] In the embodiment of the present application, according to the network information of the computing power cluster obtained, a weighted undirected graph structure is constructed, wherein the network information carries the network configuration information of each computing node in the computing power cluster and the communication link information of the computing power cluster; the nodes of the graph structure represent the computing nodes in the computing power cluster, the edges of the graph structure represent that the computing nodes corresponding to the two nodes have a communication link in the computing power cluster, and the initial weight of the edge is determined based on the network configuration information of the computing nodes corresponding to the two nodes; a preset graph algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge; and the computing power cluster is tested according to each test path included in the test path set. Through the innovative construction process of the graph structure and the test path generation strategy, the embodiment of the present application realizes fast, accurate and intelligent health diagnosis of the network infrastructure of the computing power cluster, and ensures that the network can meet the strict requirements of extremely low delay, extremely high bandwidth and zero packet loss for large-scale distributed AI training.

[0093] Embodiment 2

[0094] In order to improve the network test efficiency and test effect, on the basis of the above-mentioned embodiments, in the embodiment of the present application, the graph algorithm includes a single-source shortest path algorithm and a minimum spanning tree algorithm;

[0095] The generating each test path by using the preset graph algorithm according to each edge corresponding to each node in the graph structure and the initial weight of each edge includes:

[0096] Taking each first node in the graph structure as a source node and each second node in the graph structure as a destination node, the single-source shortest path algorithm is used to generate a first test path from each first node to each second node according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0097] adopting the single-source shortest path algorithm, and according to each edge corresponding to each node in the graph structure and the initial weight of each edge, generating a second test path from each second node to each third node;

[0098] adopting the minimum spanning tree algorithm, and according to each edge corresponding to each node in the graph structure and the initial weight of each edge, generating a third test path;

[0099] determining the first test path, the second test path and the third test path as the test path set.

[0100] In the embodiments of the present application, the graph theory algorithm includes but is not limited to the single-source shortest path algorithm and the minimum spanning tree algorithm; wherein the single-source shortest path algorithm can be the Dijkstra algorithm, and the minimum spanning tree algorithm can be the Prim algorithm.

[0101] In the embodiments of the present application, the inherent regularity and hierarchical structure of the network topology of the computing power cluster are fully utilized, a global complex problem is decomposed into multiple local and easy-to-manage sub-problems, and thus the efficiency and pertinence of path calculation are greatly improved.

[0102] Specifically, the nodes of the computing power cluster are divided into three logical levels, which are: the first computing nodes in the Spine layer at the top level, the second computing nodes in the Leaf layer at the middle level, and the third computing nodes in the access layer at the bottom level, wherein the first computing nodes and the second computing nodes are generally switches, and the third computing nodes are generally GPU servers.

[0103] The management server adopts the single-source shortest path algorithm, and according to each edge corresponding to each node in the graph structure and the initial weight of each edge, generates a first test path from each first node to each second node, wherein each first node is a first computing node in the Spine layer in the computing power cluster, and each second node is a second computing node in the Leaf layer in the computing power cluster.

[0104] The generation of the first test path focuses on the interconnection between the Spine and Leaf layers. The management server treats each first node representing the Spine layer in the graph structure as an independent source node, and then runs Dijkstra's algorithm with all second nodes representing the Leaf layers as destination nodes. Since Dijkstra's algorithm searches for the path with the minimum sum of weights in a weighted graph, and the weight design in the graph structure of this application reflects the health of the links (where lower weights indicate better link health), this process can automatically select the optimal communication path from the Spine layer to the Leaf layer.

[0105] Then, the management server uses each second node as the source node and each third computing node in the graph structure of the access layer of the computing power cluster as the destination node. Using the single-source shortest path algorithm, it generates a second test path from each second node to each third node based on each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0106] The second test path focuses on the connection from the Leaf layer to the access layer. In this case, the second node in each Leaf layer becomes the new source node, and the third node in the access layer becomes the destination node. The management server can use Dijkstra's algorithm to generate the second test path from each second node to each third node. This second test path can efficiently verify whether the connection from the network edge to the computing unit is normal.

[0107] Then, the management server uses the minimum spanning tree algorithm to generate a third test path containing each node in the graph structure, based on each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0108] In this embodiment, the generation of the third test path no longer prioritizes performance optimization but rather connectivity completeness. The management server runs the Prim algorithm to generate a third test path containing every node in the graph structure. The Prim algorithm greedily selects the edges with the smallest weights, gradually incorporating all nodes in the graph structure into a tree. The resulting tree structure contains exactly the minimum number of edges required to access all nodes in the graph. In this way, the connectivity of the graph structure can be determined by testing only a small number of edges.

[0109] The management server determines the first test path, the second test path, and the third test path as the test path set.

[0110] In this embodiment of the application, since computing power clusters generally adopt a Fat-Tree / Clos hierarchical topology, the management server can use a hierarchical strategy to generate a set of test paths.

[0111] In the first round, the management server selects all first computing nodes of the Spine layer as source nodes, runs the Dijkstra algorithm, and calculates first test paths from each first computing node to all second computing nodes of the Leaf layer. Since the Dijkstra algorithm naturally tends to select the path with the lowest weight, this process is essentially to verify the "main road" of the network in priority, to ensure the key connectivity and performance between the Spine layer and the Leaf layer.

[0112] In the second round, the management server takes each second computing node of the Leaf layer as a new source node, runs the Dijkstra algorithm again, and generates second test paths from the Leaf layer to all third computing nodes of the Access layer, thereby completing the verification of the connectivity of the Access layer.

[0113] The two rounds of Dijkstra algorithm together cover most of the key communication paths from the network core to the edge. However, the shortest path alone cannot guarantee the absolute connectivity of the network, because there may be redundant links that are not selected. For this purpose, the third round introduces the Prim minimum spanning tree algorithm. The goal of the Prim algorithm is to find a loop-free subgraph containing all nodes in the graph structure with the minimum total weight, and the number of edges is exactly one less than the number of nodes. By running the Prim algorithm, the management server can generate third test paths across racks.

[0114] The test path set constructed by the above three rounds can focus on high-performance backbone paths and ensure full coverage of basic connectivity, covering the largest risk surface with the least number of tests.

[0115] In the embodiments of the present application, the management server abstracts the physical network into a weighted undirected graph structure, and the initial weight of each edge in the graph structure is calculated based on the bandwidth, delay, historical failure rate, and the like of the corresponding link. The management server generates an optimal path set from the core node to all computing nodes by the Dijkstra algorithm, verifies the network main road in priority, and generates a minimum edge set covering all nodes in the graph structure by the minimum spanning tree algorithm, to ensure basic connectivity verification. The test path determination method reduces the network test complexity from O(N²) to nearly linear.

[0116] Embodiment 3:

[0117] To improve the network test efficiency and test effect, on the basis of the above embodiments, in the embodiments of the present application, the method further comprises:

[0118] Obtaining a test result corresponding to each test path, the test result being an actual performance index value of the test path;

[0119] According to a relationship between the actual performance index value corresponding to each test path and a preset performance index threshold, weights of each edge corresponding to each test path in the graph structure are updated.

[0120] The test path set is updated by using the graph theory algorithm and the updated graph structure, and testing is performed based on the updated test path set until a preset test completion condition is met.

[0121] In the embodiments of the present application, the management server can also iteratively optimize the graph structure based on the test results.

[0122] Specifically, the management server receives a test result corresponding to each test path returned by an agent configured in each computing node in the computing power cluster, and the test result is an actual performance index value of the test path.

[0123] The management server updates the weights of each edge corresponding to each test path in the graph structure according to a relationship between the actual performance index value corresponding to each test path and a preset performance index threshold, and updates the test path set by using the graph theory algorithm and the updated graph structure, and performs testing based on the updated test path set until a preset test completion condition is met.

[0124] For example, for each link in each test path, if the actual performance index value corresponding to the link reaches the preset performance index threshold, the management server reduces the weight of the edge corresponding to the link in the graph structure; if the actual performance index value corresponding to the link does not reach the preset performance index threshold, the management server increases the weight of the edge corresponding to the link in the graph structure.

[0125] The graph structure after the weight update has evolved into a graph structure that can better reflect the real health status of the current network. At this time, the management server will re-call the graph theory algorithm in S102, and re-calculate and generate a new test path set based on the updated graph structure.

[0126] Since the weights of the problematic links have been increased, the new shortest path calculation process will intelligently bypass these links, thereby concentrating test resources on network areas that have not been fully verified or whose status is still unknown. This iterative process will continue until a certain preset test completion condition is met.

[0127] In the embodiments of the present application, the test completion condition can be diversified.

[0128] For example, the test completion condition can be configured as an integrity mode, i.e., requiring all critical paths and edges of the minimum spanning tree to be successfully tested at least once; or the test completion condition can be configured as a problem-oriented mode, i.e., when a preset number (e.g., 10) of faulty links are found, or a plurality of rounds (e.g., 3) of tests fail to find new faults, it is considered that the current network state is clear enough; or the test completion condition can be configured as a time limit mode, i.e., stopping forcibly after reaching a maximum allowed test duration (e.g., 2 hours).

[0129] Embodiment 4:

[0130] To improve the network test efficiency and test effect, on the basis of the above embodiments, in the embodiments of the present application, the weight of each edge corresponding to each test path in the graph structure is updated according to the relationship between the actual performance index value corresponding to each test path and the preset performance index threshold value.

[0131] For each link in each test path, if the actual performance index value of the link meets the requirement of the preset performance index threshold value, the weight of the edge corresponding to the link in the graph structure is increased by a preset penalty value; if the actual performance index value of the link does not meet the requirement of the preset performance index threshold value, the weight of the edge corresponding to the link in the graph structure is updated according to a preset decay factor.

[0132] In the embodiments of the present application, if the management server determines that the actual performance index value of a link of a test path meets the preset performance index threshold value, the management server determines that the link has an abnormality. At this time, the link is regarded as a suspicious object, and the management server increases the edge corresponding to the link in the graph structure by a preset penalty value, which is used to ensure that any edge containing the penalty value in the corresponding weight will not be selected as the shortest path in the next round of path calculation, thereby realizing automatic avoidance of the known problem area.

[0133] In addition, if the actual performance index value of a link in the test path performs well, the management server reduces the weight of the edge corresponding to the link in the graph structure, which encourages the graph theory algorithm to preferentially select these links that have been proved to be healthy in the future.

[0134] Specifically, in the embodiments of the present application, for each link in each test path, if the actual performance index value of the link meets the preset performance index threshold value, the management server increases the weight of the edge corresponding to the link in the graph structure by a preset penalty value; if the actual performance index value of the link does not meet the preset performance index threshold value, the management server updates the weight of each edge corresponding to the link in the graph structure according to a preset decay factor.

[0135] It should be noted that in the embodiments of the present application, the actual performance index corresponding to each test path includes each link in the test path corresponding to each edge contained in the test path.

[0136] In order to improve the network test efficiency and test effect, on the basis of the above embodiments, in the embodiments of the present application, the actual performance index value is the measured bandwidth and / or the packet loss rate.

[0137] In the embodiments of the present application, the actual performance index value can be the measured bandwidth, the actual performance index value can also be the packet loss rate, and the actual performance index value can also be the measured bandwidth and the packet loss rate.

[0138] For example, if the actual performance index value is the measured bandwidth, the preset performance index threshold is a preset bandwidth threshold, which can be 0.9 times of the standard bandwidth, or other multiples of the standard bandwidth; if the actual performance index value is the packet loss rate, the preset performance index threshold is a preset packet loss rate threshold, which can be 0.001%, or other numerical values.

[0139] If the actual performance index value is the measured bandwidth, the actual performance index value satisfies the preset performance index threshold, that is, the measured bandwidth is not lower than the preset bandwidth threshold; if the actual performance index value is the packet loss rate, the actual performance index value satisfies the preset performance index threshold, that is, the measured bandwidth does not exceed the preset packet loss rate threshold.

[0140] For example, the management server can determine the test link corresponding to the measured bandwidth BW_actual<0.9×standard bandwidth BW_nominal or the packet loss rate Loss>0.001% as an abnormal link.

[0141] In order to improve the network test efficiency and test effect, on the basis of the above embodiments, in the embodiments of the present application, after updating the weight of the edge corresponding to the link in the graph structure according to the preset attenuation factor, the method further comprises:

[0142] For each edge corresponding to each test path in the graph structure, if the updated weight of the edge is lower than the preset minimum weight, the weight of the edge is updated to the minimum weight.

[0143] In the implementation of the present application, for a link whose actual performance index value meets the preset performance index threshold, the management server updates the weight of the edge corresponding to the link according to the preset attenuation factor, so that the weight of each edge gradually decreases. However, if the lower limit of the weight is not constrained, after multiple successful tests, the weight of some high-quality links may tend to zero, which not only may cause the problem of floating-point calculation precision, but more importantly, may weaken the discrimination of the weight system, so that all good links look the same, losing the meaning of guiding the algorithm to make fine selection.

[0144] Based on this, in the embodiments of the present application, the management server is also provided with a minimum weight, and after updating the weight according to the preset attenuation factor, the management server also performs a check to determine whether the updated weight is lower than the preset minimum weight. For each edge corresponding to each test path in the graph structure, if the updated weight corresponding to the edge is lower than the preset minimum weight, the management server forcibly updates the weight of the edge to the minimum weight.

[0145] In the embodiments of the present application, the minimum weight sets a reasonable lower limit for the graph structure, ensures that the weight value is always within an effective and comparable range, and thus maintains the effectiveness and stability of the decision of the graph theory algorithm.

[0146] For example, for a link whose actual performance index value meets the requirement of the preset performance index threshold, the management server can update the weight of the edge corresponding to the link in the graph structure by using the following formula:

[0147]

[0148] wherein, represents the updated weight, represents the original weight, is the preset attenuation factor, is the preset minimum weight, is the max algorithm.

[0149] For a link whose actual performance index value meets the requirement of the preset performance index threshold, the management server can update the weight of the edge corresponding to the link in the graph structure by using the following formula:

[0150]

[0151] wherein, represents the updated weight, represents the original weight, is the preset penalty value, wherein the penalty value P=10000.

[0152] In the embodiments of the present application, after the test is executed, the management server dynamically adjusts the weight of the edge corresponding to each link in the test path in the graph structure according to the actual performance index value (bandwidth, delay) of each link in the test path. If the measured bandwidth of a certain link is lower than 90% of the expected threshold, the weight of the link is greatly increased. After updating the weight, the management server recalculates the test path, intelligently bypasses the problem link that has been confirmed, and concentrates the test resources in the unverified or suspicious area, forming an iterative optimization closed loop of "path selection-test execution-weight update-reselection".

[0153] Figure 2 The flowchart of the dynamic weight adjustment and feedback closed loop provided by the embodiments of the present application is shown in the figure. Figure 2 The process includes:

[0154] S201: initial weight setting.

[0155] S202: test path execution.

[0156] S203: test result analysis, if the performance of the test path is abnormal, S204 is executed, if the performance of the test path is normal, S205 is executed.

[0157] S204: weight penalty increase, and S206 is executed.

[0158] S205: weight fine-tuning, and S206 is executed.

[0159] S206: update the graph structure.

[0160] S207: recalculate the test path, and S202 is executed.

[0161] Embodiment 5:

[0162] In order to improve the network test efficiency and test effect, on the basis of the above-mentioned embodiments, in the embodiments of the present application, after the weight of the edge corresponding to the link in the graph structure is increased by a preset penalty value, the method further includes:

[0163] updating the saved number of abnormalities corresponding to the link;

[0164] if the updated number of abnormalities reaches a preset abnormality number threshold, obtaining passive monitoring data corresponding to the link, the passive monitoring data including sFlow / NetFlow flow sampling, switch port cyclic redundancy check CRC error count, forward error correction FEC statistics, optical module received optical power, and digital diagnostic monitoring DDM alarm information;

[0165] using an abnormality detection model to detect abnormalities of the link according to the passive monitoring data, and determining a confidence score corresponding to the link;

[0166] If the confidence score exceeds a preset threshold, an alarm is given.

[0167] In the embodiments of the present application, after determining that a certain link is abnormal, the management server records the number of times of abnormality of the link. When a certain link is repeatedly marked as abnormal, i.e., after updating the number of times of abnormality corresponding to the link, if the updated number of times of abnormality reaches a preset number of times of abnormality threshold, the management server isolates the link from the graph structure and performs in-depth analysis on the link.

[0168] Specifically, the management server acquires passive monitoring data in the same time window as the link. The passive monitoring data is extensive, including but not limited to sFlow / NetFlow traffic sampling, switch port CRC error count, FEC statistics, optical module received optical power and DDM alarm information.

[0169] The management server performs space-time alignment on the passive monitoring data: in the time dimension, it ensures that the time window of active testing and the timestamp of passive data acquisition are accurately matched; in the space dimension, it maps the abstract link back to the specific physical link, i.e., determines which specific ports of which switches the path passes through. After alignment, the management server fuses these multi-source data into a feature vector X, for example wherein, represents the difference between the measured bandwidth of the test path in the current active test and the nominal bandwidth of the test path, represents the increment of CRC error count of the switch port through which the test path passes in the time window in which the active test occurs, represents the increment of Symbol error of the test path in the time window in which the current active test occurs, represents the value of the normalized optical power received by the optical module, represents the number of errors corrected by the forward error correction mechanism in the time window in which the current active test occurs, represents an identification value indicating whether the NCCL-based load test in the current active test fails.

[0170] The management server calls a pre-trained anomaly detection model to process the feature vector and outputs a confidence score between 0 and 1, indicating the probability that the anomaly is caused by a physical layer hardware fault. The anomaly detection model can be a model with strong interpretability such as logistic regression, random forest, or a time series model such as LSTM. The hardware fault can be optical module contamination, optical fiber loosening, etc.

[0171] If the confidence score exceeds the preset threshold, the management server considers the fault location to be sufficiently reliable and automatically triggers Syslog logging or SNMP Trap alarms to notify maintenance personnel to perform precise hardware replacement or maintenance.

[0172] For example, if a link fails three times consecutively, the weight of each edge corresponding to that test path is set to... , This is set to an infinitely large value, thus achieving logical isolation of each edge within the graph structure. Subsequently, the management server correlates test anomalies with interface error counters (such as CRC errors and Symbol errors) and optical module alarms (Rx Power Low) collected by sFlow during the same time period to generate a confidence score. When this confidence score > 0.8, a Syslog / SNMP Trap alarm is automatically triggered.

[0173] In addition, in this embodiment of the application, the management server can also calculate the overall network health index. The overall health index of this network can reflect the status of the cluster network in real time. Among them, The total weight of each edge corresponding to the abnormal test link. This represents the total weight of each edge in the graph structure.

[0174] In this embodiment, the management server performs a fusion analysis based on active test results, passive network monitoring data (NetFlow / sFlow / gRPC Telemetry), and hardware status data (optical module signal quality, switch port statistics). Anomaly detection algorithms identify phenomena such as traffic bursts and surges in error packets. Combined with AI-assisted optical module fault diagnosis, channel-by-channel fault diagnosis is achieved, improving problem location accuracy from the "path level" to the port level and even the channel level.

[0175] Example 6:

[0176] To improve network testing efficiency and effectiveness, based on the above embodiments, in this embodiment, the step of testing the computing cluster according to each test path included in the test path set includes:

[0177] Based on the network corresponding to each test path included in the test path set, the test path set is divided into multiple test path subsets;

[0178] For each subset of test paths, determine each test path of the shared-nothing link included in the subset; use a preset test command to schedule each test path of the shared-nothing link in parallel.

[0179] In order to ensure the test efficiency and minimize the impact on the production business, in the embodiments of the present application, the management server will preprocess the test path set to be executed.

[0180] Specifically, the management server will divide the entire test path set into multiple test path subsets according to the network corresponding to each test path in the test path set. The network can refer to different virtual local area networks (VLAN) or remote direct memory access (RDMA) segments. This division ensures that the test traffic of different service planes does not interfere with each other.

[0181] For each test path subset, the management server will determine the test paths in the test path subset without shared links. These test paths without shared links constitute a batch that can be completely executed in parallel. The management server uses a preset test command to schedule each test path without shared links in parallel.

[0182] In order to improve the network test efficiency and test effect, on the basis of the above embodiments, in the embodiments of the present application, the management server uses a preset test command to schedule each test path without shared links in parallel includes:

[0183] For each test path without shared links, at least one of the following ways is used for testing:

[0184] Use the ping command or hping3 command to test the delay and packet loss of the test path;

[0185] Use the iperf3 command to test the bandwidth of the test path, and execute point-to-point bandwidth throughput test;

[0186] Use the NVIDIA collection communication library NCCL to test the load of the test path.

[0187] In the embodiments of the present application, for each test path without shared links, the management server will flexibly select different test tools according to the test target.

[0188] Specifically, for fast connectivity, delay and packet loss detection, the management server will use the ping command or hping3 command to test the test path. The ping command or hping3 command is lightweight and fast, and is suitable for preliminary screening.

[0189] In order to accurately measure the maximum available bandwidth of the link, the management server will use the iperf3 command to perform bandwidth testing on the test path to perform point-to-point bandwidth throughput testing. The iperf3 command can generate high-throughput TCP or UDP traffic and is an industry standard tool for measuring network bandwidth performance.

[0190] In order to most realistically simulate the network load in the AI distributed training scenario, the management server will use NCCL to perform load testing on the test path. The NCCL can simulate complex collective communication modes such as All-Reduce and All-to-All, not only verifying the bandwidth, but also exposing deep-level problems such as micro-burst packet loss and PFC back pressure when handling many-to-many and synchronous communication.

[0191] In the embodiments of the present application, by combining the use of the above test commands, the management server can comprehensively evaluate the health status of the network link from multiple dimensions, ensuring the accuracy and practicality of the test results.

[0192] In order to realize the above-mentioned application principle, the embodiments of the present application design a modular system architecture, Figure 3 The system architecture provided by the embodiments of the present application is shown in the following figure: Figure 3 As shown in the figure, the system architecture mainly consists of four core modules: topology discovery and modeling module, path calculation and scheduling engine module, test execution and data collection module, and result analysis and graph structure updating module. These four modules work together to form a complete closed-loop system from perception, decision, execution to feedback.

[0193] Among them, the topology discovery and modeling module is the cornerstone of the system, which is used to realize the automatic conversion of physical network to logical graph structure. The topology discovery and modeling module is used for topology discovery, graph structure construction and weight initialization. Specifically, the topology discovery and modeling module uses NetworkX (Python graph computing library) to construct a dynamic graph in memory , wherein each node of the graph structure saves the device type (GPU server / Leaf / Spine), IP address, and device ID of the computing node corresponding to the node; each edge of the graph structure saves the interface pair (such as eth0<->Ethernet1 / 1), nominal bandwidth, and initial weight corresponding to the edge. In addition, the topology discovery and modeling module is also used to listen to Netlink events or gNMI / gRPC streaming subscriptions to realize millisecond-level graph structure updating of link up / down events.

[0194] The path calculation and scheduling engine module is the core of system decision, which is used to realize intelligent path planning and task arrangement. The path calculation and scheduling engine module is used for algorithm implementation, path planning and task scheduling. The path calculation and scheduling engine module has an algorithm library, a scheduler and a strategy optimizer built-in, and the algorithm library stores graph theory algorithms. The path calculation and scheduling engine module can construct a test path set according to the graph theory algorithms in the algorithm library, and through the scheduler, the test path set calculated is sliced according to network planes (such as different VLANs or RDMA network segments) and time periods to generate a test task queue, and Slurm or Kubernetes Job is used to realize task parallel scheduling, control the number of concurrent tasks to avoid test traffic congestion (such as limiting the total test bandwidth ≤10% of the network capacity). The path calculation and scheduling engine module can also use the strategy optimizer to automatically improve the test priority of high-frequency fault areas (such as a certain rack) based on historical test results, and realize risk-driven verification.

[0195] The test execution and data collection module is used to execute specific test tasks and collect data on the physical network. The test execution and data collection module is used for test agent deployment, test agent execution and data collection. The test execution and data collection module as a system execution unit realizes lightweight and non-intrusive testing, and its main functions include:

[0196] Test agent deployment: lightweight test agent programs are deployed on the computing nodes of the cluster. These agent programs can receive instructions from the scheduling engine and execute network tests.

[0197] Test agent execution: support multiple network testing tools and methods, such as iperf3 for bandwidth testing, ping or hping3 for delay and packet loss testing, and NCCL-based collective communication testing to simulate real AI training loads.

[0198] Data collection: while performing tests, relevant performance data is collected, including throughput, delay, jitter, packet loss rate, etc., and these data are reported together with test timestamps, source / destination nodes, etc.

[0199] Resource control: has resource control capability, can limit the bandwidth and CPU resources occupied by the test, and ensure that the test process does not significantly affect production business.

[0200] The result analysis and graph structure updating module is a feedback center of the system, and is used for realizing intelligent decision-making driven by data. The result analysis and graph structure updating module is used for data analysis, fault positioning and weight updating. Specifically, the result analysis and graph structure updating module can update the weight of each edge corresponding to each test path according to the test result of the test path, and also performs fault correlation analysis and network overall health quantification.

[0201] The core algorithm process of the present application is an iterative optimization process, which aims to achieve the highest network verification coverage and problem discovery rate with the least number of tests. Figure 4 The overall flowchart of network testing provided for the embodiments of the present application is as shown in the figure. Figure 4 As shown in the figure, the process includes:

[0202] S401: initialization stage, topology discovery and modeling: constructing a weighted graph structure, and assigning an initial weight to each edge in the graph structure.

[0203] The management server scans the entire computing power cluster through SNMP, API and the like, identifies all network devices and physical links, abstracts them into a graph structure, and assigns an initial weight to each edge.

[0204] This step is one-time initialization or periodic full synchronization. The management server completes the topology discovery of the ten-thousand-card cluster within 5 minutes through multi-threaded concurrent SNMP queries. When constructing the graph structure, the management server can use sparse matrix storage, and the memory occupation is <2 GB within one hundred thousand nodes. The weight initialization imports the design parameters from the CMDB, and the historical failure rate is queried from the Prometheus time series database.

[0205] S402: path selection stage, critical path calculation: determining a test path set based on a shortest path algorithm, and generating a minimum spanning tree to ensure full coverage.

[0206] The management server selects one or more core nodes as source nodes, runs the Dijkstra algorithm, calculates the shortest path set from the source nodes to all other nodes, and constitutes a critical path test set. At the same time, the management server runs the minimum spanning tree algorithm to obtain a minimum spanning tree containing all nodes in the graph structure, and ensures connectivity verification. The management server takes the critical path test set and the minimum spanning tree as the test path base.

[0207] This step is the core of the algorithm. For a 10,000-node cluster, the single Dijkstra calculation amount is reduced to about 1000 nodes after adopting the hierarchical strategy, and the time consumption is <100 ms. After generating the path set, the scheduling engine prioritizes according to the "core-converge-access" priority, ensuring that critical paths are tested first. The output is a test task list, each containing: source node IP, destination node IP, expected minimum bandwidth, test tool type, and timeout time.

[0208] S403: Test execution phase, bandwidth and connectivity test: execute test tools such as iperf3 and ping to collect performance data and indicators.

[0209] Specific network tests are performed by test agents deployed on nodes, including connectivity tests (ping), bandwidth tests (iperf3), delay tests, and NCCL-based collective communication tests to simulate real AI training loads.

[0210] Tests are executed in batches in parallel. Each batch selects mutually disjoint paths (no shared links) to avoid test traffic interference. For example, in a Fat-Tree, multiple server pairs within a pod can be tested simultaneously. Batch size is dynamically adjusted: large batch size (high parallelism) during night hours, and small batch size (low interference) during daytime.

[0211] S404: Analysis and update phase, result analysis and weight adjustment: dynamically adjust link weights based on test results and update graph structure.

[0212] After the test is completed, the result analysis and graph structure update module begins to process the collected data. For each tested link, the management server compares its test results with the preset performance indicator threshold. If the actual performance indicator value (such as bandwidth) of the link is lower than the preset performance indicator threshold, the management server will determine that the link is a "problem link". Subsequently, the management server will dynamically adjust the weight of the link in the graph structure. Conversely, if the link performance is good, the weight can remain unchanged or be slightly adjusted downward. This dynamic adjustment process enables the graph structure to reflect the network's health status in real time, evolving from a static topology graph to a dynamic, feedback-capable health status graph.

[0213] S405: Cycle optimization phase, iterative verification: recalculate test paths to bypass problem links until verification is complete or problems are found.

[0214] After the weight update, path selection is performed again. The management server re-runs the shortest path or minimum spanning tree algorithm based on the updated graph structure to generate a new round of test paths. Since the weights of the problem links have been increased, the new test paths will intelligently bypass the edges corresponding to these links, concentrating test resources in network areas that have not been fully verified or whose status is unknown. This "path selection-test execution-weight update" cycle continues until the preset test completion conditions are met.

[0215] To address the huge challenges brought by the million or even ten million computing power cluster, the present application proposes a series of targeted optimization strategies based on the above network testing process, aiming to reduce the computational complexity, improve the testing efficiency, and ensure the feasibility and leadership of the scheme in the super large scale environment, including but not limited to the following:

[0216] 1. Topology-aware path selection, optimization for Clos / Fat-Tree network:

[0217] Modern computing power clusters generally use hierarchical network topologies such as Fat-Tree or Clos network to achieve non-blocking or low-blocking communication. These topologies have high symmetry and regularity. The present application takes full advantage of this feature and designs a topology-aware path selection strategy. For example, in a typical three-layer Clos network, the network can be divided into Spine layer, Leaf layer and Server layer. The management server can first divide the test task into three levels:

[0218] Intra-layer testing: verify the connection between devices in the same layer, such as the connection between Leaf switches.

[0219] Inter-layer testing: verify the connection between devices in adjacent layers, such as the connection between all Leaf switches and all Spine switches.

[0220] End-to-end testing: verify the complete path from server to server.

[0221] Through this hierarchical testing, a huge full-grid testing problem can be divided into multiple smaller and more manageable sub-problems. In addition, the symmetry of the topology can be used to test only representative paths. For example, in a completely symmetric Fat-Tree network, only the paths within a "subtree" need to be tested, and the results can be generalized to the entire network, greatly reducing the number of test cases.

[0222] 2. Hierarchical and partitioned testing, reduce computational complexity:

[0223] For a cluster of 100,000 GPUs, even with topology-aware optimization, processing the graph structure of the entire network at once can still result in significant computational and memory overhead. To address this issue, the present application can employ a hierarchical and partitioned testing strategy. Specifically, the entire cluster network can be divided into multiple logical or physical partitions (e.g., in units of racks or pods). First, the network testing algorithm of the present application is run independently within each partition to quickly locate and resolve network problems within the partition. Then, higher-level verification is performed between partitions (e.g., at the level of spine switches across racks). This hierarchical approach breaks down a huge, complex global problem into multiple smaller, manageable local problems, greatly reducing the graph size for single-path computation, thereby effectively controlling the computational overhead of the algorithm.

[0224] 3. Parallel testing and distributed execution to improve testing efficiency:

[0225] To minimize verification time, the present application designs a highly parallel testing execution architecture. After the path computation and scheduling engine generates a list of test tasks, these tasks can be distributed to multiple test agents in the cluster for simultaneous execution. Since network testing tasks (such as bandwidth testing) are typically I / O intensive, they can be run in parallel on different node pairs without interfering with each other, as long as the total test traffic is reasonably controlled to avoid impacting the production network. Together AI's practice also emphasizes the use of schedulers such as Slurm to control the execution of NCCL tests, which in itself is a form of parallel testing. The test execution module of the present application can integrate similar scheduling capabilities to intelligently orchestrate large-scale test tasks into multiple parallel batches, fully utilizing the distributed computing power of the cluster to significantly shorten linear test time and achieve near-constant time verification efficiency, which is crucial for large-scale AI clusters that require frequent health checks or change verification.

[0226] Compared to existing technologies, the network verification method based on graph theory and dynamic feedback proposed by the present application surpasses existing tools and methods in multiple aspects, demonstrating significant innovation and leadership. Specifically, the present application includes but is not limited to the following advantages:

[0227] 1. Surpassing the efficiency of traditional point-by-point testing tools:

[0228] Traditional network verification methods, such as point-by-point testing using ping, traceroute, or iperf3, have exponentially decreasing efficiency in large-scale clusters. For a cluster of N nodes, the full connection test has a complexity of O(N²), which is completely unrealistic in a ten-thousand-node cluster. The present application reduces the complexity of the test to near-linear or lower, depending on the algorithm and topology, by abstracting the network as a graph structure and using algorithms such as shortest path and minimum spanning tree. By intelligently selecting a small number of "critical paths" for testing, the present application can cover the entire network with minimal testing overhead, with much higher efficiency than traditional exhaustive testing methods.

[0229] 2. Intelligent path planning differentiated from commercial testing platforms (such as Keysight):

[0230] Although the AI Data Center Test Platform of commercial testing platforms such as Keysight provides powerful AI workload simulation and performance benchmarking capabilities, its core may be more focused on macro performance evaluation and protocol consistency verification. The core advantage of the present application is its intelligent path planning capability. Unlike platforms that rely on pre-defined test cases or scripts, the present application can dynamically adjust subsequent testing strategies based on real-time test results. This adaptive, feedback-based testing method enables the system to actively and efficiently "explore" and locate weak links in the network like a "probe", rather than passively executing pre-set test sequences.

[0231] 3. Complement and improvement of academic research (such as SuperBench):

[0232] Academic research such as Microsoft's SuperBench proposes an innovative verification framework that predicts and diagnoses node failures through intelligent benchmarking sets and probabilistic models. Although its network verification part is efficient, it is complementary rather than a substitute for the path optimization idea based on graph theory and dynamic feedback proposed in the present application. SuperBench focuses on quickly discovering network-level problems through fixed-mode scanning, while the present application achieves deeper and continuously optimized fault localization through a dynamic feedback mechanism. The present application can be seen as an evolution of efficient scanning methods such as SuperBench, adding intelligent avoidance and deep diagnosis capabilities for problem links based on fast scanning, thereby achieving higher localization accuracy and better resource utilization.

[0233] Embodiment 7:

[0234] Based on the same inventive concept, the present application provides a data processing device, Figure 5 A structural diagram of a network testing device of a computing power cluster is provided for the embodiments of the present application, please refer toFigure 5 The device includes:

[0235] Processing module 501 is used to construct a weighted undirected graph structure based on the acquired network information of the computing power cluster. The network information carries network configuration information of each computing node in the computing power cluster and communication link information of the computing power cluster. Nodes in the graph structure represent computing nodes in the computing power cluster, and edges in the graph structure represent two connected nodes whose corresponding computing nodes have a communication link in the computing power cluster. The initial weight of the edge is determined based on the network configuration information of the two connected computing nodes. A preset graph theory algorithm is used to generate a set of test paths based on each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0236] The testing module 502 is used to test the computing power cluster according to each test path included in the test path set.

[0237] In one possible implementation, the graph theory algorithm includes a single-source shortest path algorithm and a minimum spanning tree algorithm;

[0238] The processing module 501 is specifically used to take each first computing node of the backbone Spine layer in the computing power cluster as the source node and each second computing node of the leaf layer in the computing power cluster as the destination node in the graph structure, and use the single-source shortest path algorithm to generate a first test path from each first node to each second node based on each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0239] Using each second node as the source node and each third computing node in the access layer of the computing power cluster as the destination node in the graph structure, the single-source shortest path algorithm is used to generate a second test path from each second node to each third node based on each edge corresponding to each node in the graph structure and the initial weight of each edge; the minimum spanning tree algorithm is used to generate a third test path based on each edge corresponding to each node in the graph structure and the initial weight of each edge; the first test path, the second test path, and the third test path are determined as the test path set.

[0240] In a possible implementation, the processing module 501 is further configured to: acquire a test result corresponding to each test path, the test result being an actual performance indicator value of the test path; update a weight of each edge corresponding to each test path in the graph structure according to a relationship between the actual performance indicator value of each test path and a preset performance indicator threshold; update the test path set by using the graph theory algorithm and the updated graph structure, and perform testing based on the updated test path set until a preset test completion condition is met.

[0241] In a possible implementation, the processing module 501 is specifically configured to, for each link in each test path: if an actual performance indicator value of the link meets a requirement of the preset performance indicator threshold, increase a weight of an edge corresponding to the link in the graph structure by a preset penalty value; and if the actual performance indicator value of the link does not meet the requirement of the preset performance indicator threshold, update the weight of the edge corresponding to the link in the graph structure according to a preset decay factor.

[0242] In a possible implementation, the actual performance indicator value is a measured bandwidth and / or a packet loss rate.

[0243] In a possible implementation, the processing module 501 is further configured to, for each edge corresponding to each test path in the graph structure: if an updated weight of the edge is lower than a preset minimum weight, update the weight of the edge to the minimum weight.

[0244] In a possible implementation, the processing module 501 is further configured to update the saved number of abnormalities corresponding to the link.

[0245] If the updated number of abnormalities reaches a preset number of abnormality threshold, acquire passive monitoring data corresponding to the link, the passive monitoring data including sFlow / NetFlow flow sampling, switch port cyclic redundancy check (CRC) error counting, forward error correction (FEC) statistics, optical module received optical power, and digital diagnostic monitoring (DDM) alarm information; perform abnormality detection on the link according to the passive monitoring data by using an abnormality detection model to determine a confidence score corresponding to the link; and if the confidence score exceeds a preset threshold, perform alarm.

[0246] In a possible implementation, the test module 502 is specifically configured to: divide the test path set into a plurality of test path subsets according to networks corresponding to each test path included in the test path set; for each test path subset, determine each test path including a share-free link in the test path subset; and perform parallel scheduling on each test path of the share-free link by using a preset test command.

[0247] In a possible implementation, the test module 502 is specifically configured to test each test path of the share-free link in at least one of the following manners: performing delay and packet loss tests on the test path by using a ping command or an hping3 command; performing bandwidth tests on the test path by using an iperf3 command, and performing point-to-point bandwidth throughput tests; and performing load tests on the test path by using an NVIDIA Collective Communication Library (NCCL).

[0248] Embodiment 8

[0249] Based on the same inventive concept, the embodiments of the present application provide an electronic device, which can implement the steps of the network test method of the computing power cluster discussed above, Figure 6 An electronic device structure diagram provided by the embodiments of the present application is shown in Figure 6 The electronic device structure diagram includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0250] The memory 603 stores a computer program, and when the program is executed by the processor 601, the processor 601 performs the following steps:

[0251] According to the obtained network information of the computing power cluster, a weighted undirected graph structure is constructed, wherein the network information carries the network configuration information of each computing node in the computing power cluster and the communication link information of the computing power cluster; the nodes of the graph structure represent the computing nodes in the computing power cluster, and the edges of the graph structure represent that the two nodes corresponding to the computing nodes have a communication link in the computing power cluster, and the initial weight of the edge is determined based on the network configuration information of the two nodes corresponding to the computing nodes.

[0252] A preset graph algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0253] According to each test path included in the test path set, the computing power cluster is tested.

[0254] In a possible implementation, the graph algorithm includes a single-source shortest path algorithm and a minimum spanning tree algorithm.

[0255] The preset graph algorithm is used to generate each test path according to each edge corresponding to each node in the graph structure and the initial weight of each edge.

[0256] each first computing node of a Spine layer in the computing power cluster takes each first node in the graph structure as a source node, each second computing node of a Leaf layer in the computing power cluster takes each second node in the graph structure as a destination node, and a single-source shortest path algorithm is used to generate a first test path from each first node to each second node according to each edge corresponding to each node in the graph structure and an initial weight of each edge;

[0257] each second node is taken as a source node, each third computing node of an Access layer in the computing power cluster takes each third node in the graph structure as a destination node, and the single-source shortest path algorithm is used to generate a second test path from each second node to each third node according to each edge corresponding to each node in the graph structure and an initial weight of each edge;

[0258] a minimum spanning tree algorithm is used to generate a third test path according to each edge corresponding to each node in the graph structure and an initial weight of each edge;

[0259] The first test path, the second test path and the third test path are determined as the test path set.

[0260] In a possible implementation, the method further includes:

[0261] a test result corresponding to each test path is obtained, and the test result is an actual performance index value of the test path;

[0262] a weight of each edge corresponding to each test path in the graph structure is updated according to a relationship between the actual performance index value of each test path and a preset performance index threshold;

[0263] the test path set is updated using the graph theory algorithm and the updated graph structure, and a test is performed based on the updated test path set until a preset test completion condition is met.

[0264] In a possible implementation, the updating of the weight of each edge corresponding to each test path in the graph structure according to the relationship between the actual performance index value of each test path and the preset performance index threshold includes:

[0265] for each link in each test path, if an actual performance index value of the link meets a requirement of the preset performance index threshold, a weight of an edge corresponding to the link in the graph structure is increased by a preset penalty value, and if the actual performance index value of the link does not meet the requirement of the preset performance index threshold, the weight of the edge corresponding to the link in the graph structure is updated according to a preset decay factor.

[0266] In a possible implementation, the actual performance index value is a measured bandwidth and / or a packet loss rate.

[0267] In a possible implementation, after the weight of the edge corresponding to the link in the graph structure is updated according to the preset attenuation factor, the method further includes:

[0268] For each edge corresponding to each test path in the graph structure, if the updated weight of the edge is lower than a preset minimum weight, the weight of the edge is updated to the minimum weight.

[0269] In a possible implementation, after the weight of the edge corresponding to the link in the graph structure is increased by a preset penalty value, the method further includes:

[0270] updating the saved number of abnormalities corresponding to the link;

[0271] if the updated number of abnormalities reaches a preset number of abnormality threshold, obtaining passive monitoring data corresponding to the link, the passive monitoring data including sFlow / NetFlow traffic sampling, switch port cyclic redundancy check (CRC) error count, forward error correction (FEC) statistics, optical module received optical power, and digital diagnostic monitoring (DDM) alarm information;

[0272] using an abnormality detection model to detect abnormalities of the link according to the passive monitoring data, and determining a confidence score corresponding to the link;

[0273] if the confidence score exceeds a preset threshold, performing an alarm.

[0274] In a possible implementation, the testing of the computing power cluster according to each test path included in the test path set includes:

[0275] dividing the test path set into a plurality of test path subsets according to networks corresponding to each test path included in the test path set;

[0276] for each test path subset, determining each test path including a non-shared link in the test path subset; and using a preset test command to schedule each test path of the non-shared link in parallel.

[0277] In a possible implementation, the using of a preset test command to schedule each test path of the non-shared link in parallel includes:

[0278] for each test path of the non-shared link, using at least one of the following ways to test:

[0279] Delay and packet loss test is performed on the test path using a ping command or an hping3 command;

[0280] Bandwidth test is performed on the test path using an iperf3 command to perform a point-to-point bandwidth throughput test;

[0281] Load test is performed on the test path using an NVIDIA Collective Communication Library (NCCL).

[0282] Since the principle of solving the problem of the above electronic device is similar to the network test method of the computing power cluster, the implementation of the above electronic device can refer to the embodiments of the method, and the repeated parts will not be described again.

[0283] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 602 is used for communication between the above electronic device and other devices. The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0284] The above processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0285] Embodiment 9:

[0286] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program executable by a processor. When the program runs on the processor, the processor performs the following steps:

[0287] According to the obtained network information of the computing power cluster, a weighted undirected graph structure is constructed, wherein the network information carries network configuration information of each computing node in the computing power cluster and communication link information of the computing power cluster; a node of the graph structure represents a computing node in the computing power cluster, and an edge of the graph structure represents that two connected nodes correspond to computing nodes having a communication link in the computing power cluster, and an initial weight of the edge is determined based on network configuration information of the two nodes corresponding computing nodes;

[0288] A preset graph algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge;

[0289] According to each test path included in the test path set, the computing power cluster is tested.

[0290] In a possible implementation, the graph algorithm includes a single-source shortest path algorithm and a minimum spanning tree algorithm;

[0291] The preset graph algorithm is used to generate each test path according to each edge corresponding to each node in the graph structure and the initial weight of each edge, including:

[0292] Each first node in the graph structure corresponding to each first computing node in a Spine layer of the computing power cluster is used as a source node, each second node in the graph structure corresponding to each second computing node in a Leaf layer of the computing power cluster is used as a destination node, the single-source shortest path algorithm is used to generate a first test path from each first node to each second node according to each edge corresponding to each node in the graph structure and the initial weight of each edge;

[0293] Each second node is used as a source node, each third node in the graph structure corresponding to each third computing node in an access layer of the computing power cluster is used as a destination node, the single-source shortest path algorithm is used to generate a second test path from each second node to each third node according to each edge corresponding to each node in the graph structure and the initial weight of each edge;

[0294] The minimum spanning tree algorithm is used to generate a third test path according to each edge corresponding to each node in the graph structure and the initial weight of each edge;

[0295] The first test path, the second test path and the third test path are determined as the test path set.

[0296] In a possible implementation, the method further includes:

[0297] obtaining a test result corresponding to each test path, the test result being an actual performance indicator value of the test path;

[0298] updating a weight of each edge corresponding to each test path in the graph structure according to a relationship between the actual performance indicator value corresponding to each test path and a preset performance indicator threshold;

[0299] updating the set of test paths using the graph theory algorithm and the updated graph structure, and performing testing based on the updated set of test paths until a preset test completion condition is met.

[0300] In a possible implementation, the updating of the weight of each edge corresponding to each test path in the graph structure according to the relationship between the actual performance indicator value corresponding to each test path and the preset performance indicator threshold comprises:

[0301] for each link in each test path, if the actual performance indicator value of the link meets the requirement of the preset performance indicator threshold, increasing the weight of the edge corresponding to the link in the graph structure by a preset penalty value; and if the actual performance indicator value of the link does not meet the requirement of the preset performance indicator threshold, updating the weight of the edge corresponding to the link in the graph structure according to a preset decay factor.

[0302] In a possible implementation, the actual performance indicator value is a measured bandwidth and / or a packet loss rate.

[0303] In a possible implementation, after the updating of the weight of the edge corresponding to the link in the graph structure according to the preset decay factor, the method further comprises:

[0304] for each edge corresponding to each test path in the graph structure, if the updated weight of the edge is lower than a preset minimum weight, updating the weight of the edge to the minimum weight.

[0305] In a possible implementation, after the increasing of the weight of the edge corresponding to the link in the graph structure by the preset penalty value, the method further comprises:

[0306] updating the saved number of exceptions corresponding to the link;

[0307] if the updated number of exceptions reaches a preset number of exception threshold, obtaining passive monitoring data corresponding to the link, the passive monitoring data including sFlow / NetFlow flow sampling, switch port cyclic redundancy check (CRC) error count, forward error correction (FEC) statistics, optical module received optical power, and digital diagnostic monitoring (DDM) alarm information.

[0308] An anomaly detection model is adopted to detect anomalies of the link according to the passive monitoring data, and determine a confidence score corresponding to the link;

[0309] If the confidence score exceeds a preset threshold, an alarm is given.

[0310] In a possible implementation, the testing of the computing power cluster according to each test path included in the test path set comprises:

[0311] The test path set is divided into a plurality of test path subsets according to the network corresponding to each test path included in the test path set;

[0312] For each test path subset, each test path of a non-shared link included in the test path subset is determined, and each test path of the non-shared link is scheduled in parallel using a preset test command.

[0313] In a possible implementation, the scheduling of each test path of the non-shared link in parallel using a preset test command comprises:

[0314] For each test path of the non-shared link, at least one of the following is used for testing:

[0315] A ping command or an hping3 command is used to test the delay and packet loss of the test path;

[0316] An iperf3 command is used to test the bandwidth of the test path, and a point-to-point bandwidth throughput test is performed;

[0317] An NVIDIA collective communication library (NCCL) is used to test the load of the test path.

[0318] Since the principle of solving the problem of the above computer readable storage medium is similar to that of the network testing method of the computing power cluster, the implementation of the above computer readable storage medium can be referred to the implementation of the method, and the repeated parts will not be described.

[0319] Embodiment 10:

[0320] Based on the same inventive concept, the embodiments of the present application also provide a computer program product, which comprises computer program code, when the computer program code runs on a computer, so that the computer executes any network testing method of the computing power cluster as discussed above. Since the principle of solving the problem of the above computer program product is similar to that of the network testing method of the computing power cluster, the implementation of the above computer program product can be referred to the implementation of the method, and the repeated parts will not be described.

[0321] In the embodiment of the present application, a weighted undirected graph structure is constructed according to the obtained network information of the computing power cluster, wherein the network information carries the network configuration information of each computing node in the computing power cluster and the communication link information of the computing power cluster; the nodes of the graph structure represent the computing nodes in the computing power cluster, the edges of the graph structure represent that the two connected nodes correspond to the computing nodes that exist communication links in the computing power cluster, and the initial weight of the edge is determined based on the network configuration information of the two nodes corresponding computing nodes; a preset graph algorithm is used to generate a test path set according to each edge corresponding to each node in the graph structure and the initial weight of each edge; and the computing power cluster is tested according to each test path included in the test path set. Through the innovative graph structure construction and test path generation strategy, the embodiment of the present application realizes the rapid, accurate and intelligent health diagnosis of the network infrastructure of the computing power cluster, and ensures that the network can meet the strict requirements of extremely low delay, extremely high bandwidth and zero packet loss for large-scale distributed AI training.

[0322] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0323] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0324] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0325] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart

[0326] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A network testing method of a computing power cluster, characterized in that, The method comprises: According to the obtained network information of the computing power cluster, a weighted undirected graph structure is constructed, wherein the network information carries the network configuration information of each computing node in the computing power cluster and the communication link information of the computing power cluster; the nodes of the graph structure represent the computing nodes in the computing power cluster, and the edges of the graph structure represent that the connected two nodes correspond to the computing nodes that exist communication links in the computing power cluster, and the initial weight of the edge is determined based on the network configuration information of the computing nodes corresponding to the two nodes; Using a preset graph theory algorithm, a test path set is generated according to each edge corresponding to each node in the graph structure and the initial weight of each edge; According to each test path included in the test path set, the computing power cluster is tested; The method further comprises: Obtaining a test result corresponding to each test path, the test result being an actual performance index value of the test path; According to the relationship between the actual performance index value corresponding to each test path and a preset performance index threshold, the weight of each edge corresponding to each test path in the graph structure is updated; Using the graph theory algorithm and the updated graph structure, the test path set is updated, and testing is performed based on the updated test path set until a preset test completion condition is met.

2. The method of claim 1, wherein, The graph theory algorithm comprises a single-source shortest path algorithm and a minimum spanning tree algorithm; The generation of each test path using the preset graph theory algorithm according to each edge corresponding to each node in the graph structure and the initial weight of each edge comprises: Taking each first node in the graph structure as a source node, taking each second node in the graph structure as a destination node, and using the single-source shortest path algorithm, a first test path from each first node to each second node is generated according to each edge corresponding to each node in the graph structure and the initial weight of each edge; Taking each second node as a source node, taking each third node in the graph structure as a destination node, and using the single-source shortest path algorithm, a second test path from each second node to each third node is generated according to each edge corresponding to each node in the graph structure and the initial weight of each edge; Using the minimum spanning tree algorithm, a third test path is generated according to each edge corresponding to each node in the graph structure and the initial weight of each edge; The first test path, the second test path and the third test path are determined as the test path set.

3. The method of claim 1, wherein, The updating of the weight of each edge corresponding to each test path in the graph structure according to the relationship between the actual performance index value corresponding to each test path and the preset performance index threshold comprises: For each link in each test path, if the actual performance indicator value of the link meets the requirement of the preset performance indicator threshold, the weight of the edge corresponding to the link in the graph structure is increased by a preset penalty value; if the actual performance indicator value of the link does not meet the requirement of the preset performance indicator threshold, the weight of the edge corresponding to the link in the graph structure is updated according to a preset attenuation factor.

4. The method of claim 3, wherein, The actual performance indicator value is a measured bandwidth and / or a packet loss rate.

5. The method of claim 3, wherein, After the weight of the edge corresponding to the link in the graph structure is updated according to the preset attenuation factor, the method further comprises: For each edge corresponding to each test path in the graph structure, if the updated weight of the edge is lower than a preset minimum weight, the weight of the edge is updated to the minimum weight.

6. The method of claim 3, wherein, After the weight of the edge corresponding to the link in the graph structure is increased by the preset penalty value, the method further comprises: updating the saved number of abnormalities corresponding to the link; if the updated number of abnormalities reaches a preset number of abnormality threshold, obtaining passive monitoring data corresponding to the link, the passive monitoring data including sFlow / NetFlow traffic sampling, switch port cyclic redundancy check (CRC) error count, forward error correction (FEC) statistics, optical module received optical power, and digital diagnostic monitoring (DDM) alarm information; using an abnormality detection model to detect abnormalities of the link according to the passive monitoring data, and determining a confidence score corresponding to the link; if the confidence score exceeds a preset threshold, an alarm is generated.

7. The method of claim 1, wherein, The testing of the computing power cluster according to each test path included in the test path set comprises: dividing the test path set into a plurality of test path subsets according to the network corresponding to each test path included in the test path set; for each test path subset, determining each test path including a non-shared link in the test path subset; using a preset test command, the each test path of the non-shared link is scheduled in parallel.

8. The method of claim 7, wherein, The parallel scheduling of the each test path of the non-shared link using the preset test command comprises: for each test path of the non-shared link, at least one of the following modes is used for testing: using a ping command or an hping3 command to test the delay and packet loss of the test path; using an iperf3 command to test the bandwidth of the test path, and performing a point-to-point bandwidth throughput test; using an NVIDIA collective communication library (NCCL) to test the load of the test path.

9. An electronic device, comprising: The electronic device comprises a processor configured to execute a computer program stored in a memory to implement the steps of the network testing method of the computing power cluster according to any one of claims 1-8.

Images