Multi-modal network testing method for AI application east-west flow

Abstract

The invention discloses a multi-mode network testing method for AI application east-west flow, and relates to the technical field of network testing, and the method comprises the steps: S1, constructing a testing environment; a non-blocking fat tree architecture is adopted, a high-performance switch and a GPU server are configured, and the high-performance switch and the GPU server are used for processing large-scale data transmission and calculation tasks and simulating a real AI cluster environment; s3, performing a performance isolation test; and S4, testing a congestion scene. According to the AI application east-west flow-oriented multi-modal network test method, a set of complete multi-modal network test system is provided, and the test scene and method of each sub-test are given, so that the problems of quality evaluation such as dynamic routing and congestion control of the AI application east-west flow-oriented multi-modal network are solved, and the test efficiency is improved. The multi-modal network can be evaluated in an omnibearing and multi-angle manner, and the performance of the multi-modal network in an AI application scene can be evaluated comprehensively and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network testing, in particular to a multi-modal network testing method for east-west traffic of AI applications. BACKGROUND

[0002] With the rapid development of AI technology, multi-modal networks are widely used in the AI field. Multi-modal networks integrate multiple underlying network protocols and different network architectures, and there are various upper-layer services. In this case, east-west traffic becomes the main part of AI cluster network traffic, which puts high requirements on network performance.

[0003] However, in actual use, most existing network testing methods are mainly for single network protocol or specific network architecture, and cannot comprehensively consider the influence of different protocol and architecture combinations in multi-modal networks on east-west traffic. For example, the traditional network performance testing tool iperf is mainly used to test the basic indicators such as network bandwidth and delay under TCP or UDP protocol, but it cannot adapt to the combination testing of different protocols and architectures in multi-modal networks. Therefore, the testing method for a specific network architecture can deeply analyze the performance of the architecture, but lacks consideration of the overall multi-modal network. However, in the AI training scenario, there may be parameter synchronization based on TCP / IP and large-scale data transmission based on RDMA, and the existing method is difficult to comprehensively test the traffic performance under these two protocols.

[0004] At the same time, the existing testing method often ignores the diversity of upper-layer AI application service requirements and cannot effectively simulate the diversified AI service requirements of the upper layer. Different AI applications (such as speech recognition and image classification) have great differences in network requirements, which makes the existing testing method unable to accurately match these requirements for testing, so that the test results cannot truly reflect the performance of the network in actual AI applications, and thus the network performance cannot be comprehensively evaluated from the perspective of actual applications. In addition, in multi-modal networks, the influence of network faults under different modalities on AI application data transmission is complex and diverse, and the existing testing method lacks effective detection and analysis means for these complex fault conditions, and cannot timely discover potential network risks. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal network testing method for east-west traffic of AI applications to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical solution: a multi-modal network testing method for east-west traffic of AI applications, comprising the following steps:

[0007] S1, constructing a test environment:

[0008] It adopts a non-blocking fat tree architecture, which has high scalability and low latency, making it suitable for handling large-scale data transmission and computing tasks. It is configured with high-performance switches and GPU servers to handle large-scale data transmission and computing tasks, simulating a real AI cluster environment.

[0009] S2, Benchmark Performance Test:

[0010] S21, RDMA split test: Perform RDMA split test on each network card on all servers in the cluster to evaluate bidirectional bandwidth and P99 latency performance under different network topologies;

[0011] S22, NCCL Cluster Communication Test: Through testing various NCCL workloads, the communication efficiency and bandwidth performance of the cluster are comprehensively measured, providing data support for subsequent optimization and adjustment;

[0012] S3, Performance Isolation Test:

[0013] S31. AllReduce Performance Isolation Test: Under noisy workload conditions, the average bandwidth of the target workload's NCCL All Reduce is tested. In actual operation, there are 4 workload groups, each requiring 4 nodes. Three workload groups act as noise, iteratively sending and receiving typical 4GB messages, while the other workload performs NCCL All Reduce aggregate communication. The impact between different workload groups is observed, covering optimal deployment, normal deployment, and distributed deployment scenarios. The average bandwidth (GB / s) of the target workload's NCCL All Reduce is collected and analyzed.

[0014] S32. All-to-All Performance Isolation Test: Under noisy workload conditions, test the average NCCL all-to-all bandwidth of the target workload. In actual operation, there are 4 workload groups, each requiring 4 nodes; 3 workload groups are used as noise for all-to-all aggregated communication, and 1 workload group is used as the observation object for NCCL all-to-all aggregated communication. Observe the impact between different workload groups in this scenario, covering optimal deployment, normal deployment, and distributed deployment scenarios. Collect and analyze the average NCCL all-to-all bandwidth (GB / s) of the target workload.

[0015] S33. All Reduce / All to All Performance Isolation Test: Under noisy workload conditions, test the average bandwidth of NCCL All to All and NCCL All Reduce for the target workload. In actual operation, a distributed deployment scenario is used, with a total of 4 workloads. Three workloads are used as noise for All-to-All aggregated communication, and one workload is used as the observation object for NCCL All Reduce aggregated communication. Observe the impact between different workloads in this scenario. Collect and analyze the average bandwidth (GB / s) of NCCL All to All and NCCL All Reduce for the target workload.

[0016] S34, LLM Application Performance Isolation Test: In a simulated mixed load scenario of a real production environment, the impact of network noise on the AI ​​workload is evaluated, recording the single iteration time under both no and background pressure conditions. In actual operation, an AI model is run on four nodes across two Leaf switches to construct the AI ​​workload, while iterative sending and receiving operations with typical 4GB messages are repeatedly performed on the remaining 14 nodes to generate network noise. This configuration simulates a common mixed load scenario in real production environments, thereby evaluating the impact of network noise on the AI ​​workload. Specifically, the AI ​​workload includes complex deep learning model training tasks that require frequent aggregate communication operations (All Reduce, All to All) to synchronize parameters between nodes. The network noise component simulates the bandwidth consumption of other background tasks through a large number of data transmission operations, creating interference. In this scenario, the test records the iteration time of AI model training and the bandwidth utilization of network communication to evaluate the specific impact of network noise on the AI ​​workload, ensuring that the test environment is as close as possible to the actual usage scenario, thus providing reliable data support for optimizing cluster performance. Test metrics: single iteration time (s) without background pressure; single iteration time (s) with background pressure.

[0017] S4, Congestion Scenario Test:

[0018] S41. NCCL All Reduce Test in a Multi-Target Scenario: Construct two sets of loads, one set being the NCCL All Reduce workload of the main observation object, and the other set being the background load that creates a congestion scenario. Test and record the bandwidth performance of the All Reduce workload under congestion conditions.

[0019] S42. NCCL All to All Test in a Multi-to-One Scenario: Construct two sets of workloads: one set is the NCCL All to All workload that is the main observation target, and the other set is the background workload that creates a congestion scenario. Test and record the bandwidth performance of the All to All workload under congestion conditions.

[0020] Furthermore, in step S21, Ib_write_bw is used to test bidirectional bandwidth, and Ib_write_lat is used to test P99 latency. The specific operations are as follows:

[0021] For each network interface card on all servers in the cluster, use the Ib_write_bw tool to test bidirectional bandwidth, while enabling bi-directional mode;

[0022] Use the Ib_write_lat tool to test P99 latency;

[0023] Record the average bandwidth (average bandwidth in bidirectional bandwidth test, unit: Gb / s) and P99 latency (latency in P99 latency test, unit: μs), compare the test results with those of the same Leaf switch and across Leaf switches, and analyze the impact of network topology on bandwidth and latency.

[0024] Furthermore, the test configuration includes configuring a single QP (i.e., each network card is tested using a single QP) and ensuring that the GPU and network card are mapped one-to-one according to the host topology.

[0025] Furthermore, the test scenarios include two traffic forwarding scenarios: same-leaf switch testing and cross-leaf switch testing, to evaluate performance under different network topologies, as detailed below:

[0026] Tested with Leaf switches: Test traffic does not cross Spine switches, and all traffic is forwarded within the same Leaf switch;

[0027] Cross-Leaf Switch Testing: Test traffic crosses Spine switches and needs to be forwarded between different Leaf switches.

[0028] Furthermore, in step S22, the NCCL workload is tested as follows:

[0029] Full Cluster NCCL All Reduce Test: Configure the entire cluster as a single NCCL All Reduce workload to test the average bandwidth of the full data reduction operation and evaluate the cluster's communication efficiency when handling large-scale data synchronization. Specific operation: Configure all servers in the entire cluster as a single NCCL All Reduce workload to test the cluster's average bandwidth performance in the full data reduction operation. The AllReduce operation involves reducing the data on all nodes and broadcasting the results to all nodes. The evaluation metric for this test is the average NCCL All Reduce bandwidth (GBps) across all GPU servers.

[0030] Full Cluster NCCL All to All Test: Configure the entire cluster as an NCCL All to All workload to test the average bandwidth of the full data exchange operation and evaluate the cluster's communication efficiency when handling large-scale data transmission and reception. Specific operation: Configure all servers in the entire cluster as an NCCL All to All workload and test the cluster's average bandwidth performance in the full data exchange operation. In the All to All operation, each node will send data to any node and receive data from any node. The evaluation metric for this test is the average NCCL All to All bandwidth (GB / s) across all GPU servers.

[0031] Full Cluster NCCL ReduceScatter Test: Configure the entire cluster as a single NCCL ReduceScatter workload to test the average bandwidth of the data reduction and scattering operation, and evaluate the cluster's communication efficiency when handling large-scale data reduction and distribution. Specific operation: Configure all servers in the entire cluster as a single NCCL ReduceScatter workload, and test the cluster's average bandwidth performance in the data reduction and scattering operation. The ReduceScatter operation involves reducing the data of all nodes and distributing the results to each node. The evaluation metric for this test is the average bandwidth (GB / s) of NCCLReduceScatter across all GPU servers.

[0032] Full Cluster NCCL All Gather Test: Configure the entire cluster as a single NCCL All Gather workload to test the average bandwidth of the full data collection operation and evaluate the cluster's communication efficiency when handling large-scale data collection and broadcasting. Specifically, configure all servers in the cluster as a single NCCL All Gather workload and test the cluster's average bandwidth performance during the full data collection operation. The All Gather operation involves collecting data from each node and broadcasting it to all nodes. The evaluation metric for this test is the average NCCL All Gather bandwidth (GB / s) across all GPU servers.

[0033] Furthermore, in step S3, the deployment scenarios for performance isolation testing include conventional deployment scenarios, distributed deployment scenarios, and optimal deployment scenarios, as detailed below:

[0034] Typical deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, with 2 nodes under the same Leaf switch;

[0035] Distributed deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, and each node is distributed across different Leaf switches;

[0036] Optimal deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, and all 4 nodes are under the same Leaf switch.

[0037] Furthermore, in step S41, the two load conditions are as follows:

[0038] The first load consists of 14 GPU servers, each running an NCCLAll-Reduce workload. As the primary observation targets, these servers will work together to perform a large number of data exchange operations to simulate high-load scenarios in real-world applications.

[0039] The second workload is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 through the ib_write_bw tool, forming a background workload and creating a multi-to-one congestion scenario. In this way, the bandwidth performance of the All-Reduce workload under congestion conditions can be observed.

[0040] Furthermore, the specific operations of the NCCL All Reduce test in the multi-to-one scenario are as follows:

[0041] First, the NCCLAll-Reduce workload was started on 14 GPU servers, and its initial bandwidth performance data was recorded.

[0042] Next, the ib_write_bw tool was started on two background load servers to create a multi-to-one congestion scenario, and the bandwidth changes of the All-Reduce workload were continuously monitored.

[0043] Finally, bandwidth data under different load conditions are analyzed and compared to evaluate the performance of NCCLAll-Reduce.

[0044] Furthermore, in step S42, the two load conditions are as follows:

[0045] The first load consists of 14 GPU servers, each running an NCCLAll to All workload. As the primary observation targets, these servers will perform extensive data exchange operations to simulate complex communication patterns in real-world applications.

[0046] The second load is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 through the ib_write_bw tool, forming a background load and creating a multi-to-one congestion scenario. In this way, the bandwidth performance of the All-to-All workload under congestion conditions can be observed.

[0047] Furthermore, the specific operations of the NCCL All-to-All test in the multi-to-one scenario are as follows:

[0048] First, the NCCLAll to All workload was started on 14 GPU servers, and its initial bandwidth performance data was recorded.

[0049] Next, the ib_write_bw tool was started on two background load servers to create a multi-to-one congestion scenario, and the bandwidth changes of the All-to-All workload were continuously monitored.

[0050] Finally, bandwidth data under different load conditions were analyzed and compared to evaluate the performance of NCCLAll to All.

[0051] This invention provides a multimodal network testing method for east-west traffic in AI applications, which has the following beneficial effects:

[0052] 1. This invention proposes a complete multimodal network testing system and provides test scenarios and methods for each sub-test. It solves the quality evaluation problems of dynamic routing and congestion control of multimodal networks for east-west traffic in AI applications. It can evaluate multimodal networks from all aspects and multiple angles, and comprehensively and accurately assess the performance of multimodal networks in AI application scenarios.

[0053] 2. By comprehensively considering the diversity of underlying network protocols, architectures, and upper-layer AI service requirements, this invention effectively detects the impact of network faults on AI application data transmission under different modalities, providing a strong basis for the optimization and improvement of multimodal networks, thereby improving the reliability, stability, and performance of multimodal networks in AI applications and ensuring the efficient operation of AI training tasks.

[0054] 3. During the testing process, this invention constructed a testbed that highly simulates a real-world environment, employing a non-blocking fat-tree architecture and configuring high-performance switches and GPU servers. Through benchmark performance testing, NCCL aggregated communication testing, performance isolation testing, and congestion scenario testing, the network performance under different network topologies and load conditions was comprehensively evaluated. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the operation steps of a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0056] Figure 2 This is a schematic diagram of a Leaf switch test for a multimodal network testing method for east-west traffic of AI applications according to the present invention.

[0057] Figure 3 This is a schematic diagram of cross-Leaf switch testing for a multimodal network testing method for east-west traffic of AI applications according to the present invention.

[0058] Figure 4 This is a schematic diagram of a full-cluster NCCLAll Reduce test for a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0059] Figure 5 This is a schematic diagram of the full-cluster NCCLAll to All test of a multimodal network testing method for east-west traffic in AI applications according to the present invention;

[0060] Figure 6 This is a schematic diagram of the full-cluster NCCLCReduceScatter test of a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0061] Figure 7This is a schematic diagram of the full-cluster NCCLAll Gather test of a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0062] Figure 8 This is a schematic diagram of a typical deployment scenario for a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0063] Figure 9 This is a schematic diagram of a distributed deployment scenario for a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0064] Figure 10 This is a schematic diagram illustrating the optimal deployment scenario for a multimodal network testing method for east-west traffic in AI applications according to the present invention.

[0065] Figure 11 This invention provides an All Reduce performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 1 ;

[0066] Figure 12 This invention provides an All Reduce performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 2 ;

[0067] Figure 13 This invention provides an All Reduce performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 3 ;

[0068] Figure 14 This invention provides an All-to-All performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 1 ;

[0069] Figure 15 This invention provides an All-to-All performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 2 ;

[0070] Figure 16 This invention provides an All-to-All performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 3 ;

[0071] Figure 17 This is an All Reduce / All to All performance isolation test diagram of a multimodal network testing method for east-west traffic in AI applications according to the present invention;

[0072] Figure 18This invention provides an LLM application performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 1 ;

[0073] Figure 19 This invention provides an LLM application performance isolation test for a multimodal network testing method for east-west traffic in AI applications. Figure 2 ;

[0074] Figure 20 This is a test graph of NCCL All Reduce in a multi-hit scenario for a multimodal network testing method for east-west traffic in AI applications, as described in this invention.

[0075] Figure 21 This is a multi-to-one test diagram of NCCL All to All in a multimodal network testing method for east-west traffic in AI applications, as described in this invention. Detailed Implementation

[0076] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0077] like Figures 1-21 As shown, a multimodal network testing method for east-west traffic in AI applications includes the following steps:

[0078] S1. Building the test environment:

[0079] It adopts a non-blocking fat tree architecture, which has high scalability and low latency, making it suitable for handling large-scale data transmission and computing tasks. It is configured with high-performance switches and GPU servers to handle large-scale data transmission and computing tasks, simulating a real AI cluster environment.

[0080] S2, Benchmark Performance Test:

[0081] S21. RDMA Partition Mapping Test: Perform RDMA partition mapping tests on each network interface card (NIC) on all servers within the cluster to evaluate bidirectional bandwidth and P99 latency performance under different network topologies. Specifically, Ib_write_bw is used to test bidirectional bandwidth, and Ib_write_lat is used to test P99 latency. Detailed operation:

[0082] For each network interface card on all servers in the cluster, use the Ib_write_bw tool to test bidirectional bandwidth, while enabling bi-directional mode;

[0083] Use the Ib_write_lat tool to test P99 latency;

[0084] Record the average bandwidth (average bandwidth in bidirectional bandwidth test, unit: Gb / s) and P99 latency (latency in P99 latency test, unit: μs), compare the test results with those of the same Leaf switch and across Leaf switches, and analyze the impact of network topology on bandwidth and latency.

[0085] In this embodiment, the test configuration includes configuring a single QP (i.e., each network interface card (NIC) is tested using a single QP) and ensuring a one-to-one mapping between the GPU and NIC according to the host topology. The test scenarios include two traffic forwarding scenarios: testing within the same Leaf switch and testing across Leaf switches, to evaluate performance under different network topologies, as detailed below:

[0086] Tested with Leaf switches: Test traffic did not cross Spine switches; all traffic was forwarded within the same Leaf switch. See [link / details]. Figure 2 ;

[0087] Cross-Leaf Switch Testing: Test traffic crosses Spine switches and needs to be forwarded between different Leaf switches. See [link / details]. Figure 3 .

[0088] S22, NCCL Cluster Communication Test: Through various NCCL workload tests, the cluster's communication efficiency and bandwidth performance are comprehensively measured, providing data support for subsequent optimization and adjustments. The NCCL workload tests are as follows:

[0089] Full cluster NCCL All Reduce test (see) Figure 4 ): Configure the entire cluster as a single NCCL All Reduce workload to test the average bandwidth of the full data reduction operation and evaluate the cluster's communication efficiency when handling large-scale data synchronization. Specific operation: Configure all servers in the entire cluster as a single NCCL All Reduce workload to test the cluster's average bandwidth performance in the full data reduction operation. The All Reduce operation involves reducing the data of all nodes and broadcasting the results to all nodes. The evaluation metric for this test is the average NCCL All Reduce bandwidth (GBps) across all GPU servers.

[0090] Full cluster NCCL All to All test (see) Figure 5): Configure the entire cluster as an NCCL All-to-All workload to test the average bandwidth of the full data exchange operation and evaluate the cluster's communication efficiency when handling large-scale data transmission and reception. Specific operation: Configure all servers in the entire cluster as an NCCL All-to-All workload and test the cluster's average bandwidth performance in the full data exchange operation. In the All-to-All operation, each node will send data to any node and receive data from any node. The evaluation metric for this test is the average bandwidth (GB / s) of the NCCL All-to-All operation across all GPU servers.

[0091] Full cluster NCCL ReduceScatter test (see) Figure 6 ): Configure the entire cluster as a single NCCLReduceScatter workload to test the average bandwidth of the data reduction and scattering operation, and evaluate the cluster's communication efficiency when handling large-scale data reduction and distribution. Specific operation: Configure all servers in the entire cluster as a single NCCLReduceScatter workload, and test the cluster's average bandwidth performance in the data reduction and scattering operation. The ReduceScatter operation involves reducing the data of all nodes and distributing the results to each node. The evaluation metric for this test is the average bandwidth (GB / s) of NCCL ReduceScatter across all GPU servers.

[0092] Full cluster NCCL All Gather test (see) Figure 7 ): Configure the entire cluster as a single NCCL All Gather workload to test the average bandwidth of the full data collection operation and evaluate the cluster's communication efficiency when handling large-scale data collection and broadcasting. Specific operation: Configure all servers in the entire cluster as a single NCCL All Gather workload to test the cluster's average bandwidth performance during the full data collection operation. The All Gather operation involves collecting data from each node and broadcasting it to all nodes. The evaluation metric for this test is the average bandwidth (GB / s) of NCCL All Gather across all GPU servers.

[0093] S3, Performance Isolation Test:

[0094] The deployment scenarios for performance isolation testing include standard deployment scenarios, distributed deployment scenarios, and optimal deployment scenarios, as detailed below:

[0095] Typical deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, with 2 nodes under the same Leaf switch, such as... Figure 8 As shown;

[0096] Distributed deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, and each node is distributed across different Leaf switches, such as... Figure 9 As shown;

[0097] Optimal deployment scenario: In this test scenario, each aggregate communication workload requires 4 nodes, and all 4 nodes are under the same Leaf switch, such as... Figure 10 As shown.

[0098] S31, AllReduce performance isolation test, see Figures 11-13 In the presence of noisy workloads, the average NCCL All Reduce bandwidth of the target workload is tested. In this embodiment, there are 4 workload groups, each requiring 4 nodes. Three workload groups are used as noise, repeatedly sending and receiving typical 4GB messages, while one workload group performs NCCL All Reduce aggregated communication. The impact between different workload groups is observed in this scenario, covering optimal deployment, normal deployment, and distributed deployment scenarios. The average NCCL All Reduce bandwidth (GB / s) of the target workload is collected and analyzed.

[0099] S32, Allto All performance isolation test, see Figures 14-16 In this scenario, the average NCCL all-to-all bandwidth of the target workload is tested under noisy workload conditions. In this example, there are four workload groups, each requiring four nodes. Three workload groups are used as noise for all-to-all aggregated communication, and one workload group is used as the observation target for NCCL all-to-all aggregated communication. The impact between different workload groups is observed, covering optimal deployment, conventional deployment, and distributed deployment scenarios. The average NCCL all-to-all bandwidth (GB / s) of the target workload is collected and analyzed.

[0100] S33, All Reduce / All to All performance isolation test, see Figure 17 In the presence of noisy workloads, the average bandwidth of NCCL11 to All and NCCL11 Reduce for the target workload is tested. In this embodiment, a distributed deployment scenario is used, with a total of 4 workloads. Three workloads are used as noise for All-to-All aggregated communication, and one workload is used as the observation object for NCCL11 to All and Reduce aggregated communication. The impact between different workload groups in this scenario is observed. The average bandwidth (GB / s) of NCCL11 to All and NCCL11 Reduce for the target workload is collected and analyzed.

[0101] S34 and LLM application performance isolation tests, see Figures 18-19 This study assesses the impact of network noise on AI workloads under a simulated mixed load scenario in a real-world production environment, recording the single iteration time under both no-background-pressure and background-pressure conditions. In this embodiment, an AI model is run on four nodes across two Leaf switches to construct the AI ​​workload. Simultaneously, iterative sending and receiving operations with typical 4GB messages are repeatedly performed on the remaining 14 nodes to generate network noise. This configuration simulates a common mixed load scenario in real-world production environments, thereby evaluating the impact of network noise on the AI ​​workload. Specifically, the AI ​​workload includes complex deep learning model training tasks that require frequent aggregate communication operations (All Reduce, All to All) to synchronize parameters between nodes. The network noise component simulates the bandwidth consumption of other background tasks through a large number of data transmission operations, creating interference. In this scenario, the test records the iteration time of AI model training and the bandwidth utilization of network communication to assess the specific impact of network noise on the AI ​​workload, ensuring the test environment closely approximates real-world usage scenarios, thus providing reliable data support for optimizing cluster performance. Test metrics: Single iteration time (s) under no-background-pressure; Single iteration time (s) under background-pressure.

[0102] S4, Congestion Scenario Test:

[0103] S41. NCCL All Reduce Test in a Multi-Target Scenario: Construct two load sets: one is the NCCL All Reduce workload for the main observation, and the other is the background workload to create a congestion scenario. Test and record the bandwidth performance of the All Reduce workload under congestion conditions. The two load sets are as follows:

[0104] The first load consists of 14 GPU servers, each running an NCCLAll-Reduce workload. As the primary observation targets, these servers will work together to perform a large number of data exchange operations to simulate high-load scenarios in real-world applications.

[0105] The second workload is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 through the ib_write_bw tool, forming a background workload and creating a multi-to-one congestion scenario. In this way, the bandwidth performance of the All-Reduce workload under congestion conditions can be observed.

[0106] In this embodiment, the specific operations of the NCCL All Reduce test in a multi-target scenario are as follows:

[0107] First, the NCCLAll-Reduce workload was started on 14 GPU servers, and its initial bandwidth performance data was recorded.

[0108] Next, the ib_write_bw tool was started on two background load servers to create a multi-to-one congestion scenario, and the bandwidth changes of the All-Reduce workload were continuously monitored.

[0109] Finally, bandwidth data under different load conditions are analyzed and compared to evaluate the performance of NCCLAll-Reduce.

[0110] S42. NCCL All-to-All Test in a Multi-Target Scenario: Construct two load sets: one is the NCCL All-to-All workload for the main observation, and the other is the background workload to create a congestion scenario. Test and record the bandwidth performance of the All-to-All workload under congestion conditions. The two load sets are as follows:

[0111] The first workload consists of 14 GPU servers, each running an NCCLAll to All workload. As the primary observation targets, these servers will perform extensive data exchange operations to simulate complex communication patterns in real-world applications.

[0112] The second load is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 through the ib_write_bw tool, forming a background load and creating a multi-to-one congestion scenario. In this way, the bandwidth performance of the All-to-All workload under congestion conditions can be observed.

[0113] In this embodiment, the specific operation of NCCL All to All testing in a multi-to-one scenario is as follows:

[0114] First, the NCCLAll to All workload was started on 14 GPU servers, and its initial bandwidth performance data was recorded.

[0115] Next, the ib_write_bw tool was started on two background load servers to create a multi-to-one congestion scenario, and the bandwidth changes of the All-to-All workload were continuously monitored.

[0116] Finally, bandwidth data under different load conditions were analyzed and compared to evaluate the performance of NCCLAll to All.

[0117] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A multimodal network testing method for east-west traffic in AI applications, characterized in that, Includes the following steps: S1. Building the test environment: It adopts a non-blocking fat tree architecture and is configured with high-performance switches and GPU servers to handle large-scale data transmission and computing tasks, simulating a real AI cluster environment; S2, Benchmark Performance Test: S21, RDMA split test: Perform RDMA split test on each network card on all servers in the cluster to evaluate bidirectional bandwidth and P99 latency performance under different network topologies; S22, NCCL Cluster Communication Test: Through testing various NCCL workloads, the communication efficiency and bandwidth performance of the cluster are comprehensively measured; S3, Performance Isolation Test: S31, AllReduce performance isolation test: Under noisy workload conditions, test the average bandwidth of NCCL All Reduce for the target workload; S32, All-to-All Performance Isolation Test: Under noisy workload conditions, the average NCCL All-to-All bandwidth of the target workload is tested; S33, All Reduce / All to All Performance Isolation Test: Under noisy workload conditions, test the average bandwidth of NCCLAll to All and NCCLAll Reduce for the target workload; S34, LLM application performance isolation test: In a mixed load scenario simulating a real production environment, evaluate the impact of network noise on AI workload, and record the single iteration time under no background pressure and with background pressure. S4, Congestion Scenario Test: S41. NCCL All Reduce Test in a Multi-Target Scenario: Build two sets of workloads, test and record the bandwidth performance of the All Reduce workload under congestion conditions; S42. NCCL All to All Test in a Multi-to-One Scenario: Build two sets of workloads, test and record the bandwidth performance of the All to All workload under congestion conditions.

2. The multimodal network testing method for east-west traffic in AI applications according to claim 1, characterized in that, In step S21, Ib_write_bw is used to test bidirectional bandwidth, and Ib_write_lat is used to test P99 latency. The specific operations are as follows: For each network interface card on all servers in the cluster, use the Ib_write_bw tool to test bidirectional bandwidth, while enabling bi-directional mode; Use the Ib_write_lat tool to test P99 latency; Record the average bandwidth and P99 latency, compare the test results with those of the same Leaf switch and across Leaf switches, and analyze the impact of network topology on bandwidth and latency.

3. The multimodal network testing method for east-west traffic in AI applications according to claim 2, characterized in that, The test configuration includes configuring a single QP and ensuring that the GPU and network card are mapped one-to-one according to the host topology.

4. A multimodal network testing method for east-west traffic in AI applications according to claim 3, characterized in that, The test scenarios include two traffic forwarding scenarios: same-Leaf switch testing and cross-Leaf switch testing, as detailed below: Tested with Leaf switches: Test traffic does not cross Spine switches, and all traffic is forwarded within the same Leaf switch; Cross-Leaf Switch Testing: Test traffic crosses Spine switches and needs to be forwarded between different Leaf switches.

5. A multimodal network testing method for east-west traffic in AI applications according to claim 1, characterized in that, In step S22, the NCCL workload is tested as follows: Full Cluster NCCL All Reduce Test: Configure the entire cluster as a single NCCL All Reduce workload, test the average bandwidth of the full data reduction operation, and evaluate the cluster's communication efficiency when handling large-scale data synchronization. Full Cluster NCCL All to All Test: Configure the entire cluster as an NCCL All to All workload, test the average bandwidth of the full data exchange operation, and evaluate the communication efficiency of the cluster when handling large-scale data transmission and reception; Full Cluster NCCL ReduceScatter Test: Configure the entire cluster as a single NCCL ReduceScatter workload to test the average bandwidth of the data reduction and scattering operation, and evaluate the cluster's communication efficiency when handling large-scale data reduction and distribution. Full Cluster NCCL All Gather Test: Configure the entire cluster as an NCCL All Gather workload to test the average bandwidth of the full data collection operation and evaluate the cluster's communication efficiency when handling large-scale data collection and broadcasting.

6. A multimodal network testing method for east-west traffic in AI applications according to claim 1, characterized in that, In step S3, the deployment scenarios for performance isolation testing include conventional deployment scenarios, distributed deployment scenarios, and optimal deployment scenarios, as detailed below: Typical deployment scenario: Each aggregate communication workload has 4 nodes, with 2 nodes under the same Leaf switch; Distributed deployment scenario: Each aggregate communication workload has 4 nodes, and each node is distributed across different Leaf switches; Optimal deployment scenario: Each aggregate communication workload has 4 nodes, and all 4 nodes are under the same Leaf switch.

7. A multimodal network testing method for east-west traffic in AI applications according to claim 1, characterized in that, In step S41, the two load conditions are as follows: The first load consists of 14 GPU servers, each running an NCCLAll-Reduce workload. As the primary observation targets, these servers will work together to perform a large number of data exchange operations to simulate high-load scenarios in real-world applications. The second workload is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 through the ib_write_bw tool to form a background workload, thereby creating a multi-to-one congestion scenario to observe the bandwidth performance of the All-Reduce workload under congestion conditions.

8. A multimodal network testing method for east-west traffic in AI applications according to claim 7, characterized in that, The specific steps of the NCCL All Reduce test in the multi-to-one scenario are as follows: First, start the NCCL All-Reduce workload on 14 GPU servers and record its initial bandwidth performance data; then, start the ib_write_bw tool on two background load servers to create a multi-to-one congestion scenario and continuously monitor the bandwidth changes of the All-Reduce workload; finally, analyze and compare the bandwidth data under different load conditions to evaluate the performance of NCCL All-Reduce.

9. A multimodal network testing method for east-west traffic in AI applications according to claim 1, characterized in that, In step S42, the two load conditions are as follows: The first load consists of 14 GPU servers, each running an NCCLAll to All workload. As the primary observation targets, these servers will perform extensive data exchange operations to simulate complex communication patterns in real-world applications. The second load is built on the remaining two GPU servers: the eight GPUs on GPU server 16 will send data to one GPU on GPU server 1 via the ib_write_bw tool to create background load, thereby creating a multi-to-one congestion scenario to observe the bandwidth performance of the All-to-All workload under congestion conditions.

10. A multimodal network testing method for east-west traffic in AI applications according to claim 9, characterized in that, The specific operation of the NCCL All to All test in the multi-to-one scenario is as follows: First, start the NCCL All to All workload on 14 GPU servers and record its initial bandwidth performance data; then, start the ib_write_bw tool on two background load servers to create a multi-to-one congestion scenario and continuously monitor the bandwidth changes of the All to All workload; finally, analyze and compare the bandwidth data under different load conditions to evaluate the performance of NCCL All to All.

Images