Network performance monitoring system of large-scale intelligent computing center

Abstract

The invention discloses a network performance monitoring system of a large-scale intelligent computing center. The system comprises a task management module, a data acquisition module and a monitoring execution module. The task management module is configured to execute various test tasks on the network of the large-scale intelligent computing center; the data acquisition module is used for acquiring various parameters of a plurality of network devices in the process that the network executes various test tasks; the monitoring execution parameters are used for processing various parameters to obtain various network performance parameters of the large-scale intelligent computing center. On the basis of obtaining corresponding network performance parameters, a user can optimize a network architecture for a large-scale intelligent calculation center, and powerful support is provided for improving calculation efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent computing technology, and more specifically, to a network performance monitoring system for a large-scale intelligent computing center. Background Art

[0002] Large-scale intelligent computing centers are important infrastructure in modern society. They are used to provide users with super computing services and play an important role in modern meteorology, petroleum, chemical industry, military, manufacturing, scientific research and teaching. In order to ensure the safe and stable operation of the computing center, it is necessary to monitor and evaluate its operating status in order to provide strong support for optimizing the network architecture and improving computing efficiency.

[0003] Since large-scale intelligent computing centers are generally composed of massive computing nodes connected by a network, their monitoring and evaluation rely on the detection of network performance. That is, it is necessary to collect network-related data in real time and accurately, and effectively process the collected data in order to obtain network performance indicators. Summary of the invention

[0004] In view of this, the present application provides a network performance monitoring system for a large-scale intelligent computing center, which is used to monitor the network performance of the large-scale intelligent computing center in real time, so as to provide strong support for optimizing the network architecture and improving computing efficiency.

[0005] In order to achieve the above objectives, the proposed solution is as follows:

[0006] A network performance monitoring system for a large intelligent computing center, the network performance monitoring system comprising a task management module, a data acquisition module and a monitoring execution module, wherein:

[0007] The task management module is configured to perform a variety of test tasks on the network of the large intelligent computing center;

[0008] The data acquisition module is used to collect various parameters of multiple network devices during the process of executing the various test tasks on the network;

[0009] The monitoring execution parameters are used to process the multiple parameters to obtain multiple network performance parameters of the large-scale intelligent computing center.

[0010] Optionally, the multiple test tasks include part or all of a bandwidth delay test task, a many-to-one test task and a robustness test task.

[0011] Optionally, the multiple network devices include part or all of high-speed interfaces, optical modules, switches and network cards, and the multiple parameters include part or all of high-speed interface parameters, optical module parameters, network card parameters and switch parameters.

[0012] Optionally, the high-speed interface parameters include part or all of channel quantity information, connection status information and error information.

[0013] Optionally, the optical module parameters include part or all of bias current, module temperature, laser input and output power, receiver optical power and module voltage.

[0014] Optionally, the network card parameters include part or all of the link status, total number of received packets, total number of sent packets, total number of received packet bytes, total number of sent packet bytes, number of error packets, number of lost packets, and number of oversized packets.

[0015] Optionally, the switch parameters include switch temperature, version number, manufacturer model, serial number, power, CPU occupancy parameters, memory parameters and multiple port information.

[0016] Optionally, the monitoring execution module includes a link detection unit, a load balancing detection unit, a congestion and fault tolerance detection unit and an optical module fault detection unit.

[0017] Optionally, the link detection unit is configured to perform data accuracy detection, out-of-order reassembly effectiveness detection and transmission delay detection, as well as detection of the number and percentage of error packets, packet loss, and retransmission packets on each link, and is also used to detect bandwidth changes.

[0018] Optionally, the load balancing detection unit is configured to detect uniformity of traffic distribution in units of links.

[0019] It can be seen from the above technical solutions that the present application discloses a network performance monitoring system for a large intelligent computing center, which includes a task management module, a data acquisition module, and a monitoring execution module. The task management module is configured to perform various test tasks on the network of the large intelligent computing center; the data acquisition module is used to collect various parameters of multiple network devices during the execution of various test tasks on the network; the monitoring execution parameters are used to process various parameters to obtain various network performance parameters of the large intelligent computing center. On the basis of obtaining the corresponding network performance parameters, users can optimize the network architecture of the large intelligent computing center and provide strong support for improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1A block diagram of a network performance monitoring system for a large intelligent computing center according to an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a multiple-to-one test according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a data acquisition module according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a monitoring execution module according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Figure 1 This is a block diagram of a network performance monitoring system for a large intelligent computing center according to an embodiment of the present application.

[0027] The network performance monitoring system provided in this example is used to monitor the network performance of a large intelligent computing center and obtain its network performance parameters. The large intelligent computing center in this application includes but is not limited to the intelligent computing center. The network performance monitoring system can perform comprehensive management of large intelligent computing centers, including bandwidth delay test tasks, multiple-to-one test tasks, load balancing test tasks, and robustness test tasks. It is also responsible for the real-time and accurate collection of data after the task is started, including network card data collection, high-speed interface data collection, optical module data collection, switch data collection, and is also used to make comprehensive detection of the collected data, including link traffic detection, load balancing detection, congestion and fault tolerance detection, and optical module fault detection.

[0028] like Figure 1 As shown, the network performance monitoring system provided by this embodiment includes a task management module 10, a data collection module 20 and a monitoring execution module 30. The monitoring execution module is connected to the data collection module and the task management module respectively, and the task management module is also connected to the data collection module.

[0029] The task management module is configured to perform various test tasks on the network of the large intelligent computing center, including but not limited to bandwidth delay test tasks, many-to-one test tasks and robustness test tasks. The data acquisition module is used to collect various parameters of multiple network devices during the execution of various test tasks on the network; the monitoring execution parameters are used to process various parameters to obtain various network performance parameters of the large intelligent computing center.

[0030] The bandwidth and delay test tasks are used to achieve the maximum data transmission rate of the test network link to evaluate the network's carrying capacity. The delay test measures the transmission delay of the data packet from the source to the destination to evaluate the real-time performance of the network.

[0031] The bandwidth and latency testing tasks designed and completed in this application include testing within a single node, data transmission testing from CPU to GPU, data transmission latency testing of GPU within a single node, and data transmission bandwidth and latency testing from one node GPU to another node GPU in the network.

[0032] In addition to the traditional bandwidth and latency tests, this application proposes a queue pair performance test method for the intelligent computing center network to detect QP connection scalability issues. Generally, as the number of QP connections increases, context switching between QPcontexts causes cache loss, so when the number of QP connections exceeds a certain threshold, RDMA performance will drop sharply. The test of this indicator is helpful to provide a reference for the training and push tasks of the intelligent computing center.

[0033] The Many-to-One test task is used to simulate the scenario where multiple data sources send data to a single target, and is used to test the network's congestion control and traffic management capabilities under high load conditions. In the Many-to-One situation, the network will experience momentary congestion, service delays are likely to occur, and even packet loss may occur.

[0034] This application has two types of test tasks: Figure 2 One is to detect the reaction of the network card of the intelligent computing node when encountering a multiple-to-one situation, and the other is to detect the reaction of the switch port when encountering a multiple-to-one situation.

[0035] When designing the many-to-one test task, multiple senders plan the data to be sent in advance. Not only are the data of different senders marked, but the data sent by the same sender each time is also marked. Ultimately, it can ensure that the receiver can detect whether the data sent by each sender is complete and out of order.

[0036] After the multiple-to-one test task is executed, the relevant data summary should be sent to the monitoring execution module so that the monitoring execution module can make comprehensive tests on data integrity, data transmission time, packet loss rate, congestion recovery time, etc.

[0037] The load balancing test task is used to test the uniformity of traffic distribution, that is, to test whether the load balancer can distribute traffic evenly to the backend servers. The traffic on the intelligent computing system network is significantly different from the traffic on the traditional network. The vast majority of flows in traditional networks are ant flows, with a huge number of flows but a small flow of a single flow. However, most of the flows in the intelligent computing system network are "elephant flows", with a limited number of flows but a large flow of a single flow. The very single characteristic of the five-tuple of a data packet (source / destination address, source / port number and protocol) is called the low entropy characteristic, which requires the load balancing test task to be designed and implemented in a targeted manner.

[0038] This application not only implements the test model of ant flows with a large number of flows, but also uses the deep learning framework pytorch and the underlying collective communication library to implement the test model of elephant flows with a large single flow volume, and tests the balance of multiple paths to detect the comprehensive load balancing capabilities of the intelligent computing network.

[0039] The robustness test task is used to send the relevant data of the task to the monitoring execution module for comprehensive analysis after the task is completed. For example, during bandwidth testing and delay testing, the total traffic sent must be sent to the detection unit to facilitate comparative and comprehensive analysis of the data output by the detection unit and the acquisition unit.

[0040] Link failure simulates link failure, including network card, optical module, switch and other situations, to test the fault tolerance of the network. This application designs a scenario where the link is congested, including congestion of a certain network port of the switch and congestion of a certain network port of the intelligent computing node, to test the fault tolerance of the network.

[0041] The data acquisition module is used to collect multiple parameters of multiple network devices. The multiple network devices here include high-speed interfaces, optical modules, switches and network cards, and the multiple parameters include some or all of the high-speed interface parameters, optical module parameters, network card parameters and switch parameters. Figure 3 shown.

[0042] The high-speed interface here refers to the PCIe / NVLink high-speed interface. As high-speed interconnect interfaces, the performance of PCIe and NVLink directly affects the overall performance of the system, especially for high-performance computing (HPC) and artificial intelligence. Therefore, it is very important to monitor and analyze the performance indicators of these interfaces.

[0043] The data acquisition module in this application collects the number information of PCIe / NVLink channels and summarizes the topological structure information of the GPU. The module also collects connection status information, whether the connection between each GPU is established, the bandwidth of the connection, the error information of the connection, the bandwidth utilization of each NVLink channel, etc. The module also collects error information for the PCIe / NVLink high-speed interface, including the type and number of errors in the PCIe / NVLink connection, CRC errors, packet loss, etc., and the timestamp of the error.

[0044] The demand for optical modules in the intelligent computing center continues to upgrade with the growth of computing power to meet the needs of large bandwidth and low latency network connections. As key components for high-speed data transmission, 200G, 400G and 800G optical modules play an important role in the construction and upgrading of the intelligent computing center. Therefore, the parameter acquisition module of this application uniformly collects data such as link length, supplier name, supported distance, version number, serial number, etc. supported by the optical module. And this application proposes the unified collection of bias current threshold, module temperature threshold, laser input and output power threshold, receiver optical power threshold and module voltage threshold, as well as the collection of bias current, module temperature, laser input and output power, receiver optical power and module voltage. And in this application, the module can also collect optical module information on the switch.

[0045] The intelligent computing center network generally configures an intelligent network card for each GPU, which needs to consider the characteristics of high bandwidth and low latency. Therefore, the information collection module is also used to collect core indicators related to these characteristics of the network card for analysis by the detection unit.

[0046] This module collects data such as the link status of the network card of the intelligent computing node, the total number of received packets, the total number of sent packets, the total number of received packet bytes, the total number of sent packet bytes, the number of error packets, the number of lost packets, the number of oversized packets, etc. in real time. This application collects the above indicators in units of 100ms.

[0047] The present application proposes real-time collection of the total number of received packets, the total number of sent packets, the total number of received packet bytes, the total number of sent packet bytes, the number of lost packets, the number of received PAUSE frames, the number of sent PAUSE frames, the received PAUSE duration, and the sent PAUSE duration for each priority level.

[0048] This application also collects QP-related data of the smart network card in real time according to the characteristics of the intelligent computing network, including key indicators such as the number of write requests, the number of read requests, the cache application failure count, the number of disordered requests, the number of repeated requests, the number of ECNs received, the number of CNPs sent, the number of CNPs processed and the number of CNPs ignored.

[0049] The intelligent computing center switch plays an important role in the effective use of computing power. The collection and analysis of switch indicators helps to determine the program that the intelligent computing center switch efficiently supports the use of computing power and meets the needs of high-performance computing and data processing. In this application, the parameter collection module also collects the switch temperature, version number, manufacturer model, serial number, power, CPU occupancy, memory status and the status of each port of the switch.

[0050] This application also collects core indicators related to lossless Ethernet that are unique to the intelligent computing network, including the total chip shared cache usage information, queue headroom cache usage information, interface inbound packet loss count, port-based ECN statistics, received and sent PFC back pressure frames, queue deadlock detection times, and queue deadlock recovery times.

[0051] The monitoring execution module in the present application specifically includes a link detection unit 31, a load balancing detection unit 32, a congestion and fault tolerance detection unit 33 and an optical module fault detection unit 34. Figure 4 shown.

[0052] The link detection unit is used to detect the accuracy of the data and whether the data received by the final receiver is the data sent by the sender, including comparing the size and md5sum of the content. It is also used to detect the validity of the disordered reorganization, and detect whether the receiver puts the data in the correct position in order after receiving the data each time. This is more effective when the data is not completely transmitted.

[0053] It is also used to detect the transmission delay of the network and the total delay from the sender to the receiver, including transmission, processing and queuing delays. The unit is also used to detect the number and percentage of error packets, packet loss, and retransmission packets on each link. This application detects the bandwidth changes with the increase of QP connections based on the characteristics of the intelligent computing center network, detects the threshold of obvious bandwidth changes, and provides a reference for the design of intelligent computing tasks.

[0054] The load balancing detection unit is used to detect the uniformity of traffic distribution based on links. The detection unit summarizes the traffic information of each interface at each time point collected by the collection unit for the load balancing unit to detect.

[0055] This application uses the rate data collected from multiple paths at a certain moment to calculate the average, variance, and standard deviation, and records the upper limit as the average plus the standard deviation, and the lower limit as the average minus the standard deviation. And calculate the number of traffic rates of each path collected at that moment that fall within the upper and lower limits, and calculate the proportion of paths that fall within the upper and lower limits.

[0056] This application uses the above statistical method to detect the traffic conditions under multiple paths based on the start and end time of the ant flow task model and the expected load balancing path designed for the task. This application uses the above statistical method to detect the traffic conditions under multiple paths based on the start and end time of the elephant flow task model and the expected load balancing path designed for the task.

[0057] The congestion and fault tolerance detection unit is used to determine the ability of congestion recovery in the intelligent computing network, which facilitates network management personnel to do end-to-end congestion management and slow down the spread of congestion. Based on the collection of PFC data of network card devices and switch devices, this unit provides congestion detection function based on port priority.

[0058] This application is based on the collection of ECN and CNP data of network card devices and switch devices, provides link-based congestion detection functions, priority-based and link-based congestion recovery time indicators, and provides a basis for the evaluation of intelligent computing networks. This unit cooperates with the robustness test task to simulate various failure scenarios and test the high availability of the intelligent computing network. When abnormalities occur in different devices such as network cards, optical modules, switches, etc., the integrity of data transmission is tested.

[0059] The optical module fault detection unit is used to support aggregate detection of relevant dimensions of optical modules. For example, it can count the number of optical modules from different manufacturers and the number of optical modules with different speeds. And for optical modules with the same speed from the same manufacturer, it can count the number of optical modules with different version numbers.

[0060] This unit detects the bias current of the corresponding optical module, the module temperature, the laser input and output power, the receiver optical power, and the module voltage in the scenario where many-to-one task congestion occurs.

[0061] The means for detecting the fault of the optical module in the present application includes detecting whether the bias current is within a threshold range, judging whether the module temperature is within a threshold range, judging whether the laser input and output powers are within a threshold range, judging whether the receiver optical power is within a threshold range, and judging whether the module voltage is within a threshold range.

[0062] It can be seen from the above technical solution that this embodiment provides a network performance monitoring system for a large intelligent computing center, which includes a task management module, a data acquisition module and a monitoring execution module. The task management module is configured to perform various test tasks on the network of the large intelligent computing center; the data acquisition module is used to collect various parameters of multiple network devices during the execution of various test tasks on the network; the monitoring execution parameters are used to process various parameters to obtain various network performance parameters of the large intelligent computing center. On the basis of obtaining the corresponding network performance parameters, users can optimize the network architecture of the large intelligent computing center and provide strong support for improving computing efficiency.

[0063] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0064] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0065] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0066] The technical solution provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A network performance monitoring system for a large intelligent computing center, characterized in that: The network performance monitoring system includes a task management module, a data collection module and a monitoring execution module, wherein: The task management module is configured to perform a variety of test tasks on the network of the large intelligent computing center; The data acquisition module is used to collect various parameters of multiple network devices during the process of executing the various test tasks on the network; The monitoring execution parameters are used to process the multiple parameters to obtain multiple network performance parameters of the large-scale intelligent computing center.

2. The network performance monitoring system according to claim 1, characterized in that: The multiple test tasks include part or all of a bandwidth delay test task, a many-to-one test task and a robustness test task.

3. The network performance monitoring system according to claim 1, characterized in that: The multiple network devices include part or all of high-speed interfaces, optical modules, switches and network cards, and the multiple parameters include part or all of high-speed interface parameters, optical module parameters, network card parameters and switch parameters.

4. The network performance monitoring system according to claim 3, characterized in that: The high-speed interface parameters include part or all of channel quantity information, connection status information and error information.

5. The network performance monitoring system according to claim 3, characterized in that: The optical module parameters include part or all of bias current, module temperature, laser input and output power, receiver optical power and module voltage.

6. The network performance monitoring system according to claim 3, characterized in that: The network card parameters include part or all of the link status, total number of received packets, total number of sent packets, total number of received packet bytes, total number of sent packet bytes, number of error packets, number of lost packets, and number of oversized packets.

7. The network performance monitoring system according to claim 3, characterized in that: The switch parameters include switch temperature, version number, manufacturer model, serial number, power, CPU occupancy parameters, memory parameters and multiple port information.

8. The network performance monitoring system according to claim 1, characterized in that: The monitoring execution module includes a link detection unit, a load balancing detection unit, a congestion and fault tolerance detection unit and an optical module fault detection unit.

9. The network performance monitoring system according to claim 8, characterized in that: The link detection unit is configured to perform data accuracy detection, out-of-order reassembly effectiveness detection and transmission delay detection, as well as the number and percentage of error packets, packet loss and retransmission packets on each link, and is also used to detect bandwidth changes.

10. The network performance monitoring system according to claim 8, characterized in that: The load balancing detection unit is configured to detect the uniformity of traffic distribution in units of links.

Images