A method for diagnosing internal network delay of host based on loopback test
Through a loopback test-based method, RDMA devices and RNIC are used to diagnose the host's internal network delay, which solves the host's internal network bottleneck problem and achieves rapid fault location and stable operation of network performance.
Patent Information
- Application Number
- CN202510014026.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-06
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2045-01-06
AI Technical Summary
Existing network performance analysis tools cannot effectively diagnose network bottlenecks within the host. Especially in RDMA technology, the complex internal network topology of the host leads to increased data transmission delay and reduced bandwidth, affecting the overall network performance.
A loopback test-based host internal network delay diagnosis method is adopted to directly access the memory area through the RDMA device, monitor the network communication line, and use the RDMA network interface card (RNIC) to measure data transmission and delay to identify the cause of network performance bottlenecks.
It achieves rapid fault location and smooth operation of the host's internal network. It monitors the status of the host's internal components in real time through the hardware monitor, dynamically adjusts the diagnosis frequency, and accurately identifies the hardware or software factors that cause network delays.
Smart Images
Figure CN119788571B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of network bottleneck diagnosis, and in particular relates to a host internal network delay diagnosis method based on loopback test. Background Art
[0002] With the rapid development of the internet and the increasingly sophisticated information infrastructure in modern society, the speed of data generation and transmission worldwide has reached unprecedented heights. Industries such as artificial intelligence, the Internet of Things, and big data, which require ultra-large-scale data transmission and processing, are also rapidly maturing. Demand for data transmission speed and efficiency continues to increase across all industries. In the fields of cloud computing, big data, and the Internet of Things, the immediate transmission and processing of data has become a crucial performance indicator. Large-scale data centers have become the core hubs for data processing and storage. These data centers must handle massive amounts of data traffic, emphasizing real-time and accurate data processing, placing stringent demands on network stability and high performance. Traditional TCP / IP network communication technology has struggled to meet these industry demands.
[0003] Remote Direct Memory Access (RDMA) network technology is a new, high-performance network communication technology driven by these industry demands. Compared to traditional network communication technologies, RDMA offers a significant advantage: during data transmission, RDMA bypasses the host's operating system and CPU, allowing one host to directly access the memory of another. This significantly reduces data transmission latency and significantly improves information transfer efficiency.
[0004] However, with the continuous development of RDMA technology and improvements to related hardware, the host's internal network, long overlooked by the industry, has begun to emerge as a potential performance bottleneck. In particular, the speed of Remote Direct Memory Access Network Interface Cards (RNICs) has now reached hundreds of Gbps, which is quite close to the host's internal bandwidth. The impact of internal network bottlenecks on overall performance has become very significant. RDMA technology is extremely sensitive to network bottlenecks. Increased packet loss, decreased bandwidth, and reduced throughput can lead to a significant decline in overall network performance. Even more problematic is that the internal topology of server hosts is becoming increasingly complex with the increasing number and variety of devices. Internal traffic patterns are complex, and scheduling issues are prone to occur within the host, leading to congestion and network bottlenecks. In a network topology, servers are generally referred to as nodes, and memory nodes can store various types of data, such as numbers, text, images, and so on.
[0005] In high-speed data center network scenarios, bottlenecks are becoming increasingly frequent, including reduced intra-host bandwidth and increased intra-host latency. Existing network performance analysis tools are unable to effectively diagnose intra-host network bottlenecks. Summary of the Invention
[0006] In order to address the defects in the prior art, the present invention provides a host internal network delay diagnosis method based on loopback testing, which is applicable to data center networks. This diagnostic method can quickly locate faults and ensure smooth network operation.
[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:
[0008] The present invention is a method for diagnosing host internal network delay based on loopback testing, which specifically includes the following steps:
[0009] Step 1: Memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area, ensure high data transmission throughput, and prevent the memory from being swapped to the disk during the transmission process. Two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations. Similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations. In preparation for subsequent read / write operations, the memory area obtained after memory registration is mapped to the remote direct memory access network interface card (RNIC). The remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, allowing the remote direct memory access network interface card (RNIC) to directly access and monitor the network communication line and detect abnormal links.
[0010] Step 2: The terminal device sends a work request to the remote direct memory access network interface card (RNIC): the work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA);
[0011] Step 3: At the network driver layer of the remote direct memory access network interface card, convert the work request of step 2 into a work queue element (WQE), and send the work queue element (WQE) to a send queue of the remote direct memory access network interface card (RNIC) cache to wait for execution;
[0012] Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3, and the remote direct memory access network interface card (RNIC) sends a read operation request of the graphics processing unit (GPU) and the server memory node to the terminal device;
[0013] Step 5: After the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU) and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed. The remote direct memory access network interface card (RNIC) places this notification into the remote direct memory access network interface card cache queue.
[0014] Step 6: Poll from the head to the tail of the remote direct memory access network interface card cache queue, poll the completed cache queue to obtain a completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the round-trip delay of the entire operation, measure the round-trip delay between the remote direct memory access network interface card and the graphics processor or server memory node, and compare and analyze the round-trip delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node to determine the internal delay of the server host;
[0015] Step 7. Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify hardware or software factors that cause network delays, and analyze the causes of network congestion, including buffer overflow, excessive queue length, and flow control mechanisms.
[0016] A further improvement of the present invention is that in step 2, the work request is specifically: writing the data read from the graphics processing unit (GPU) read area and the data read from the server memory node read area into the graphics processing unit write area and the memory node write area.
[0017] A further improvement of the present invention is that in step 3, a work queue element includes the source address, target address and data length of the data.
[0018] A further improvement of the present invention is that in step 1, the network communication line is monitored to detect abnormal links, which specifically includes the following steps:
[0019] Step 1.1: Call the initialization function to initialize the status of all network communication lines to "uncertain" and set the anomaly counter of each link to 0. This step ensures that all links are not marked as abnormal or normal in the initial state, providing a benchmark for subsequent analysis.
[0020] Step 1.2: Traverse all normal paths and mark all links as "normal." This is done to clarify the link status on normal paths for comparison and analysis in subsequent steps. Routers that choose the wrong path to forward packets or choose different paths at different times due to routing failures are called abnormal paths.
[0021] Step 1.3: The diagnostic method of the present invention will traverse all abnormal paths and perform the following operations: If there are links in an uncertain state in the abnormal path, these links will be marked as abnormal and the abnormality counter will be increased. This step is used to mark potential abnormal links. Secondly, if there are links in the abnormal path that have been marked as abnormal, the inventive method will further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormality counter will be increased again. This process will further confirm the abnormal state of the link. Finally, if the status of all links in the abnormal path is normal, these links will be marked as "uncertain", indicating that the links marked as "uncertain" have fluctuations and need further observation. Finally, the links marked as abnormal and uncertain are returned. These abnormal and uncertain links may be the main reason for the bandwidth reduction and need further inspection and processing.
[0022] A further improvement of the present invention is that step 4 specifically includes the following steps:
[0023] Step 4.1: After receiving the notification sent by the Doorbell mechanism, the Remote Memory Access Network Interface Card (RNIC) begins to parse the work queue element. The work queue element requires the Remote Memory Access Network Interface Card (RNIC) to read the data in the work queue element from the read area of the GPU and the read area of the server memory stage, then write the data to the write area of the GPU and the write area of the server memory node, and prepare to perform the actual data transfer operation.
[0024] Step 4.2: After the parsing is completed, the remote direct memory access network interface card sends a read operation request to the terminal device through the PCIe bus.
[0025] A further improvement of the present invention is that in step 4.1, parsing the work queue element specifically includes the following steps:
[0026] Step 4.1.1. Receive notifications from the Doorbell mechanism: The operating system notifies the RNIC that a new work queue element (WQE) needs to be processed by writing to the Doorbell register. The RNIC detects the update of the Doorbell register through a hardware interrupt or polling mechanism.
[0027] Step 4.1.2: The Remote Direct Memory Access Network Interface Card (RNIC) responds to the notification sent by the Doorbell mechanism:
[0028] Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed;
[0029] Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE);
[0030] Step 4.1.3, Parsing the Work Queue Element (WQE): Reading the Work Queue Element (WQE) Header Information: The Remote Direct Memory Access Network Interface Card (RNIC) reads the Work Queue Element (WQE) header information from the read area of the server's memory node. The header information includes the operation type, data length, source address, and destination address. The operation type determines whether it is a send or receive operation. The data length indicates the amount of data to be processed. The source / destination address indicates the location of the data in memory.
[0031] Step 4.1.4. Execute the following operations on the Work Queue Element (WQE):
[0032] Read data: Based on the source address in the work queue element (WQE), the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of the server memory node;
[0033] Direct Memory Access (MDA) Read Operation: Uses Direct Memory Access (DMA) to read data from memory without CPU intervention;
[0034] Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data through the physical network. If the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC);
[0035] Step 4.1.5: Write data to the GPU write area and the server memory node write area:
[0036] Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE);
[0037] Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of the server memory node;
[0038] Update memory: Data is written to the specified memory address, completing the data transfer.
[0039] Step 4.1.6: Complete the process and update the status:
[0040] Update the work queue element (WQE) status: the remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed;
[0041] Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.
[0042] A further improvement of the present invention is that in step 6, a test data packet is sent from the terminal device to a remote direct memory access network interface card (RNIC). After receiving the test data packet, the remote direct memory access network interface card (RNIC) immediately returns it to the sender, and the time difference between sending and receiving is recorded as the round trip delay. Specifically, the steps include:
[0043] Step 6.1: The round-trip delay is expressed as:
[0044]
[0045] Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. host is the time it takes from the remote direct memory access network interface card (RNIC) to send the first read request to the time it receives the information returned from the read area. Size is the size of the test message sent in the loopback test. BW host The bandwidth inside the server host;
[0046] Step 6.2: However, in practice, it's difficult to measure the specific value of RNIC's T. Fortunately, this value is usually constant. Therefore, we can send a 1KB message to estimate the internal latency of the host:
[0047]
[0048] Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is:
[0049]
[0050] Calculate the bandwidth BW inside the server host using the above formula host .
[0051] A further improvement of the present invention is that in step 7, the network performance bottleneck is analyzed and the factors causing network delay are identified as follows:
[0052] When the host is in different states, the causes of the host's internal network bottleneck will vary.
[0053] When the host is idle, link failure or configuration errors may be the main cause of the bottleneck: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is a bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, the PCIe bus of the remote direct memory access network interface card (RNIC) may be faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck may be caused by a configuration error.
[0054] When the host is busy, the remote direct memory access network interface card (RNIC) needs to process a large amount of service traffic. In this case, the path bandwidth measured by the diagnostic method may be reduced due to competition from application traffic. However, in such cases, the diagnostic method may still indicate an abnormal path. Applications typically use the remote direct memory access network interface card (RNIC) to communicate with their adjacent terminal devices (such as GPUs under the same port) to achieve optimal performance. If the measured bandwidth of one endpoint is significantly lower than that of other endpoints, the path may be abnormal.
[0055] The beneficial effects of the present invention are:
[0056] (1) The present invention performs a loopback test between the RNIC (RDMA Network Interface Card) and the terminal device to measure the loopback delay between the RNIC and the terminal, thereby calculating the two key indicators of the host's internal bandwidth and delay.
[0057] (2) The present invention uses a hardware monitor to read data from the host to determine the overall status of the host. The hardware monitor can monitor the operating status of key components such as the CPU, memory, and storage in real time, collect and analyze this data, and help the present invention make more accurate judgments. To achieve a balance between performance overhead and diagnostic efficiency, the present invention dynamically adjusts the startup timing and frequency of the present invention based on the operating status of the host.
[0058] (3) The present invention conducts an in-depth analysis of the measured data to identify the causes of internal host bottlenecks. Through data analysis, several causes of internal host network anomalies were discovered, including Remote Direct Memory Access Network Interface Card (RNIC) or memory being occupied by abnormal applications. These findings provide a basis for further optimizing internal host network performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 This is a model diagram of the loopback test of the present invention.
[0060] Figure 2 Flowchart of loopback test performed by RNIC of the present invention.
[0061] Figure 3 When the host is in idle state, the path status diagram between the RNIC and the terminal device shows that blue indicates a normal path status, red indicates an abnormal path status, and orange indicates an uncertain path status.
[0062] Figure 4 This is a state diagram of the path between the RNIC and the terminal device when the host is in a busy state.
[0063] Figure 5 This is a flow chart of network delay diagnosis of the present invention. DETAILED DESCRIPTION
[0064] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.
[0065] like Figure 1-2As shown in FIG5 , the present invention is a method for diagnosing delay in a host internal network based on a loopback test, which is applied to a data center network and is characterized in that the method for diagnosing delay in a host internal network specifically comprises the following steps:
[0066] Step 1: Memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area, ensure high throughput of data transmission, and prevent the memory from being swapped to the disk during the transmission process. Two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations. Similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations. In preparation for subsequent read / write operations, the memory area obtained after memory registration is mapped to the remote direct memory access network interface card (RNIC). The remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, allowing the remote direct memory access network interface card (RNIC) to directly access and monitor the network communication line and detect abnormal links. Specifically, the following steps are included:
[0067] Step 1.1: Call the initialization function to initialize the status of all network communication lines to "uncertain" and set the abnormality counter of each link to 0. This step ensures that all links are not marked as abnormal or normal in the initial state, providing a benchmark for subsequent analysis.
[0068] Step 1.2: Traverse all normal paths and mark all links as "normal." This is done to clarify the link status on normal paths for comparison and analysis in subsequent steps. Routers that choose the wrong path to forward packets or choose different paths at different times due to routing failures are called abnormal paths.
[0069] Step 1.3: The diagnostic method of the present invention will traverse all abnormal paths and perform the following operations: If there are links in an uncertain state in the abnormal path, these links will be marked as abnormal and the abnormality counter will be increased. This step is used to mark potential abnormal links. Secondly, if there are links in the abnormal path that have been marked as abnormal, the inventive method will further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormality counter will be increased again. This process will further confirm the abnormal state of the link. Finally, if the status of all links in the abnormal path is normal, these links will be marked as "uncertain", indicating that the links marked as "uncertain" have fluctuations and need further observation. The inventive method finally returns the links marked as abnormal and uncertain. These abnormal and uncertain links may be the main reason for the bandwidth reduction and need further inspection and processing.
[0070] The process of detecting abnormal links is as follows
[0071]
[0072]
[0073] Step 2: The terminal device sends a work request to the remote direct memory access network interface card (RNIC). The work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA). The work request specifically includes writing data read from the graphics processing unit (GPU) read area and data read from the server memory node read area to the graphics processing unit's write area and the memory node's write area.
[0074] Step 3: At the network driver layer of the remote direct memory access network interface card, the work request from step 2 is converted into a work queue element (WQE) and the work queue element (WQE) is sent to a send queue in the cache of the remote direct memory access network interface card (RNIC) for execution. A work queue element includes the source address, destination address, and data length of the data.
[0075] Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3. The remote direct memory access network interface card (RNIC) sends a read operation request of the graphics processing unit (GPU) and the server memory node to the terminal device. Specifically, the steps include:
[0076] Step 4.1: After receiving the notification sent by the Doorbell mechanism, the Remote Direct Memory Access Network Interface Card (RNIC) begins parsing the work queue element. The work queue element requires the Remote Direct Memory Access Network Interface Card (RNIC) to read the data in the work queue element from the read area of the graphics processing unit (GPU) and the read area of the server memory stage, then write this data to the write area of the graphics processing unit (GPU) and the write area of the server memory node, and prepare to perform the actual data transfer operation. The parsing of the work queue element specifically includes the following steps:
[0077] Step 4.1.1. Receive notifications from the Doorbell mechanism: The operating system notifies the RNIC that a new work queue element (WQE) needs to be processed by writing to the Doorbell register. The RNIC detects the update of the Doorbell register through a hardware interrupt or polling mechanism.
[0078] Step 4.1.2: The Remote Direct Memory Access Network Interface Card (RNIC) responds to the notification sent by the Doorbell mechanism:
[0079] Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed;
[0080] Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE);
[0081] Step 4.1.3, Parsing the Work Queue Element (WQE): Reading the Work Queue Element (WQE) Header Information: The Remote Direct Memory Access Network Interface Card (RNIC) reads the Work Queue Element (WQE) header information from the read area of the server's memory node. The header information includes the operation type, data length, source address, and destination address. The operation type determines whether it is a send or receive operation. The data length indicates the amount of data to be processed. The source / destination address indicates the location of the data in memory.
[0082] Step 4.1.4. Execute the following operations on the Work Queue Element (WQE):
[0083] Read data: Based on the source address in the work queue element (WQE), the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of the server memory node;
[0084] Direct Memory Access (MDA) Read Operation: Uses Direct Memory Access (DMA) to read data from memory without CPU intervention;
[0085] Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data through the physical network. If the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC);
[0086] Step 4.1.5: Write data to the GPU write area and the server memory node write area:
[0087] Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE);
[0088] Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of the server memory node;
[0089] Update memory: Data is written to the specified memory address, completing the data transfer.
[0090] Step 4.1.6: Complete the process and update the status:
[0091] Update the work queue element (WQE) status: the remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed;
[0092] Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.
[0093] Step 4.2: After parsing is complete, the remote direct memory access network interface card sends a read operation request to the terminal device via the PCIe bus. The read operation request includes information such as the source address and size of the required data.
[0094] Step 5: After the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU) and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed. The remote direct memory access network interface card (RNIC) places this notification into the remote direct memory access network interface card cache queue.
[0095] Step 6: Poll from the head to the tail of the remote direct memory access network interface card cache queue, poll the cache queue to obtain a completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the round-trip delay of the entire operation, measure the round-trip delay between the remote direct memory access network interface card and the graphics processor or server memory node, compare and analyze the round-trip delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node, and determine the internal delay of the server host.
[0096] In step 6, a test data packet is sent from the terminal device to the remote direct memory access network interface card (RNIC). After receiving the test data packet, the remote direct memory access network interface card (RNIC) immediately returns it to the sender, and the time difference between sending and receiving is recorded, which is the round trip delay. The specific steps include:
[0097] Step 6.1: The round-trip delay is expressed as:
[0098]
[0099] Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. host is the time it takes from the remote direct memory access network interface card (RNIC) to send the first read request to the time it receives the information returned from the read area. Size is the size of the test message sent in the loopback test. BW host The bandwidth inside the server host;
[0100] Step 6.2: However, in practice, it's difficult to measure the specific value of RNIC's T. Fortunately, this value is usually constant. Therefore, we can send a 1KB message to estimate the internal latency of the host:
[0101]
[0102] Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is:
[0103]
[0104] Calculate the bandwidth BW inside the server host using the above formula host .
[0105] Step 7. Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify hardware or software factors that cause network delays, and analyze the causes of network congestion, including buffer overflow, excessive queue length, and flow control mechanisms.
[0106] When the host is in different states, the causes of the host's internal network bottleneck will vary.
[0107] When the host is idle, link failure or configuration errors may be the main cause of the bottleneck: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is a bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, the PCIe bus of the remote direct memory access network interface card (RNIC) may be faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck may be caused by a configuration error.
[0108] When the host is busy, the remote direct memory access network interface card (RNIC) needs to process a large amount of service traffic. In this case, the path bandwidth measured by the diagnostic method may be reduced due to competition from application traffic. However, in such cases, the diagnostic method may still indicate an abnormal path. Applications typically use the remote direct memory access network interface card (RNIC) to communicate with their adjacent terminal devices (such as GPUs under the same port) to achieve optimal performance. If the measured bandwidth of one endpoint is significantly lower than that of other endpoints, the path may be abnormal.
[0109] Based on the RoCE-v2 protocol, we used NS-3 (Network Simulator 3) network simulation software to build an RDMA network simulation environment. We also implemented an RDMA host intra-network diagnostic tool within this network. This tool uses the host's internal network bandwidth and latency as measurement indicators, and measures these indicators by performing loopback tests between the host's internal RDMA NIC and PCIe endpoints.
[0110] In an embodiment of the present invention:
[0111] (1) Network bottleneck analysis when the host is idle:
[0112] During the test process of the present invention, according to the measurement results, the following Figure 3 The path status diagram shown is used to analyze the network bottleneck status within the host.
[0113] exist Figure 3 In (a), the paths between RNIC1 and the memory and terminal are all normal, while the connections between RNIC2 and other devices are abnormal. This phenomenon indicates that there may be a fault in the connection between RNIC2 and the host. Figure 3 (b) shows another situation: the paths between RNIC1 and RNIC2 and memory 1 and terminal 2 are abnormal, while the path between RNIC1 and memory 2 is normal. In this case, there may be a problem with the connection between memory 1 and terminal 2 and the host.
[0114] Analyzing the above two abnormal situations, we can infer that they are caused by problems in the connection between the hardware device and the host. For example, Figure 3 In (a), RNIC2 may not be correctly connected to the host, so the diagnostic system determines that the path is abnormal. Figure 3 In (b), there are problems with the connection between Memory 1 and Terminal 2 and the host, making both RNICs unable to properly transmit information to them. As the internal topology of the host becomes more complex, such connection failures will occur more frequently.
[0115] In addition, there is a potential possibility that Figure 3 In (b), the communication between memory 1 and the two RNICs is not smooth, which means that there may be a problem with the memory itself.
[0116] Figure 3In (c), the paths between RNIC 1 and the memory and terminal 2 are abnormal, while the paths to terminal 1 are normal. However, RNIC 2's connections to the memory are normal, while its connections to both terminal devices are uncertain. This may be due to a CPU root port failure. In this case, the link between the RNIC and the terminal device is normal through the failed root port. Traffic transmitted through the failed root port will experience bandwidth degradation, while the RNICs connected to other root ports will be unaware of the abnormality.
[0117] Figure 3 (d) illustrates a host internal network bottleneck caused by a configuration error. In this embodiment of the present invention, RNIC1's send window is set lower than normal, causing anomalies in the link between RNIC1 and the two end devices. A smaller send window means more frequent ACKs are required for the same amount of data. Each ACK increases the RTT delay, thereby increasing overall latency. The receiver's buffer fills quickly due to the small window, and the sender uses a congestion control algorithm to reduce the packet sending rate, reducing the line throughput.
[0118] (2) Network bottleneck analysis when the host is busy:
[0119] Figure 4 In (a), RNIC 1's buffer is heavily occupied, triggering the congestion control mechanism and sending a large number of send pause frames, causing the send rate to drop rapidly. In contrast, the path between RNIC 2 and Memory 2 remains normal. Comparing these two paths reveals a large amount of abnormal traffic between RNIC 1 and Memory 2, leading to the congestion problem. This abnormality may be caused by an application failure and requires further analysis.
[0120] Figure 4 (b) shows another abnormal situation: Memory 1 is heavily occupied by applications, causing a rapid drop in memory bandwidth, which in turn triggers a bottleneck in the path from the two RNICs to Memory 1. Due to the heavy use of Memory 1's resources, the two RNICs experience severe bandwidth shortages when communicating with Memory 1. This significantly reduces data transmission efficiency and impacts overall system performance.
[0121] The present invention realizes network bottleneck diagnosis of a host in different states, can quickly locate faults, and ensures stable operation of the network.
[0122] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.
Claims
1. A method for diagnosing internal network delay in a host based on loopback testing, which is applied to a data center network and is characterized by: The host internal network delay diagnosis method specifically includes the following steps: Step 1: Memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area. Two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations. Similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations. The memory areas obtained after memory registration are mapped to the remote direct memory access network interface card (RNIC). The remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, allowing the remote direct memory access network interface card (RNIC) to directly access and monitor the network communication line and detect abnormal links. Step 2: The terminal device sends a work request to the remote direct memory access network interface card (RNIC): the work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA); Step 3: At the network driver layer of the remote direct memory access network interface card, convert the work request of step 2 into a work queue element (WQE), and send the work queue element (WQE) to a send queue of the remote direct memory access network interface card (RNIC) cache to wait for execution; Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3, and the remote direct memory access network interface card (RNIC) sends a read operation request of the graphics processing unit (GPU) and the server memory node to the terminal device; Step 5: After the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU) and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed. The remote direct memory access network interface card (RNIC) places this notification into the remote direct memory access network interface card cache queue. Step 6: Poll from the head to the tail of the remote direct memory access network interface card cache queue, poll the completed cache queue to obtain a completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the round-trip delay of the entire operation, measure the round-trip delay between the remote direct memory access network interface card and the graphics processor or server memory node, and compare and analyze the round-trip delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node to determine the internal delay of the server host; Step 7: Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify factors that cause network delays, and analyze the causes of network congestion.
2. The method for diagnosing internal network delay of a host based on loopback testing according to claim 1, wherein: The step 4 specifically includes the following steps: Step 4.1: After receiving the notification sent by the Doorbell mechanism, the Remote Memory Access Network Interface Card (RNIC) begins to parse the work queue element. The work queue element requires the Remote Memory Access Network Interface Card (RNIC) to read the data in the work queue element from the read area of the GPU and the read area of the server memory stage, then write the data to the write area of the GPU and the write area of the server memory node, and prepare to perform the actual data transfer operation. Step 4.2: After the parsing is completed, the remote direct memory access network interface card sends a read operation request to the terminal device through the PCIe bus.
3. The method for diagnosing internal network delay of a host based on loopback testing according to claim 2, wherein: In step 4.1, parsing the work queue element specifically includes the following steps: Step 4.1.
1. Receive notifications from the Doorbell mechanism: The operating system notifies the RNIC that a new work queue element (WQE) needs to be processed by writing to the Doorbell register. The RNIC detects the update of the Doorbell register through a hardware interrupt or polling mechanism. Step 4.1.2: The Remote Direct Memory Access Network Interface Card (RNIC) responds to the notification sent by the Doorbell mechanism: Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed; Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE); Step 4.1.
3. Parse the Work Queue Element (WQE): Read the Work Queue Element (WQE) header information: The Remote Direct Memory Access Network Interface Card (RNIC) reads the Work Queue Element (WQE) header information from the read area of the server memory node, including the operation type, data length, source address, and destination address; Step 4.1.
4. Execute the following operations on the Work Queue Element (WQE): Read data: Based on the source address in the work queue element (WQE), the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of the server memory node; Direct Memory Access (MDA) Read Operation: Uses Direct Memory Access (DMA) to read data from memory without CPU intervention; Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data out; if the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC); Step 4.1.5: Write data to the GPU write area and the server memory node write area: Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE); Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of the server memory node; Update memory: data is written to the specified memory address, completing the data transmission; Step 4.1.6: Complete the process and update the status: Update the work queue element (WQE) status: the remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed; Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.
4. The method for diagnosing internal network delay of a host based on loopback testing according to claim 1, wherein: In step 6, a test data packet is sent from the terminal device to the remote direct memory access network interface card (RNIC). After receiving the test data packet, the remote direct memory access network interface card (RNIC) immediately returns it to the sender, and the time difference between sending and receiving is recorded, which is the round trip delay. The specific steps include: Step 6.1: The round-trip delay is expressed as: Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. host is the time it takes from the remote direct memory access network interface card (RNIC) to send the first read request to the time it receives the information returned from the read area. Size is the size of the test message sent in the loopback test. BW host The bandwidth inside the server host; Step 6.2: Send a 1KB message to evaluate the internal latency of the host: Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is: Calculate the bandwidth BW inside the server host using the above formula host .
5. The method for diagnosing internal network delay of a host based on loopback testing according to claim 1, wherein: In step 1, the network communication lines are monitored to detect abnormal links, which specifically includes the following steps: Step 1.1: Call the initialization function to initialize the network communication line state to "uncertain" and set the exception counter of each link to 0; Step 1.2: Traverse all normal paths and mark all links as "normal". Routers that choose the wrong path to forward packets or choose different paths at different times due to routing failures are called abnormal paths. Step 1.3: Traverse all abnormal paths and perform the following operations: If there are links in an uncertain state in the abnormal path, these links will be marked as abnormal and the abnormal counter will be increased. If there are links in the abnormal path that have already been marked as abnormal, further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormal counter will be increased again to further confirm the abnormal state of the link. Finally, if the status of all links in the abnormal path is normal, these links will be marked as "uncertain", indicating that there is fluctuation in the links marked as "uncertain", and finally the links marked as abnormal and uncertain are returned.
6. The method for diagnosing internal network delay of a host based on loopback testing according to claim 1, characterized in that: In step 7, analyze the network performance bottleneck and identify the factors that cause network delays: When the host is idle, a link failure or configuration error is the cause of the bottleneck: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is the bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, the PCIe bus of the remote direct memory access network interface card (RNIC) is faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck is caused by a configuration error. When the host is busy, the remote direct memory access network interface card (RNIC) needs to handle a large amount of service traffic. At this time, the measured path bandwidth will be reduced due to competition from application traffic. In such cases, the diagnostic method may still indicate an abnormal path. Applications typically use remote direct memory access network interface cards (RNIC) to communicate with their adjacent terminal devices to achieve optimal performance. If the measured bandwidth to one endpoint is significantly lower than that of other endpoints, the path is abnormal.