Host internal network delay diagnosis method based on loopback test

Abstract

The invention belongs to the field of network bottleneck diagnosis, and discloses a host internal network delay diagnosis method based on a loopback test, which is characterized in that an RDMA network simulation environment is established by using NS-3 (Network Simulation 3) network simulation software based on a RoCE-v2 protocol, and an RDMA host internal network diagnosis tool is realized in the network. According to the tool, network bandwidth and delay in a host are used as measurement indexes, and the indexes are measured by carrying out loopback testing between RDMA NIC and PCIe endpoint in the host. After the measurement data is collected, the internal network condition of the host can be analyzed according to the data, and the internal bottleneck of the host is diagnosed according to the internal network condition.

Description

Technical Field

[0001] The invention belongs to the field of network bottleneck diagnosis, and in particular relates to a host internal network delay diagnosis method based on loopback test. Background Art

[0002] With the rapid development of the Internet and the increasingly improved information infrastructure in modern society, the speed of data generation and transmission has reached an unprecedented level worldwide. Industries such as artificial intelligence, the Internet of Things, and big data that require ultra-large-scale data transmission and processing are also maturing rapidly. The demand for data transmission speed and efficiency in all walks of life is constantly increasing. In the fields of cloud computing, big data, and the Internet of Things, the instant transmission and processing of data have become critical performance indicators. Large-scale data centers have become the core hubs for data processing and storage. These data centers need to handle ultra-large-scale data traffic, emphasizing the real-time and accuracy of data processing, and therefore have strict requirements on network stability and high performance. However, traditional TCP / IP network communication technology has been unable to meet these industry needs.

[0003] Remote Direct Memory Access (RDMA) network technology is a new high-performance network communication technology that was born out of such industry demand. Compared with traditional network communication technologies, RDMA has a significant advantage, that is, during the data transmission process, RDMA can bypass the host's operating system and CPU, allowing one host to directly access the memory of another host remotely. This greatly reduces the delay in the data transmission process and greatly improves the efficiency of information transmission.

[0004] However, with the continuous development of RDMA technology itself and the improvement of related hardware, the internal network of the host, which was long ignored by the industry, has begun to become a potential performance bottleneck. In particular, the speed of the Remote Direct Memory Access Network Interface Card (RNIC) has been able to reach hundreds of Gbps, which is quite close to the bandwidth inside the host. The impact of the internal network bottleneck of the host on the overall performance has become very significant. RDMA technology is extremely sensitive to network bottlenecks. Increased packet loss rate, decreased bandwidth and decreased throughput will lead to a significant decline in the overall network performance. What is more difficult is that the topology of the server host has become more and more complex with the increase in the number and type of devices. The internal traffic pattern is complex, and scheduling problems are prone to occur inside the host, leading to congestion and network bottlenecks. In the network topology, we generally call the server a node, and the memory node can store various types of data, such as numbers, text, images, etc.

[0005] In high-speed data center network scenarios, bottlenecks are becoming more frequent, including reduced bandwidth within the host and increased latency within the host. Existing network performance analysis tools are unable to effectively diagnose network bottlenecks within the host. Summary of the invention

[0006] In order to solve the defects existing in the prior art, the present invention provides a host internal network delay diagnosis method based on loopback test, which is applicable to data center networks. The diagnosis method can quickly locate faults and ensure smooth operation of the network.

[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0008] The present invention is a method for diagnosing delay in a host internal network based on a loopback test, and the method specifically comprises the following steps:

[0009] Step 1: Memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area, ensure high throughput of data transmission, and prevent the memory from being swapped to the disk during the transmission process. Two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations. Similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations. In preparation for the next read / write operation, the memory area obtained after the memory registration is mapped to the remote direct memory access network interface card (RNIC). The remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, so that the remote direct memory access network interface card (RNIC) can directly access and monitor the network communication line and abnormal link.

[0010] Step 2, the terminal device sends a work request to the remote direct memory access network interface card (RNIC): the work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA);

[0011] Step 3: at the network driver layer of the remote direct memory access network interface card, convert the work request of step 2 into a work queue element (WQE), and send the work queue element (WQE) to the sending queue of the remote direct memory access network interface card (RNIC) cache to wait for execution;

[0012] Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3, and the remote direct memory access network interface card (RNIC) sends a read operation request of a graphics processing unit (GPU) and a read operation request of a server memory node to the terminal device;

[0013] Step 5, after the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU), and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed, and the remote direct memory access network interface card (RNIC) puts this notification into the remote direct memory access network interface card cache queue;

[0014] Step 6: Perform a polling operation from the head to the tail of the remote direct memory access network interface card cache queue, poll the cached queue to obtain a completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the loopback delay of the entire operation, measure the loopback delay between the remote direct memory access network interface card and the graphics processor or the server memory node, compare and analyze the loopback delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node, and determine the internal delay of the server host;

[0015] Step 7. Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify hardware or software factors that cause network delays, and analyze the causes of network congestion, including buffer overflow, excessive queue length, and flow control mechanisms.

[0016] A further improvement of the present invention is that in step 2, the work request is specifically: writing the data read from the graphics processing unit (GPU) read area and the data read from the server memory node read area into the graphics processing unit write area and the memory node write area.

[0017] A further improvement of the present invention is that in step 3, a work queue element includes a source address, a target address, and a data length.

[0018] A further improvement of the present invention is that in step 1, the network communication line is monitored to monitor abnormal links, which specifically includes the following steps:

[0019] Step 1.1, call the initialization function to initialize the status of all network communication lines to "uncertain" and set the abnormality counter of each link to 0. This step ensures that all links are not marked as abnormal or normal in the initial state, providing a benchmark for subsequent analysis.

[0020] Step 1.2: Traverse all normal paths and mark all links as "normal". The purpose of this is to clarify the link status on the normal path so that it can be compared and analyzed in subsequent steps. The router that selects the wrong path to forward data packets or selects different paths at different times due to routing failure is called an abnormal path.

[0021] Step 1.3, the diagnostic method of the present invention will traverse all abnormal paths and perform the following operations: If there are links in an uncertain state in the abnormal path, these links will be marked as abnormal, and the abnormal counter will be increased. This step is used to mark potential abnormal links. Secondly, if there are links in the abnormal path that have been marked as abnormal, the inventive method will further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormal counter will be increased again. This process will further confirm the abnormal state of the link. Finally, if all link states in the abnormal path are normal, these links will be marked as "uncertain", indicating that the links marked as "uncertain" have fluctuations and need further observation. Finally, the links marked as abnormal and uncertain are returned. These abnormal and uncertain links may be the main reason for the bandwidth reduction and need further inspection and processing.

[0022] A further improvement of the present invention is that the step 4 specifically comprises the following steps:

[0023] Step 4.1, after receiving the notification sent by the Doorbell mechanism, the remote direct memory access network interface card (RNIC) starts to parse the work queue element. The work queue element requires the remote direct memory access network interface card (RNIC) to read the data in the work queue element from the read area of ​​the graphics processing unit (GPU) and the read area of ​​the server memory stage, and then write the data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node, and prepare to perform the actual data transfer operation;

[0024] Step 4.2: After the parsing is completed, the remote direct memory access network interface card sends a read operation request to the terminal device through the PCIe bus.

[0025] A further improvement of the present invention is that in step 4.1, parsing the work queue element specifically includes the following steps:

[0026] Step 4.1.1. Accept the notification sent by the Doorbell mechanism: The operating system notifies the remote direct memory access network interface card (RNIC) that there is a new work queue element (WQE) to be processed by writing to the Doorbell register. The remote direct memory access network interface card (RNIC) detects the update of the Doorbell register through a hardware interrupt or polling mechanism.

[0027] Step 4.1.2, the remote direct memory access network interface card (RNIC) responds to the notification sent by the Doorbell mechanism:

[0028] Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed;

[0029] Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE);

[0030] Step 4.1.3, Parse the Work Queue Element (WQE): Read the Work Queue Element (WQE) header information: The remote direct memory access network interface card (RNIC) reads the Work Queue Element (WQE) header information from the read area of ​​the server memory node, including the operation type, data length, source address, and destination address; Operation type: determines whether it is a send or receive operation. Data length: indicates the amount of data to be processed. Source / destination address: indicates the location of the data in the memory.

[0031] Step 4.1.4, perform the following operations on the work queue element (WQE):

[0032] Read data: Based on the source address in the work queue element WQE, the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of ​​the server memory node;

[0033] Direct memory access (MDA) read operation: Use direct memory access (DMA) to read data from memory without CPU intervention;

[0034] Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data through the physical network. If the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC);

[0035] Step 4.1.5: Write data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node:

[0036] Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE);

[0037] Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of ​​the server memory node;

[0038] Update memory: Data is written to the specified memory address, completing the data transfer.

[0039] Step 4.1.6, complete the processing and update the status:

[0040] Update work queue element (WQE) status: The remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed;

[0041] Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.

[0042] A further improvement of the present invention is that in step 6, a test data packet is sent from the terminal device to the remote direct memory access network interface card (RNIC), and the remote direct memory access network interface card (RNIC) immediately returns the test data packet to the sender after receiving it, and records the time difference between sending and receiving, which is the round trip delay, specifically including the following steps:

[0043] Step 6.1, the round-trip delay is expressed as:

[0044]

[0045] Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. hos is the time from when the remote direct memory access network interface card (RNIC) sends the first read request to when the remote direct memory access network interface card receives the information returned from the read area. Size refers to the size of the test message sent in the loopback test. BW host The bandwidth inside the server host;

[0046] Step 6.2: However, in actual applications, it is difficult to measure the specific value of T of RNIC. Fortunately, this value of RNIC usually remains unchanged. Therefore, we can send a message of 1KB to evaluate the internal delay of the host:

[0047]

[0048] Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is:

[0049]

[0050] Calculate the bandwidth BW inside the server host using the above formula host .

[0051] A further improvement of the present invention is that in step 7, the network performance bottleneck is analyzed and the factors causing network delay are identified as follows:

[0052] When the host is in different states, the causes of the host's internal network bottleneck may vary.

[0053] When the host is in an idle state, link failure or configuration error may be the main bottleneck cause: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is a bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, it may be that the PCIe bus of the remote direct memory access network interface card (RNIC) is faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck cause may be caused by a configuration error.

[0054] When the host is busy, the remote direct memory access network interface card (RNIC) needs to handle a large amount of service traffic. At this time, the path bandwidth measured by the diagnostic method may be reduced due to the competition of application traffic. However, in such cases, the diagnostic method may still indicate an abnormal path. Applications usually use the remote direct memory access network interface card (RNIC) to communicate with their adjacent terminal devices (such as GPUs under the same port) to achieve optimal performance. If the measured bandwidth to one endpoint is significantly lower than that of other endpoints, the path may be abnormal.

[0055] The beneficial effects of the present invention are:

[0056] (1) The present invention performs a loopback test between RNIC (RDMA Network Interface Card) and a terminal device to measure the loopback delay between RNIC and the terminal, thereby calculating two key indicators, bandwidth and delay, within the host.

[0057] (2) The present invention reads the data inside the host through the hardware monitor to determine the overall status of the host. The hardware monitor can monitor the operating status of key components such as the CPU, memory, and storage in real time, collect and analyze these data, and help the present invention make more accurate judgments. In order to achieve a balance between performance overhead and diagnostic efficiency, the present invention will dynamically adjust the startup timing and frequency of the present invention according to the operating status of the host.

[0058] (3) The present invention conducts an in-depth analysis of the measured data and finds out the cause of the bottleneck inside the host. Through data analysis, several causes of abnormal network inside the host are found. For example, the remote direct memory access network interface card (RNIC) or the memory is occupied by abnormal applications. These findings provide a basis for further optimizing the performance of the internal network of the host. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a model diagram of the loopback test of the present invention.

[0060] Figure 2 The flowchart of the loopback test performed by the RNIC of the present invention.

[0061] Figure 3 When the host is in idle state, the path status diagram between the RNIC and the terminal device. Blue indicates a normal path status, red indicates an abnormal path status, and orange indicates an uncertain path status.

[0062] Figure 4 This is a path status diagram between the RNIC and the terminal device when the host is in a busy state.

[0063] Figure 5 This is a flow chart of network delay diagnosis of the present invention. DETAILED DESCRIPTION

[0064] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0065] like Figure 1-2As shown in FIG. 5 , the present invention is a method for diagnosing delay in a host internal network based on a loopback test, and the method is applied to a data center network, and is characterized in that the method for diagnosing delay in a host internal network specifically comprises the following steps:

[0066] Step 1: Memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area, ensure high throughput of data transmission, and prevent the memory from being swapped to the disk during the transmission process. Two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations. Similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations. In preparation for the next read / write operation, the memory area obtained after the memory registration is mapped to the remote direct memory access network interface card (RNIC). The remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, so that the remote direct memory access network interface card (RNIC) can directly access the network communication line and monitor the abnormal link. Specifically, the following steps are included:

[0067] Step 1.1, call the initialization function to initialize the status of all network communication lines to "uncertain", and set the abnormal counter of each link to 0. This step ensures that all links are not marked as abnormal or normal in the initial state, providing a benchmark for subsequent analysis. Next,

[0068] Step 1.2: Traverse all normal paths and mark all links as "normal". The purpose of this is to clarify the link status on the normal path so that it can be compared and analyzed in subsequent steps. The router that selects the wrong path to forward data packets or selects different paths at different times due to routing failure is called an abnormal path.

[0069] Step 1.3, the diagnostic method of the present invention will traverse all abnormal paths and perform the following operations: If there are links in an uncertain state in the abnormal path, these links will be marked as abnormal, and the abnormal counter will be increased. This step is used to mark potential abnormal links. Secondly, if there are links in the abnormal path that have been marked as abnormal, the inventive method will further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormal counter will be increased again. This process will further confirm the abnormal state of the link. Finally, if all link states in the abnormal path are normal, these links will be marked as "uncertain", indicating that the links marked as "uncertain" have fluctuations and need further observation. The inventive method finally returns the links marked as abnormal and uncertain. These abnormal and uncertain links may be the main reason for the bandwidth reduction and need further inspection and processing.

[0070] The process of detecting abnormal links is as follows

[0071]

[0072]

[0073] Step 2: The terminal device sends a work request to the remote direct memory access network interface card (RNIC): the work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA), wherein the work request specifically includes: writing the data read from the read area of ​​the graphics processing unit (GPU) and the data read from the read area of ​​the server memory node into the write area of ​​the graphics processing unit and the write area of ​​the memory node.

[0074] Step 3: At the network driver layer of the remote direct memory access network interface card, the work request of step 2 is converted into a work queue element (WQE), and the work queue element (WQE) is sent to the send queue of the remote direct memory access network interface card (RNIC) cache to wait for execution. A work queue element includes a source address, a target address, and a data length.

[0075] Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3, and the remote direct memory access network interface card (RNIC) sends a read operation request of a graphics processing unit (GPU) and a read operation request of a server memory node to the terminal device, specifically including the following steps:

[0076] Step 4.1: After receiving the notification sent by the Doorbell mechanism, the remote direct memory access network interface card (RNIC) starts parsing the work queue element. The work queue element requires the remote direct memory access network interface card (RNIC) to read the data in the work queue element from the read area of ​​the graphics processing unit (GPU) and the read area of ​​the server memory stage, and then write the data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node, and prepare to perform the actual data transfer operation. The parsing of the work queue element specifically includes the following steps:

[0077] Step 4.1.1. Accept the notification sent by the Doorbell mechanism: The operating system notifies the remote direct memory access network interface card (RNIC) that there is a new work queue element (WQE) to be processed by writing to the Doorbell register. The remote direct memory access network interface card (RNIC) detects the update of the Doorbell register through a hardware interrupt or polling mechanism.

[0078] Step 4.1.2, the remote direct memory access network interface card (RNIC) responds to the notification sent by the Doorbell mechanism:

[0079] Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed;

[0080] Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE);

[0081] Step 4.1.3, Parse the Work Queue Element (WQE): Read the Work Queue Element (WQE) header information: The remote direct memory access network interface card (RNIC) reads the Work Queue Element (WQE) header information from the read area of ​​the server memory node, including the operation type, data length, source address, and destination address; Operation type: determines whether it is a send or receive operation. Data length: indicates the amount of data to be processed. Source / destination address: indicates the location of the data in the memory.

[0082] Step 4.1.4, perform the following operations on the work queue element (WQE):

[0083] Read data: Based on the source address in the work queue element WQE, the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of ​​the server memory node;

[0084] Direct memory access (MDA) read operation: Use direct memory access (DMA) to read data from memory without CPU intervention;

[0085] Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data through the physical network. If the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC);

[0086] Step 4.1.5: Write data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node:

[0087] Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE);

[0088] Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of ​​the server memory node;

[0089] Update memory: Data is written to the specified memory address, completing the data transfer.

[0090] Step 4.1.6, complete the processing and update the status:

[0091] Update work queue element (WQE) status: The remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed;

[0092] Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.

[0093] Step 4.2: After parsing is completed, the remote direct memory access network interface card sends a read operation request to the terminal device through the PCIe bus. The read operation request includes information such as the source address and size of the required data.

[0094] Step 5, after the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU), and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed, and the remote direct memory access network interface card (RNIC) puts this notification into the remote direct memory access network interface card cache queue;

[0095] Step 6: Perform polling from the head to the tail of the remote direct memory access network interface card cache queue, poll the cache queue to obtain the completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the loopback delay of the entire operation, measure the loopback delay between the remote direct memory access network interface card and the graphics processor or server memory node, compare and analyze the loopback delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node, and determine the internal delay of the server host.

[0096] In step 6, a test data packet is sent from the terminal device to the remote direct memory access network interface card (RNIC). After receiving the test data packet, the remote direct memory access network interface card (RNIC) immediately returns it to the sender, and the time difference between sending and receiving is recorded, which is the round-trip delay. Specifically, the following steps are included:

[0097] Step 6.1, the round-trip delay is expressed as:

[0098]

[0099] Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. hos is the time from when the remote direct memory access network interface card (RNIC) sends the first read request to when the remote direct memory access network interface card receives the information returned from the read area. Size refers to the size of the test message sent in the loopback test. BW host The bandwidth inside the server host;

[0100] Step 6.2: However, in actual applications, it is difficult to measure the specific value of T of RNIC. Fortunately, this value of RNIC usually remains unchanged. Therefore, we can send a message of 1KB to evaluate the internal delay of the host:

[0101]

[0102] Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is:

[0103]

[0104] Calculate the bandwidth BW inside the server host using the above formula host .

[0105] Step 7. Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify hardware or software factors that cause network delays, and analyze the causes of network congestion, including buffer overflow, excessive queue length, and flow control mechanisms.

[0106] When the host is in different states, the causes of the host's internal network bottleneck may vary.

[0107] When the host is in an idle state, link failure or configuration error may be the main bottleneck cause: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is a bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, it may be that the PCIe bus of the remote direct memory access network interface card (RNIC) is faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck cause may be caused by a configuration error.

[0108] When the host is busy, the remote direct memory access network interface card (RNIC) needs to handle a large amount of service traffic. At this time, the path bandwidth measured by the diagnostic method may be reduced due to the competition of application traffic. However, in such cases, the diagnostic method may still indicate an abnormal path. Applications usually use the remote direct memory access network interface card (RNIC) to communicate with their adjacent terminal devices (such as GPUs under the same port) to achieve optimal performance. If the measured bandwidth to one endpoint is significantly lower than that of other endpoints, the path may be abnormal.

[0109] Based on the RoCE-v2 protocol, an RDMA network simulation environment is built using NS-3 (Network Simulator 3) network simulation software, and an RDMA host network diagnostic tool is implemented in the network. The tool uses the network bandwidth and latency inside the host as measurement indicators, and measures the indicators by performing a loopback test between the RDMA NIC and PCIe endpoint inside the host.

[0110] In an embodiment of the present invention:

[0111] (1) Network bottleneck analysis when the host is idle:

[0112] In the test process of the present invention, according to the measurement results, the following Figure 3 The path status diagram shown is used to analyze the network bottleneck status inside the host.

[0113] exist Figure 3 In (a), the paths between RNIC1 and the memory and terminal are all normal, while the connections between RNIC2 and other devices are all abnormal. This phenomenon indicates that there may be a fault in the connection between RNIC2 and the host. Figure 3 (b) shows another situation: the paths between RNIC1 and RNIC2 and memory 1 and terminal 2 are abnormal, while the paths between RNIC1 and memory 2 are normal. In this case, there may be a problem with the connection between memory 1 and terminal 2 and the host.

[0114] Analyzing the above two abnormal situations, we can infer that they are caused by problems in the connection between the hardware device and the host. Figure 3 In (a), RNIC2 may not be correctly connected to the host, so the diagnostic system determines that the path is abnormal. Figure 3 In (b), there is a problem with the connection between memory 1 and terminal 2 and the host, so that both RNICs cannot normally transmit information to them. As the internal topology of the host becomes more complex, such connection failures will occur frequently.

[0115] In addition, there is another potential possibility that Figure 3 In (b), the communication between memory 1 and the two RNICs is not smooth, which means that there may be a problem with the memory itself.

[0116] Figure 3In (c), the path between RNIC 1 and the memory and terminal 2 is abnormal, and the path between RNIC 1 and terminal 2 is normal, while the connection between RNIC 2 and the memory is normal, and the connection between RNIC 2 and the two terminal devices is in an uncertain state. This may be caused by the failure of the CPU root port. In this failure case, the link between the RNIC and the terminal device is normal under the failed root port. The traffic transmitted through the failed root port will be affected by the bandwidth reduction, and the RNICs under other root ports cannot perceive the abnormality.

[0117] Figure 3 (d) shows the internal network bottleneck of the host caused by configuration errors. In the embodiment of the present invention, the sending window of RNIC1 is set to a lower value than normal, resulting in an abnormal link between RNIC1 and the two terminal devices. A smaller sending window means that the same amount of data requires more frequent ACKs. Each ACK increases the RTT delay, thereby increasing the overall delay. The receiver's buffer fills up quickly due to the small window. The sender reduces the sending speed of data packets through the congestion control algorithm, and the line throughput decreases.

[0118] (2) Network bottleneck analysis when the host is busy:

[0119] Figure 4 In (a), the buffer of RNIC 1 is heavily occupied, triggering the congestion control mechanism, sending a large number of send pause frames, causing the sending rate to drop rapidly. In contrast, the path between RNIC 2 and Memory 2 remains normal. Comparing these two paths, it can be determined that there is a large amount of abnormal traffic between RNIC 1 and Memory 2, causing the congestion problem. This abnormality may be caused by application failure and requires further analysis.

[0120] Figure 4 (b) shows another abnormal situation: Memory 1 is occupied by a large number of applications, causing the memory bandwidth to drop rapidly, thus causing a bottleneck on the path from the two RNICs to Memory 1. Since the resources of Memory 1 are heavily occupied, the two RNICs have a serious bandwidth shortage problem when communicating with Memory 1. This leads to a significant decrease in the efficiency of data transmission and affects the overall system performance.

[0121] The present invention realizes network bottleneck diagnosis of the host under different states, can quickly locate the fault, and ensure the smooth operation of the network.

[0122] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for diagnosing delay in a host internal network based on a loopback test, the method being applied to a data center network, and characterized in that: The host internal network delay diagnosis method specifically comprises the following steps: Step 1, memory registration is performed through the ibv_reg_mr interface on the remote direct memory access (RMDA) to allow the RDMA device to directly access and operate the memory area, two memory areas are registered on the graphics processing unit (GPU): a read area for read operations and a write area for write operations, and similarly, two memory areas are registered on the server memory node: a read area for read operations and a write area for write operations, and the memory area obtained after the memory registration is mapped to the remote direct memory access network interface card (RNIC), and the remote direct memory access network interface card (RNIC) caches the memory area information of the graphics processing unit (GPU) and the memory area information of the server memory node, so that the remote direct memory access network interface card (RNIC) directly accesses, monitors the network communication line, and monitors the abnormal link; Step 2, the terminal device sends a work request to the remote direct memory access network interface card (RNIC): the work request is submitted to the remote direct memory access network interface card (RNIC) through the ibv_post_send interface on the remote direct memory access (RMDA); Step 3: at the network driver layer of the remote direct memory access network interface card, convert the work request of step 2 into a work queue element (WQE), and send the work queue element (WQE) to the sending queue of the remote direct memory access network interface card (RNIC) cache to wait for execution; Step 4: Notify the remote direct memory access network interface card (RNIC) through the Doorbell mechanism to process the work queue element (WQE) converted in step 3, and the remote direct memory access network interface card (RNIC) sends a read operation request of a graphics processing unit (GPU) and a read operation request of a server memory node to the terminal device; Step 5, after the terminal device receives the read operation request sent in step 4, it obtains data from the read operation of the graphics processing unit (GPU), and transmits the obtained data back to the remote direct memory access network interface card (RNIC) through the PCIe bus. After the remote direct memory access network interface card (RNIC) receives the data returned from the terminal device, it inputs the data into the write operation of the server memory node according to the description of the work queue element. When the data is successfully written into the write operation, the remote direct memory access network interface card (RNIC) generates a completion notification to inform that the transmission operation is completed, and the remote direct memory access network interface card (RNIC) puts this notification into the remote direct memory access network interface card cache queue; Step 6: Perform a polling operation from the head to the tail of the remote direct memory access network interface card cache queue, poll the cached queue to obtain a completion notification, compare the time of initiating the work request ibv_post_send and the time of receiving the completion notification, calculate the loopback delay of the entire operation, measure the loopback delay between the remote direct memory access network interface card and the graphics processor or the server memory node, compare and analyze the loopback delay data between the remote direct memory access network interface card, the graphics processor, and the server memory node, and determine the internal delay of the server host; Step 7: Compare the transmission delays of different test data packets, analyze network performance bottlenecks, identify factors that cause network delays, and analyze the causes of network congestion.

2. A method for diagnosing host internal network delay based on loopback test according to claim 1, characterized in that: The step 4 specifically includes the following steps: Step 4.1, after receiving the notification sent by the Doorbell mechanism, the remote direct memory access network interface card (RNIC) starts to parse the work queue element. The work queue element requires the remote direct memory access network interface card (RNIC) to read the data in the work queue element from the read area of ​​the graphics processing unit (GPU) and the read area of ​​the server memory stage, and then write the data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node, and prepare to perform the actual data transfer operation; Step 4.2: After the parsing is completed, the remote direct memory access network interface card sends a read operation request to the terminal device through the PCIe bus.

3. A method for diagnosing host internal network delay based on loopback test according to claim 2, characterized in that: In step 4.1, parsing the work queue element specifically includes the following steps: Step 4.1.1, accept the notification sent by the Doorbell mechanism: the operating system notifies the remote direct memory access network interface card (RNIC) that there is a new work queue element (WQE) to be processed by writing to the Doorbell register. The remote direct memory access network interface card (RNIC) detects the update of the Doorbell register through a hardware interrupt or polling mechanism; Step 4.1.2, the remote direct memory access network interface card (RNIC) responds to the notification sent by the Doorbell mechanism: Read the work queue head pointer: The remote direct memory access network interface card (RNIC) reads the head pointer of the current work queue element (WQE) to determine the location of the next work queue element (WQE) to be processed; Update the work queue head pointer: After processing the current work queue element (WQE), the remote direct memory access network interface card (RNIC) updates the head pointer to point to the next work queue element (WQE); Step 4.1.3, parsing the work queue element (WQE): reading the work queue element (WQE) header information: the remote direct memory access network interface card (RNIC) reads the header information of the work queue element (WQE) from the read area of ​​the server memory node, including the operation type, data length, source address and destination address; Step 4.1.4, perform the following operations on the work queue element (WQE): Read data: Based on the source address in the work queue element WQE, the remote direct memory access network interface card (RNIC) reads 8KB of data from the read area of ​​the server memory node; Direct memory access (MDA) read operation: Use direct memory access (DMA) to read data from memory without CPU intervention; Data transmission: If the work queue element (WQE) is a send operation, the remote direct memory access network interface card (RNIC) sends the data out; if the work queue element (WQE) is a receive operation, the data has been received by the remote direct memory access network interface card (RNIC) and stored in the internal buffer of the remote direct memory access network interface card (RNIC); Step 4.1.5: Write data to the write area of ​​the graphics processing unit (GPU) and the write area of ​​the server memory node: Determine the write area address: The remote direct memory access network interface card (RNIC) determines the memory area where the data should be written based on the destination address in the work queue element (WQE); Perform DMA write operations: The remote direct memory access network interface card (RNIC) uses DMA to write data to the write area of ​​the server memory node; Update memory: Data is written to the specified memory address, completing the data transfer. Step 4.1.6, complete the processing and update the status: Update work queue element (WQE) status: The remote direct memory access network interface card (RNIC) updates the status field of the work queue element WQE to indicate that the operation is completed; Send Completion Queue Element (CQE): If a completion queue is configured, the remote direct memory access network interface card (RNIC) inserts a completion queue element (CQE) into the completion queue to notify the operating system that the operation is complete.

4. The method for diagnosing internal network delay of a host based on loopback test according to claim 1, characterized in that: In step 6, a test data packet is sent from the terminal device to the remote direct memory access network interface card (RNIC). After receiving the test data packet, the remote direct memory access network interface card (RNIC) immediately returns it to the sender, and the time difference between sending and receiving is recorded, which is the round-trip delay. Specifically, the following steps are included: Step 6.1, the round-trip delay is expressed as: Among them, T proc Lat refers to the time from when the remote direct memory access network interface card (RNIC) receives the work queue element (WQE) and starts parsing until the remote direct memory access network interface card (RNIC) sends the first read request. hos is the time from when the remote direct memory access network interface card (RNIC) sends the first read request to when the remote direct memory access network interface card receives the information returned from the read area. Size refers to the size of the test message sent in the loopback test. BW host The bandwidth inside the server host; Step 6.2: Send a 1KB message to evaluate the internal latency of the host: Step 6.3: When the test message sent in the loopback test is larger than 32KB, T proc and Lat host , the expression of the loop delay is: Calculate the bandwidth BW inside the server host using the above formula host .

5. The method for diagnosing host internal network delay based on loopback test according to claim 1, characterized in that: In step 1, the network communication line is monitored to detect abnormal links, which specifically includes the following steps: Step 1.1, call the initialization function to initialize the network communication line state to "uncertain", and set the abnormal counter of each link to 0; Step 1.2: traverse all normal paths, mark all links in them as "normal", and call the router that selects the wrong path to forward data packets or selects different paths at different times due to routing failure as abnormal path; Step 1.3, traverse all abnormal paths and perform the following operations: If there are links in uncertain status in the abnormal path, these links will be marked as abnormal and the abnormal counter will be increased. If there are links in the abnormal path that have been marked as abnormal, further check whether there is a new remote direct memory access network interface card (RNIC) mark. If there is a new remote direct memory access network interface card (RNIC) mark, the abnormal counter will be increased again to further confirm the abnormal status of the link. Finally, if the status of all links in the abnormal path is normal, these links will be marked as "uncertain", indicating that there is fluctuation in the links marked as "uncertain", and finally return the links marked as abnormal and uncertain.

6. The method for diagnosing delay in a host internal network based on a loopback test according to claim 1, characterized in that: In step 7, analyze the network performance bottleneck and identify the factors that cause network delays: When the host is in an idle state, link failure or configuration error is the bottleneck cause: the module with data analysis function first determines whether the remote direct memory access network interface card (RNIC) is a bottleneck. If the path status between the remote direct memory access network interface card (RNIC) and all associated terminal devices is abnormal, the PCIe bus of the remote direct memory access network interface card (RNIC) is faulty. If the PCIe link connected to the terminal device is marked as abnormal, the data analysis module further checks the path delay between the terminal device and the remote direct memory access network interface card (RNIC). If the path delay is also abnormal, the bottleneck cause is caused by a configuration error. When the host is busy, the remote direct memory access network interface card (RNIC) needs to handle a large amount of service traffic. At this time, the measured path bandwidth will be reduced due to the competition of application traffic. In this case, the diagnostic method may still indicate an abnormal path. Applications usually use remote direct memory access network interface cards (RNIC) to communicate with their adjacent terminal devices to achieve optimal performance. If the measured bandwidth to one endpoint is significantly lower than that of other endpoints, the path is abnormal.

Images