Method for locating root cause of performance bottleneck in host, electronic device and storage medium

Abstract

The present disclosure relates to the field of computer network technology, and specifically to a method for locating the root cause of a bottleneck in a host, an electronic device, and a storage medium, the method comprising obtaining a path topology structure in the host to be tested; performing a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested; performing a path state analysis on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested. Root cause locating is performed based on a loopback test, because the use of a loopback test can ensure the reliability of the obtained path performance, and the performance bottleneck in the host is characterized by the path state, on this basis, the root cause of the performance bottleneck can be accurately located.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer networks, and in particular to a method for locating the root cause of a performance bottleneck in a host, an electronic device, and a storage medium. Background Art

[0002] Remote Direct Memory Access (RDMA) has been applied to many applications in data centers, such as distributed machine learning and distributed storage, to achieve high throughput and low latency. As the last hop of network communication, the intra-host network significantly affects the performance of network applications. However, bottlenecks may also occur in the intra-host network. The intra-host bandwidth may be reduced due to sudden link failures or being occupied by other traffic. When the intra-host bandwidth decreases, the business traffic on the RDMA network card is more likely to be blocked. Furthermore, in multi-machine systems such as distributed training scenarios, a performance bottleneck on a host may cause a serious drop in the throughput of the entire system, and in severe cases may even cause the training task to stagnate. Therefore, it is necessary to accurately locate the root cause of the performance bottleneck within the host to avoid affecting business performance. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a method for locating the root cause of a performance bottleneck in a host, an electronic device, and a storage medium to solve the problem of locating the root cause of a performance bottleneck in a host.

[0004] According to a first aspect, an embodiment of the present disclosure provides a method for locating the root cause of a performance bottleneck in a host, comprising:

[0005] Obtain the path topology structure within the host under test;

[0006] Performing a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested;

[0007] A path status analysis is performed on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested.

[0008] According to a second aspect, an embodiment of the present disclosure further provides a device for locating the root cause of a performance bottleneck in a host, comprising:

[0009] An acquisition module is used to obtain the path topology structure in the host to be tested;

[0010] A testing module, configured to perform a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested;

[0011] The analysis module is used to perform path status analysis on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested.

[0012] According to a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for locating the root cause of a performance bottleneck in a host as described in the first aspect or any one of the embodiments of the first aspect by executing the computer instructions.

[0013] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable the computer to execute the method for locating the root cause of a performance bottleneck in a host as described in the first aspect or any one of the embodiments of the first aspect.

[0014] According to a fifth aspect, an embodiment of the present disclosure provides a root cause location system for a performance bottleneck in a data center, including:

[0015] Host;

[0016] A root cause locating device, wherein the root cause locating device determines the root cause of each of the host performance bottlenecks by executing the root cause locating method of the bottleneck in the host described in the first aspect or any one of the embodiments of the first aspect, and when the host has no performance bottleneck, determines that the performance bottleneck of the data center is a network bottleneck.

[0017] The method for locating the root cause of a performance bottleneck in a host provided by the embodiment of the present disclosure performs a loopback test on a target path in the host to be tested, determines the path performance of the target path, and then analyzes the path status of the target path based on the path performance to locate the root cause of the performance bottleneck in the host to be tested. This method locates the root cause based on a loopback test, because the reliability of the obtained path performance can be guaranteed by using a loopback test, and the performance bottleneck in the host is characterized by the path status, on this basis, the root cause of the performance bottleneck can be accurately located. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure;

[0020] Figure 2 is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure;

[0021] Figure 3 is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure;

[0022] Figure 4 is a structural block diagram of a device for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure;

[0024] Figure 6 It is a structural block diagram of a system for locating the root cause of a performance bottleneck in a data center according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0026] As mentioned above, the RDMA network card is the last hop of network communication, and the performance of the host network significantly affects the performance of network applications. However, the host network may also have bottlenecks. The bandwidth within the host may be reduced due to sudden link failures or being occupied by other traffic. In the past, the bandwidth within the host was much larger than the line rate of the RDMA network card. For example, the 25Gb / s RDMA network card is compared with the 62.96Gb / s PCIe Gen3 x8 bandwidth, which provides sufficient bandwidth redundancy for the traffic on the RDMA network card. Therefore, the host network rarely becomes a bottleneck for network communication.

[0027] However, as applications continue to pursue high throughput and low latency, the line rate of RDMA network cards has increased rapidly, for example, from 25Gb / s to 200Gb / s. In contrast, the increase in bandwidth within the host does not match, for example, the PCIe bandwidth has increased from 62.96Gb / s to 252.06Gb / s. Therefore, when the bandwidth within the host decreases, the service traffic on the RDMA network card is more likely to be blocked. In addition, the topology and traffic patterns within the host have become more complex, which makes the bandwidth drop within the host caused by link failures or traffic competition more frequent. Furthermore, as the business within the host becomes more complex, the configuration items in the host have also increased significantly, which increases the probability of misconfiguration. Some of these configurations, such as enabling the access control service, will redirect the traffic between the GPU (Graphics Processing Unit, referred to as GPU) and RDMA to the CPU root complex device, resulting in a sharp increase in latency within the host and a serious reduction in bandwidth.

[0028] Performance bottlenecks within the host can seriously affect business performance. If the bandwidth within the host is lower than the receiving rate of the RDMA network card, the receiving buffer of the RDMA network card may accumulate or even be occupied. If this happens in a lossy environment, the RDMA network card may lose packets. Since RDMA is very sensitive to packet loss, even a very low packet loss rate can cause a sharp drop in throughput. If this happens in a lossless environment, the RDMA network card will send a priority-based flow control (PFC) pause frame to the egress port of the upstream switch to stop its traffic. If the RDMA network card continues to send PFC pause frames, it may trigger a PFC storm and cause the entire network to paralyze. In addition, in a multi-machine system such as a distributed training scenario, a performance bottleneck on a host may cause a serious drop in the throughput of the entire system, and in severe cases, it may even cause the training task to stagnate. Therefore, when a network bottleneck occurs within the host, it needs to be discovered and accurately located as soon as possible.

[0029] The existing bottleneck location mechanism lacks a monitoring system for network bottlenecks within the host. When performance bottlenecks occur within the host, they are usually not discovered in time. However, when the business-oriented network group reports that the business performance has degraded, the business performance has usually been seriously affected. Since the performance degradation caused by bottlenecks within the host and network bottlenecks may be similar, when network performance degrades, it is first necessary to determine whether the problem occurs in the network or within the host. Even if it is confirmed that the bottleneck occurs within the host, it is necessary to log in to the host, execute a series of test cases, and use some analysis tools to infer the bottleneck. The whole process is very time-consuming.

[0030] At the same time, existing analysis tools can only monitor specific devices, but since each host may have a different combination of devices, each diagnosis may require a different combination of tools, which will bring additional learning and execution overhead.

[0031] Based on this, the embodiments of the present disclosure provide a method for locating the root cause of a performance bottleneck in a host, which can quickly discover the performance bottleneck in the host and diagnose the root cause of the bottleneck. The solution can be deployed on all RDMA servers with low overhead. When running, it will identify the vendor of the device in the host, thereby automatically calling the API provided by different device vendors. For example, when the solution recognizes that the type of CPU is Intel, it will call the interface provided by Intel; and when the solution recognizes that the type of CPU is AMD, it will call the interface provided by AMD. Therefore, the solution can adapt to devices from different vendors. When bottlenecks occur in the host, the solution can quickly discover them and automatically diagnose their root causes. Therefore, when network performance degrades, based on this method, it can be quickly determined whether the host or the network should be blamed.

[0032] The core idea of ​​this solution is to perform a loopback test on the path within the host to determine the path status of the path within the host, and use the path status within the host to infer the performance bottleneck. This solution can be implemented based on commercial RDMA network cards and can be deployed on all RDMA servers. Of course, this solution can also be deployed on other types of servers, and its specific application scenarios are not limited here.

[0033] According to an embodiment of the present disclosure, an embodiment of a method for locating the root cause of a performance bottleneck in a host is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] In this embodiment, a method for locating the root cause of a performance bottleneck in a host is provided, which can be used in electronic devices such as computers, test equipment, etc. Figure 1 is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0035] S11, obtaining the path topology structure in the host to be tested.

[0036] The path topology is predictable. For hosts of the same type, their internal path topology is also the same. When the network connection of the host to be tested is run, the path topology in the host to be tested is obtained based on the model information.

[0037] S12, performing a loopback test on the target path in the host to be tested based on the path topology structure to determine the path performance of the target path.

[0038] The target path is a path between a network card in the host to be tested and an endpoint in the host to be tested.

[0039] The target path in the host to be tested can be all the paths in the host to be tested, that is, the paths between all the network cards and all the endpoints in the host to be tested, or it can be a specified path in the host to be tested, for example, the path connected to the abnormal network card, etc. The selection of the target path can be set according to the user's interaction, or it can be set according to the state of the host to be tested. This is because the more target paths involved in the loopback test, the more computing resources will be occupied. Therefore, setting the target path according to the state of the host to be tested can greatly reduce the impact of the loopback test on the service flow of the host to be tested.

[0040] Loopback testing refers to the data transmission process from the endpoint in the host to be tested to the network card to which it is connected and then back to the endpoint. By recording the time taken for this process, the path performance of the target path can be determined. Among them, path performance includes but is not limited to latency or bandwidth. For each target path, a loopback test is performed to determine the path performance of each target path.

[0041] S13, performing path status analysis on the target path based on the path performance, and determining the root cause of the performance bottleneck in the host to be tested.

[0042] The target path may include multiple links. Specifically, the target path refers to the path between the network card and the endpoint, and there may be nodes between the network card and the endpoint, which will divide the path into multiple links, and the connection between adjacent nodes is called a link. After the above-mentioned S12 processing, the path performance of the target path is obtained, and the path performance can be compared with the performance threshold to determine whether the path state of the target path is a normal path or an abnormal path. Further, the links in the abnormal path are analyzed to analyze the abnormal links, and then the root cause of the performance bottleneck in the process to be tested is determined. Wherein, the root cause includes but is not limited to link failure, abnormal configuration or bus overload, etc.

[0043] The method for locating the root cause of the performance bottleneck in the host provided in this embodiment performs a loopback test on the target path in the host to be tested to determine the path performance of the target path, and then analyzes the path status of the target path based on the path performance to locate the root cause of the performance bottleneck in the host to be tested. This method is based on loopback testing to locate the root cause. Since the use of loopback testing can ensure the reliability of the obtained path performance, and the performance bottleneck in the host to be tested is characterized by the path status, on this basis, the root cause of the performance bottleneck can be accurately located.

[0044] In this embodiment, a method for locating the root cause of a performance bottleneck in a host is provided, which can be used in electronic devices such as computers, test equipment, etc. Figure 2 is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0045] S21, obtaining a path topology structure in the host to be tested.

[0046] For details, please see Figure 1 S11 of the illustrated embodiment will not be described in detail here.

[0047] S22, performing a loopback test on the target path in the host to be tested based on the path topology structure to determine the path performance of the target path.

[0048] The target path is a path between a network card in the host to be tested and an endpoint in the host to be tested.

[0049] Specifically, the above S22 includes:

[0050] S221, registering a first memory space and a second memory space in a target endpoint of the host to be tested.

[0051] The target endpoint is the endpoint of the target path.

[0052] For each target path, two memory spaces are registered in the endpoint of the target path, namely, a first memory space for sending data and a second memory space for receiving data. The two memory spaces can be registered when a loopback test is required, and when the loopback test is completed, the two memory spaces are unregistered to release the memory resources of the endpoint.

[0053] S222: Send a first data read instruction to the network card to read first data of a first length from the first memory space to the cache space of the network card and write the read first data into the second memory space.

[0054] The path between the network card and the endpoint is the target path, that is, for each target path, the network card corresponding to each target path is determined, and a first data read instruction is sent to the network card to inform the network card to read the first data of the first length from the first memory space to its cache space, and then directly write the first data into the second memory space. The first length is set according to actual needs and is not limited here.

[0055] S223, recording a first time when the first data is read from the first memory space, and a second time when the network card stores the first data into the second memory space.

[0056] S224: Determine a delay of the target path based on a first time difference between the second time and the first time to determine a path performance of the target path.

[0057] When the network card corresponding to the target path reads the first data from the first memory space, the reading time is recorded as the first time; correspondingly, when the network card stores the first data in the second memory space, the storage time is recorded as the second time. The time difference between the second time and the first time is then used to measure the time of the loopback test. Since the entire process occurs completely inside the host under test and does not require the participation of the network outside the host under test, the result of the loopback test can provide feedback on the path performance of the target path in the host under test. (1)

[0058] The loopback test time Lat can be expressed by the following formula:

[0059]

[0060] Where, T proc Lat is the processing delay of the network card, which includes two parts: the interval time from the time of issuing the first data read instruction to the time when the network card issues the first read request and the time from the time when the network card sends all write data packets to the time when the first data storage is completed; Lat host It is the propagation delay of the data packet in the host under test, that is, the round-trip delay of the data packet from the network card to the endpoint and then back to the network card; The transmission delay of the message sent in the host under test, that is, the message size Size / the bus bandwidth BW in the host under test host , where the bus bandwidth is determined by the minimum value of the bus read bandwidth and the bus write bandwidth.

[0061] In some implementations, the above S224 includes:

[0062] (1) Sending a second data read instruction to the network card to read second data of a second length from the first memory space to the cache space of the network card and write the read second data to the second memory space, wherein the second length is greater than the first length.

[0063] (2) Recording a third time when the network card reads the second data from the first memory space, and a fourth time when the network card stores the second data into the second memory space.

[0064] (3) Calculate a second time difference between the fourth time and the third time, and determine the difference between the second time difference and the first time difference as the bandwidth of the target path.

[0065] The reading and storing method of the second data of the second length is similar to the reading and storing method of the first data of the first length, which will not be repeated here. After the second data of the second length is read and stored, the second time difference is obtained. The difference between the second time difference and the first time difference is the bandwidth of the target path.

[0066] Specifically, in combination with the above formula (1), when the first data of the first length is used, the above formula (1) can be approximated as follows:

[0067]

[0068] When the network card utilization is low and there is no bottleneck in the host, the processing delay T in the above formula is proc Usually remain unchanged. Therefore, the measured loop delay can feedback the delay of the target path in the host to be tested. When the second data of the second length is selected, the above formula (1) can be approximated as:

[0069]

[0070] The measured loop delay can then provide feedback on the bandwidth of the target path in the host to be tested. Based on this, the bandwidth of the target path in the host to be tested can be reflected by taking the difference between the second time difference and the first time difference, so there is no need to use a very large data size.

[0071] Since the path delay is composed of three parts, namely the processing delay of the network card, the propagation delay of the data packet in the host, and the transmission delay of the message sent in the host, whether using the first data of the first length or the second data of the second length, the first two parts of the measured path delay can be approximately considered to be consistent, so the third part of the path delay can be obtained by subtracting the delay of the first data of the first length from the delay of the second data of the second length, thereby calculating the bandwidth of the target path. Therefore, this method does not need to use very large data for bandwidth testing.

[0072] In some implementations, the above S22 further includes:

[0073] (1) Obtain the status of the host to be tested, where the status includes idle or busy.

[0074] (2) Determine the target path based on the status of the host under test.

[0075] (3) Perform a loopback test on the target path to determine the path performance of the target path.

[0076] During the operation of the host under test, the status of the host under test is monitored in real time. If the host under test is in the idle state, in order to further ensure the accuracy of the root cause location, a loopback test is performed on all paths in the host under test; if the host under test is in the busy state, a loopback test can be performed only on the specified paths in the host under test. For the specific processing details of the loopback test, please refer to the above description and will not be repeated here.

[0077] Specifically, when the state of the host to be tested is idle, the target path is determined to be the path between all network cards in the host to be tested and all endpoints. When the state of the host to be tested is busy, the target path is determined to be the path between the abnormal network card in the host to be tested and the first target endpoint, and the first target endpoint includes all memory nodes of the host to be tested and the graphics processor under the same CPU root node as the abnormal network card.

[0078] According to the different states of the host to be tested, the target path in each state is determined, and the loopback test is performed on the corresponding target path, which can reduce the impact of the loopback test on the service flow. For the links under the same root node as the abnormal network card, this solution only uses the abnormal network card to determine the status of these links to reduce the impact on the service flow and avoid the wrong inference of the bottleneck in the host to be tested.

[0079] S23, performing path status analysis on the target path based on the path performance, and determining the root cause of the performance bottleneck in the host to be tested.

[0080] For details, please see Figure 1 S13 of the illustrated embodiment will not be described in detail here.

[0081] The method for locating the root cause of a performance bottleneck in a host provided in this embodiment implements a loopback test of a target path by reading first data from a first memory space and storing the first data in a second memory space. Due to the convenience of data reading and writing, the path performance of the target path can be determined relatively quickly.

[0082] In this embodiment, a method for locating the root cause of a performance bottleneck in a host is provided, which can be used in electronic devices such as computers, test equipment, etc. Figure 3 is a flow chart of a method for locating the root cause of a performance bottleneck in a host according to an embodiment of the present disclosure. Figure 3 As shown, the process includes the following steps:

[0083] S31, obtaining a path topology structure in the host to be tested.

[0084] For details, please see Figure 1 S11 of the illustrated embodiment will not be described in detail here.

[0085] S32, performing a loopback test on the target path in the host to be tested based on the path topology structure to determine the path performance of the target path.

[0086] The target path is a path between a network card in the host to be tested and an endpoint in the host to be tested.

[0087] For details, please see Figure 1 S12 of the illustrated embodiment will not be described in detail here.

[0088] S33, performing path status analysis on the target path based on the path performance, and determining the root cause of the performance bottleneck in the host to be tested.

[0089] Specifically, the above S33 includes:

[0090] S331, determining an abnormal link in the target path based on the path performance.

[0091] As mentioned above, path performance includes but is not limited to delay or bandwidth. Based on this, the delay or bandwidth obtained by the loopback test is compared with the corresponding delay threshold or bandwidth threshold to determine the abnormal path in the target path. The abnormal path is further analyzed to determine the abnormal link.

[0092] Alternatively, the path performance is compared to determine the mean of the path performance; the path performance of each target path is then compared with the mean to determine the abnormal path and then the abnormal link.

[0093] In some implementations, the above S331 includes:

[0094] (1) Based on the path performance, determine the state information of the target path, where the state information includes a normal path or an abnormal path.

[0095] (2) Query the status of each link in the abnormal path and classify the link with uncertain link status as an abnormal link.

[0096] (3) When there is an abnormal path where all the links are in a normal state, the state of the abnormal path where all the links are in a normal state is identified as a jitter state.

[0097] The process of determining the state information of the target path is as described above and will not be repeated here. Of course, if it is determined that the state information of all target paths is a normal path, then it can be determined that there is no performance bottleneck in the host under test; if there is an abnormal path, it is determined that there is a performance bottleneck in the host under test, and the root cause of the bottleneck needs to be located.

[0098] After the above steps, the normal path and the abnormal path can be determined. First, all target paths are marked as uncertain, then all normal paths are traversed, and all links on the normal paths are marked as normal. Then all abnormal paths are traversed, and if there are uncertain links on the abnormal paths, these links are marked as abnormal links.

[0099] Furthermore, if the status of all links on an abnormal path is normal, then there may be a jittering link on this path, and the status of all links on the path is marked as a jittering state, for example, a gray state is used to represent the jittering state. If a link is judged to be in a gray state for N consecutive measurement cycles, then the link will be diagnosed as a jittering link. Among them, N can be set according to actual needs, for example, N = 2 or 3, etc.

[0100] In some implementations, when the state of the host to be tested is busy, the path performance includes bandwidth. Based on this, the above S331 includes:

[0101] (1) Compare the bandwidth of each target path.

[0102] (2) Determine the abnormal path in the target path based on the comparison result of the bandwidth.

[0103] (3) Query the status of each link in the abnormal path, and set the link with uncertain link status as an optional abnormal link.

[0104] (4) Count the number of network cards that are connected to the optional abnormal link, and determine the optional abnormal link with the largest number as the abnormal link.

[0105] When the host to be tested is busy, there may be business traffic on some network cards. The loopback bandwidth measured by these network cards will be significantly reduced by the business traffic. Based on this, the path status is judged by comparison to ensure the accuracy of the abnormal link determined. Specifically, the bandwidth of each target path is compared to determine the abnormal path. For example, relative values ​​are used to infer abnormal paths. When the bandwidth of a target path measured by the network card is significantly lower than the bandwidth of other target paths, the target path is judged to be an abnormal path.

[0106] At the same time, since the network card with business traffic cannot determine whether a path is normal, some links may be mistakenly marked as abnormal, resulting in a decrease in the inference accuracy of abnormal links. In this case, by querying the status of each link in the abnormal path, the link with uncertain link status is selected as an abnormal link, the number of network cards with connection relationship with the optional abnormal link is counted, and the optional abnormal link with the largest number is determined as an abnormal link. That is, the link with the highest number is most likely to be a bottleneck link and should receive more attention. For example, in the process of traversing the abnormal path, the abnormal path 1 (network card 1 to a certain endpoint) is first traversed. This path passes through link 1 (in an uncertain state), then link 1 will be marked as an abnormal state, and the count value abnormal_cnt of link 1 will change from 0 to 1; then if abnormal path 2 (network card 1 to another endpoint) also passes through link 1, since it is also marked by network card 1, the count value abnormal_cnt of link 1 will remain unchanged; and if abnormal path 3 (network card 2 to a certain endpoint) also passes through link 1, since it is marked abnormal by the new network card, the count value abnormal_cnt of link 1 will increase by 1. That is, by counting the number of different network cards connected to the abnormal link, the count value abnormal_cnt of the abnormal link is obtained. The role of abnormal_cnt is to infer the abnormal link when it is triggered by the abnormal indicator on the network card. This is because the amount of information that can be obtained by the loopback test at this time is reduced, so more paths may be marked as abnormal. At this time, the link with a higher abnormal_cnt is more likely to be an abnormal link.

[0107] S332: Determine the root cause of the performance bottleneck in the host to be tested based on the information of the abnormal link.

[0108] The information of the abnormal link includes the location or utilization rate of the abnormal link.

[0109] After obtaining the abnormal link, analyze the root cause of the performance bottleneck in the host under test based on the information of the abnormal link. When the host under test is in an idle state, the root cause of the bottleneck is usually a link failure or abnormal configuration; when the host under test is in a busy state, the root cause of the bottleneck is usually overload. When the host under test is in an idle state, regularly monitor the bottleneck in the host under test; when the host under test is in a busy state, run the root cause location of the bottleneck in the host under test only when abnormal indicators appear on the network card.

[0110] When the state of the host to be tested is idle, the above S332 includes:

[0111] (1) Use the path topology structure to determine the location of the abnormal link.

[0112] (2) When there are abnormal links on the path from the network card to the second target endpoint, it is determined that the root cause of the performance bottleneck in the host under test is a link failure, and the second target endpoint is all memory nodes of the host under test and a graphics processor under the same central processing unit root node as the network card.

[0113] The path topology is used to represent the connection of the path in the host to be tested. After the above processing, the abnormal link is determined, and then the location of the abnormal link is determined. That is, the node connected by the abnormal link. If the path between the network card and all the second target endpoints is abnormal, it is determined that the root cause of the bottleneck is the bus link failure of the network card.

[0114] When the host to be tested is idle, the location of the abnormal path is used to determine the root cause, thereby ensuring the accuracy of the determined root cause.

[0115] In some embodiments, step (2) of S332 above includes:

[0116] 2.1) When there are abnormal links on the path from the network card to the target endpoint, the delay of the path to be tested from the network card to the graphics processor under the same root node as the graphics processor is obtained.

[0117] 2.2) When the delay is abnormal, it is determined that the root cause of the performance bottleneck of the path to be tested is abnormal configuration, and the root cause of the performance bottleneck of other paths is link failure, where the other paths are paths between the network card and the target endpoint except the path to be tested.

[0118] When there are abnormal links on the path from the network card to the target endpoint, if the bus link status connected to the GPU is abnormal, the delay from the network card under the same root node as the GPU to the GPU will be further checked. If the delay is also abnormal, it is determined that the root cause of the bottleneck is abnormal configuration; for other abnormal links, their root cause will be determined as link failure.

[0119] In the case of an abnormal link, it is necessary to make another judgment based on the delay to further distinguish whether the root cause is an abnormal configuration or a link failure, thereby improving the reliability of the determined root cause.

[0120] When the state of the host to be tested is busy, the root cause is located when an abnormal indicator appears on the network card, that is, the network card with the abnormal indicator is defined as an abnormal network card. Based on this, the above S332 includes:

[0121] (1) Obtain the utilization of abnormal links.

[0122] (2) When the utilization rate is higher than a first utilization rate threshold, it is determined that the root cause of the performance bottleneck in the host to be tested is a bus overload.

[0123] (3) When the utilization rate is lower than a second utilization rate threshold, it is determined that the root cause of the performance bottleneck in the host to be tested is a link failure, and the second utilization rate threshold is lower than the first utilization rate threshold.

[0124] The link utilization is obtained through the bus monitoring module, that is, the bus monitoring module is responsible for monitoring the bus link utilization in the host to be tested, including the PCIe link connected to the RDMA network card, the PCIe link connected to the GPU, the CPU root port, the cross-CPU socket bus, the memory channel, etc. When a link is determined to be abnormal, the utilization of the link can be monitored to determine whether the root cause of the problem is a link failure or traffic competition.

[0125] When the abnormal indicator on the abnormal network card is triggered, the abnormal link is usually in an overloaded state. Taking advantage of this, the bus monitoring module is used to locate the root cause of the problem. When the utilization rate of the abnormal link is high, it is determined that the root cause of the bottleneck is bus overload. When the utilization rate of the abnormal link is low, it is determined that the root cause of the bottleneck is link failure. Among them, higher means higher than the first utilization threshold, and lower means lower than the second utilization threshold. The specific values ​​of the first utilization threshold and the second utilization threshold are set according to actual needs and are not limited here.

[0126] When there is an abnormal network card, the abnormal link is usually in an overloaded state. Taking advantage of this, the utilization rate of the abnormal link is used to locate the root cause of the problem, ensuring the accuracy of root cause location.

[0127] The method for locating the root cause of a performance bottleneck in a host provided in this embodiment selects abnormal links in a target path by the size of the path performance, and then locates the root cause using the information of the abnormal link, thereby ensuring the accuracy of the determined root cause.

[0128] In this embodiment, a device for locating the root cause of a performance bottleneck in a host is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a device for locating the root cause of a performance bottleneck in a host. Figure 4 As shown, including:

[0130] An acquisition module 41 is used to acquire a path topology structure in the host to be tested;

[0131] A testing module 42, configured to perform a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested;

[0132] The analysis module 43 is used to perform path status analysis on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested.

[0133] In some embodiments, the testing module 42 includes:

[0134] A registration unit, used to register a first memory space and a second memory space in a target endpoint of the host to be tested, wherein the target endpoint is an endpoint of the target path;

[0135] an instruction issuing unit, configured to issue a first data reading instruction to the network card, so as to read first data of a first length from the first memory space to a cache space of the network card and write the read first data into the second memory space;

[0136] a recording unit, configured to record a first time when the first data is read from the first memory space, and a second time when the network card stores the first data into the second memory space;

[0137] The second determining unit is configured to determine a delay of the target path based on a first time difference between the second time and the first time, so as to determine a path performance of the target path.

[0138] In some embodiments, the determining unit includes:

[0139] an instruction issuing subunit, configured to issue a second data reading instruction to the network card, so as to read second data of a second length from the first memory space to the cache space of the network card and write the read second data into the second memory space, wherein the second length is greater than the first length;

[0140] a recording subunit, used to record a third time when the network card reads the second data from the first memory space, and a fourth time when the network card stores the second data into the second memory space;

[0141] The first determining subunit is configured to calculate a second time difference between the fourth time and the third time, and determine a difference between the second time difference and the first time difference as a bandwidth of the target path.

[0142] In some embodiments, the testing module 42 further includes:

[0143] An acquiring unit, used for acquiring the status of the host to be tested, wherein the status includes idle or busy;

[0144] A second determining unit, configured to determine the target path based on the state of the host to be tested;

[0145] The testing unit is used to perform a loopback test on the target path to determine the path performance of the target path.

[0146] In some implementations, the second determining unit includes:

[0147] A first determining subunit is used to determine, when the state of the host to be tested is idle, that the target path is a path between all network cards in the host to be tested and all endpoints;

[0148] The second determination subunit is used to determine that the target path is a path between the abnormal network card in the host to be tested and the first target endpoint when the state of the host to be tested is busy, and the first target endpoint includes all memory nodes of the host to be tested and a graphics processor under the same central processing unit root node as the abnormal network card.

[0149] In some embodiments, the analysis module 43 includes:

[0150] A third determining unit, configured to determine an abnormal link in the target path based on the magnitude of the path performance;

[0151] The fourth determining unit is used to determine the root cause of the performance bottleneck in the host to be tested based on the information of the abnormal link, and the information of the abnormal link includes the location or utilization rate of the abnormal link.

[0152] In some implementations, the third determining unit includes:

[0153] A third determining subunit, configured to determine state information of the target path based on the size of the path performance, wherein the state information includes a normal path or an abnormal path;

[0154] A first query subunit is used to query the status of each link in the abnormal path, and regard the link with uncertain link status as the abnormal link;

[0155] The identification subunit is used to identify the state of the abnormal path where the states of all links are normal as a jitter state when there is an abnormal path where the states of all links are normal.

[0156] In some implementations, when the state of the host to be tested is idle, the fourth determining unit includes:

[0157] A fourth determining subunit, configured to determine the location of the abnormal link using the path topology structure;

[0158] The fifth determination subunit is used to determine that the root cause of the performance bottleneck in the host to be tested is a link failure when the abnormal link exists on the path from the network card to the second target endpoint, and the second target endpoint is all memory nodes of the host to be tested and the graphics processor under the same central processing unit root node as the network card.

[0159] In some embodiments, the fifth determining subunit includes:

[0160] A first acquisition subunit is used to acquire the delay of the path to be tested between the network card under the same root node as the graphics processor and the graphics processor when the abnormal link exists on the path between the network card and the target endpoint;

[0161] The sixth determination subunit is used to determine that when the delay is abnormal, the root cause of the performance bottleneck caused by the path to be tested is abnormal configuration, and to determine that the root cause of the performance bottleneck caused by other paths is link failure. The other paths are paths between the network card and the target endpoint except the path to be tested.

[0162] In some implementations, when the state of the host to be tested is busy, the path performance includes bandwidth, and the third determining unit includes:

[0163] A comparison subunit, used for comparing the bandwidth of each of the target paths;

[0164] a seventh determining subunit, configured to determine an abnormal path in the target path based on the comparison result of the bandwidth;

[0165] A second query subunit is used to query the status of each link in the abnormal path, and to define the link with uncertain link status as an optional abnormal link;

[0166] The counting subunit is used to count the number of network cards connected to the optional abnormal link, and determine the optional abnormal link with the largest number as the abnormal link.

[0167] In some implementations, the fourth determining unit includes:

[0168] A second acquisition subunit is used to acquire the utilization rate of the abnormal link;

[0169] an eighth determination subunit, configured to determine that the root cause of the performance bottleneck in the host to be tested is a bus overload when the utilization rate is higher than a first utilization rate threshold;

[0170] The ninth determination subunit is used to determine that the root cause of the performance bottleneck in the host to be tested is a link failure when the utilization rate is lower than a second utilization rate threshold, and the second utilization rate threshold is smaller than the first utilization rate threshold.

[0171] The root cause location device for the performance bottleneck in the host in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0173] The present disclosure also provides an electronic device having the above Figure 4 The device shown is used to locate the root cause of the performance bottleneck within the host.

[0174] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 5 As shown, the electronic device may include: at least one processor 51, such as a CPU (Central Processing Unit), at least one communication interface 53, a memory 54, and at least one communication bus 52. The communication bus 52 is used to realize the connection and communication between these components. The communication interface 53 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 53 may also include a standard wired interface and a wireless interface. The memory 54 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 54 may optionally be at least one storage device located away from the aforementioned processor 51. The processor 51 may be combined with Figure 4 In the described device, the memory 54 stores an application program, and the processor 51 calls the program code stored in the memory 54 to execute any of the above method steps.

[0175] The communication bus 52 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 52 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0176] Among them, the memory 54 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 54 may also include a combination of the above types of memory.

[0177] The processor 51 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0178] The processor 51 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0179] Optionally, the memory 54 is also used to store program instructions. The processor 51 can call the program instructions to implement the method for locating the root cause of the performance bottleneck in the host as shown in any embodiment of the present application.

[0180] The disclosed embodiment also provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the root cause location method of the performance bottleneck in the host in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0181] The disclosed embodiment also provides a root cause location system for performance bottlenecks in a data center, such as Figure 6 As shown, it includes: a host 10 and a root cause locating device 20, wherein the root cause locating device 20 is used to locate the root cause of the performance bottleneck in the data center. It should be noted that the root cause locating device 20 here can be a host 10 in the data center, or it can be an additional electronic device, and there is no restriction on the root cause locating device 20. A plurality of hosts 10 are arranged in the data center, and each host 10 serves as a processing node in the data center.

[0182] The root cause location of the performance bottleneck in the data center can be determined by timing triggering or by demand triggering. There is no limit on the triggering time. For example, when the host in the data center is idle, it can be triggered by timing; if the host in the data center is busy, it can be triggered only when the network card in the host is abnormal.

[0183] The root cause location device 20 uses the above-mentioned root cause location method of the performance bottleneck in the host to first determine whether the performance bottleneck in the data center is a bottleneck in the host or a network bottleneck. If the above-mentioned method is used to determine that there is no performance bottleneck in the host, the performance bottleneck of the data center is determined to be a network bottleneck.

[0184] The root cause location system for performance bottlenecks in a data center provided in this embodiment deploys a root cause location method in a data center, which can quickly locate whether the performance bottleneck of the data center occurs in a computing node or a network. For known performance bottlenecks in a host, the root cause of the bottleneck can be located within minutes. At the same time, the solution can assist operation and maintenance personnel in discovering various host performance bottlenecks that were not discovered before the data center was deployed.

[0185] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method embodiment, since it is basically similar to the device and system embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the device and system embodiments.

[0186] It is understandable that in the specific implementation of the present disclosure, the collection of data in the host is involved. When the above embodiments of the present disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0187] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for locating the root cause of a performance bottleneck in a host, characterized in that: include: Obtain the path topology structure within the host under test; Performing a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested; Performing path status analysis on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested; The step of performing a loopback test on the target path in the host to be tested based on the path topology structure to determine the path performance of the target path includes: Registering a first memory space and a second memory space in a target endpoint of the host to be tested, wherein the target endpoint is an endpoint of the target path; Sending a first data read instruction to the network card to read first data of a first length from the first memory space to a cache space of the network card, and writing the read first data into the second memory space; Recording a first time when the first data is read from the first memory space, and a second time when the network card stores the first data into the second memory space; Based on a first time difference between the second time and the first time, a delay of the target path is determined to determine a path performance of the target path.

2. The method according to claim 1, characterized in that The determining, based on a first time difference between the second time and the first time, a delay of the target path to determine a path performance of the target path includes: Sending a second data read instruction to the network card to read second data of a second length from the first memory space to the cache space of the network card and write the read second data into the second memory space, wherein the second length is greater than the first length; Recording a third time when the network card reads the second data from the first memory space, and a fourth time when the network card stores the second data into the second memory space; A second time difference between the fourth time and the third time is calculated, and a difference between the second time difference and the first time difference is determined as a bandwidth of the target path.

3. The method according to claim 1, characterized in that The loopback test is performed on the target path in the host to be tested based on the path topology structure to determine the path performance of the target path, and further includes: Obtaining the status of the host to be tested, where the status includes idle or busy; Determine the target path based on the state of the host to be tested; A loopback test is performed on the target path to determine the path performance of the target path.

4. The method according to claim 3, characterized in that: The determining the target path based on the state of the host to be tested includes: When the state of the host to be tested is idle, determining that the target path is a path between all network cards in the host to be tested and all endpoints; When the state of the host to be tested is busy, the target path is determined to be a path between an abnormal network card in the host to be tested and a first target endpoint, wherein the first target endpoint includes all memory nodes of the host to be tested and a graphics processor under the same central processing unit root node as the abnormal network card.

5. The method according to claim 1, characterized in that The performing path status analysis on the target path based on the path performance to determine the root cause of the performance bottleneck in the host to be tested includes: Based on the magnitude of the path performance, determining an abnormal link in the target path; The root cause of the performance bottleneck in the host to be tested is determined based on the information of the abnormal link, where the information of the abnormal link includes the location or utilization rate of the abnormal link.

6. The method according to claim 5, characterized in that The determining, based on the magnitude of the path performance, an abnormal link in the target path includes: Based on the size of the path performance, determining the state information of the target path, the state information including a normal path or an abnormal path; Querying the status of each link in the abnormal path, and defining the link with uncertain link status as the abnormal link; When there is an abnormal path in which the states of all links are normal, the state of the abnormal path in which the states of all links are normal is identified as a jitter state.

7. The method according to claim 5, characterized in that When the state of the host to be tested is idle, determining the root cause of the performance bottleneck in the host to be tested based on the information of the abnormal link includes: Determining the location of the abnormal link using the path topology structure; When the abnormal link exists on the path between the network card and the second target endpoint, it is determined that the root cause of the performance bottleneck in the host to be tested is a link failure, and the second target endpoint is all memory nodes of the host to be tested and a graphics processor under the same central processing unit root node as the network card.

8. The method according to claim 7, characterized in that When the abnormal link exists on the path between the network card and the second target endpoint, determining that the root cause of the performance bottleneck in the host to be tested is a link failure includes: When the abnormal link exists on the path between the network card and the target endpoint, obtaining the delay of the path to be tested between the network card under the same root node as the graphics processor and the graphics processor; When the delay is abnormal, it is determined that the root cause of the performance bottleneck of the path to be tested is abnormal configuration, and the root cause of the performance bottleneck of other paths is link failure. The other paths are paths between the network card and the target endpoint except the path to be tested.

9. The method according to claim 5, characterized in that When the state of the host to be tested is busy, the path performance includes bandwidth, and determining the abnormal link in the target path based on the size of the path performance includes: comparing bandwidths of the target paths; determining an abnormal path in the target path based on the comparison result of the bandwidth; Querying the status of each link in the abnormal path, and setting the link with uncertain link status as an optional abnormal link; The number of network cards connected to the optional abnormal link is counted, and the optional abnormal link with the largest number is determined as the abnormal link.

10. The method according to claim 9, characterized in that The determining the root cause of the performance bottleneck in the host to be tested based on the information of the abnormal link includes: Obtaining the utilization rate of the abnormal link; When the utilization rate is higher than a first utilization rate threshold, determining that the root cause of the performance bottleneck in the host to be tested is a bus overload; When the utilization rate is lower than a second utilization rate threshold, it is determined that the root cause of the performance bottleneck in the host to be tested is a link failure, and the second utilization rate threshold is smaller than the first utilization rate threshold.

11. A device for locating the root cause of a performance bottleneck in a host, characterized in that: include: An acquisition module is used to obtain the path topology structure in the host to be tested; A testing module, configured to perform a loopback test on a target path in the host to be tested based on the path topology structure to determine the path performance of the target path, wherein the target path is a path between a network card in the host to be tested and an endpoint in the host to be tested; An analysis module, configured to perform path status analysis on the target path based on the path performance, and determine the root cause of the performance bottleneck in the host to be tested; Wherein, the test module includes: A registration unit, used to register a first memory space and a second memory space in a target endpoint of the host to be tested, wherein the target endpoint is an endpoint of the target path; an instruction issuing unit, configured to issue a first data reading instruction to the network card, so as to read first data of a first length from the first memory space to a cache space of the network card, and write the read first data into the second memory space; a recording unit, configured to record a first time when the first data is read from the first memory space, and a second time when the network card stores the first data into the second memory space; The second determining unit is configured to determine a delay of the target path based on a first time difference between the second time and the first time, so as to determine a path performance of the target path.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for locating the root cause of a performance bottleneck in a host according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for locating the root cause of a performance bottleneck in a host according to any one of claims 1-10.

14. A root cause location system for performance bottlenecks in a data center, characterized in that: include: Host; A root cause locating device, wherein the root cause locating device determines the root cause of each of the host performance bottlenecks by executing the root cause locating method for the bottleneck within the host according to any one of claims 1 to 10, and when the host has no performance bottleneck, determines that the performance bottleneck of the data center is a network bottleneck.

Images