Method, apparatus, device, medium and program product for host probing

By recording the sending and receiving time of detection messages, accurately determining the communication network delay in the host cluster, it solves the problem of difficult to distinguish between network failures and host failures in traditional detection strategies, and improves the troubleshooting efficiency.

CN118984289APending Publication Date: 2024-11-19DOUYIN VISION CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411179784.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In large-scale host clusters, traditional detection strategies are difficult to accurately determine the causes of communication network failures and host failures, resulting in low efficiency in handling exception events and inaccurately affecting the processing priority of services.

Method used

The network delay of the communication network is determined based on these times to accurately determine the cause of the abnormality.

Benefits of technology

It realizes accurate positioning of abnormal events in the host cluster, improves troubleshooting efficiency, accurately locates communication network or host failures, and optimizes operation and maintenance processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984289A_ABST
    Figure CN118984289A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method, device and equipment for host detection, a medium and a program product. The method comprises the following steps: in response to a detection task received by a detection host, sending a detection message to a target host through a network card of the detection host; adding first operation information related to the sending of the detection message to a completion queue of a network card of the detection host, wherein the first operation information at least comprises the sending time of the detection message; under the condition that at least one response message is received from the target host through the network card of the detection host, second operation information related to receiving of the at least one response message is added to a completion queue of the network card of the detection host, and the second operation information indicates receiving time comprising the at least one response message; and determining a first detection result based on the first operation information and the second operation information, the first detection result indicating a network delay of a communication network between the detection host and the target host.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, computer-readable storage media, and computer program products for host detection. Background Art

[0002] A host cluster is a collection of computer resources consisting of a group of hosts. The hosts are connected through the host cluster's internal communication network (such as one or more switches), and the hosts work together to perform tasks. With the rapid development of information technology, the scale of host clusters is getting larger and larger in order to provide greater computing power. However, at the same time, the probability of failure of hosts or communication networks in the host cluster is also increasing. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for host detection is provided. The method includes: in response to a detection host receiving a detection task, sending a detection message to a target host through a network card of the detection host; adding first operation information related to the sending of the detection message to a completion queue of the network card of the detection host, the first operation information at least including the sending time of the detection message; in the case of receiving at least one reply message from the target host via the network card of the detection host, adding second operation information related to the reception of at least one reply message to the completion queue of the network card of the detection host, the second operation information indicating the receiving time of at least one reply message; and determining a first detection result based on the first operation information and the second operation information, the first detection result indicating the network delay of the communication network between the detection host and the target host.

[0004] In a second aspect of the present disclosure, a device for host detection is provided. The device includes: a sending module, configured to send a detection message to a target host through a network card of the detection host in response to the detection host receiving a detection task; a first adding module, configured to add first operation information related to the sending of the detection message to the completion queue of the network card of the detection host, the first operation information at least including the sending time of the detection message; a second adding module, configured to add second operation information related to the reception of at least one reply message to the completion queue of the network card of the detection host in the case of receiving at least one reply message from the target host via the network card of the detection host, the second operation information indicating the receiving time of at least one reply message; and a determining module, configured to determine a first detection result based on the first operation information and the second operation information, the first detection result indicating the network delay of the communication network between the detection host and the target host.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and comprises computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart of a process for host detection according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing an example architecture for host detection according to some embodiments of the present disclosure;

[0013] Figure 4 A flow chart showing a signaling flow for host detection according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic structural block diagram of a device for host detection according to some embodiments of the present disclosure is shown; and

[0015] Figure 6 An electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that perform operations of the technical solution of the present disclosure based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, the environment 100 includes a controller 110 , an analyzer 120 , and hosts 130 , 150 , with the host 130 and the host 150 being connected via a communication network 140 .

[0025] Host 130 and host 150 may be any two hosts in a host cluster, and communication network 140 may be an internal communication network in the host cluster that connects host 130 and host 150. It is understandable that, although only hosts 130 and 150 are shown in environment 100, in actual applications, the host cluster may include far more than two hosts, for example, the host cluster may include hundreds, thousands, or tens of thousands of hosts.

[0026] like Figure 1 As shown, the host 130 may include one or more network cards 131 (e.g., network card 131-1, network card 131-2, network card 131-3, ..., network card 131-N, etc.). The host 150 may also include one or more network cards 151 (e.g., network card 151-1, network card 151-2, network card 151-3, ..., network card 151-N, etc.). The network cards 131 and 151 may be remote direct memory access (RDMA) network cards, and the RDMA network card (i.e., RNIC) may perform RDMA read operations or RDMA write operations, and exchange data without any kernel memory involvement. Thus, the data transmission efficiency of the RDMA network card is improved.

[0027] The communication network 140 may include multiple switches 141 (e.g., switch 141-1, switch 141-2, ..., switch 141-N, etc.). The host 130 and the host 150 may be connected through one or more switches 141. One or more network cards 131 of the host 130 may be connected to the same switch 141, or may be connected to multiple switches 141 respectively. For example, multiple network cards 131 may be connected to the switch 141-1, so that the multiple network cards 131 may communicate via the switch 141-1. One or more network cards 151 of the host 150 may be connected to the same switch 141, or may be connected to multiple switches 141 respectively. It is understandable that although the communication network 140 in the environment 100 only shows the switch 141, in actual application, the communication network 140 may also include other network devices, and it should not be understood that the communication network 140 is limited to including only the switch 141.

[0028] In some embodiments, the host 130 can communicate with the host 150, other hosts or other electronic devices through the RDMA network card 141 to achieve data transmission. The hosts 130 and 150 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the hosts 130 and 150 can support any type of interface for the user (such as "wearable" circuits, etc.). In some embodiments, the hosts 130 and 150 can also be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0029] In an embodiment of the present disclosure, the controller 110 is used to formulate a detection task according to the topological structure of the host cluster. Afterwards, the detection task is issued to the host 130, the host 150 or other hosts in the host cluster. The host that receives the detection task can be called a detection host, and the detected target of the detection task is usually also a host in the host cluster, and the detected host can be called a target host. For example, if the host 130 receives a detection task from the controller 110, and the detected target of the detection task is the host 150, then the host 130 is the detection host and the host 150 is the target host. It should also be noted that the detection host and the target host can be the same host or different hosts. After the detection host executes the detection task, the detection result can be obtained, and the detection host can send the detection result to the analyzer 120.

[0030] The analyzer 120 can analyze the operating status of the host cluster according to one or more detection results, such as the operating status of the network card, the operating status of the host, the network status of the communication network, etc. The controller 110 and the analyzer 120 here can be formed by a software program or by a hardware device. In the case where the controller 110 and the analyzer 120 are formed by a software program, the controller 110 and the analyzer 120 can be deployed on the same host in the host cluster, or can be deployed on different hosts respectively.

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0032] As mentioned above, with the rapid development of information technology, the scale of host clusters is getting larger and larger, especially the host clusters using RDMA technology. Due to the characteristics of RDMA technology such as low latency, high throughput and low CPU consumption, RDMA networks have been widely used in the internal communication network of host clusters. In particular, the scale of RDMA networks used in distributed training services is relatively large. The development trend of distributed training services is to use a large number of graphics processing units (GPUs) to train large-scale neural network models to solve complex problems. The scale of host clusters used for distributed training services has grown from hundreds of GPUs to tens of thousands of GPUs. As the number of GPUs increases, host clusters also need more RDMA network cards, switches, and network links for networking.

[0033] The increase in the size of the host cluster also leads to an increase in the probability of abnormal events in the host cluster. These abnormal events may be caused by failures in the communication network of the host cluster. The communication network of the host cluster of large-scale distributed training services (such as tens of thousands of RDMA network cards) is prone to single point failures (such as switch port jitter, optical module failure). Distributed training services are very sensitive to single point failures, and even a slight single point of packet loss will cause a very serious drop in network throughput. For example, when a hash collision occurs in the service traffic, resulting in an unbalanced network load, the average network throughput of the service will decrease. These abnormal events may also be caused by performance bottlenecks of the host, such as CPU overload.

[0034] Traditional RoCE (RDMA over Converged Ethernet) network location methods usually check the logs and abnormal indicators of the host, network card, and all switches in the network used by the business party to find abnormal events. After that, the test tool is executed in the host cluster to locate the abnormal device. This method can effectively locate abnormal devices when applied to a smaller host cluster. However, when applied to a relatively large host cluster, this method is very time-consuming.

[0035] When a performance anomaly occurs in a business, it is helpful to quickly determine whether the anomaly is caused by the communication network or the host, which helps to quickly solve the problem. However, in a large-scale host cluster using RDMA technology, it is difficult to accurately determine the problem that caused the anomaly. For example, "NCCL error code 12" often appears in distributed training business. This error code indicates that the local process of the host cannot communicate with the remote process through the RDMA communication network. This failure may be caused by a communication network failure, or it may be caused by a decrease in computing power due to a GPU frequency reduction in the host.

[0036] In traditional technology, there is a detection strategy for the Transmission Control Protocol (TCP), which sends TCP probes through the network card to measure the connectivity, packet loss rate, and end-to-end round-trip time (RTT) between hosts. However, in a host cluster based on RoCE (RDMA over Converged Ethernet) technology, TCP protocol traffic and RoCE protocol traffic are usually allocated to different traffic queues in the RDMA network card and switch, and PFC (Priority-based Flow Control) is enabled in the RoCE queue, resulting in the TPC probe being unable to detect problems specific to the RoCE queue, such as packet loss caused by RoCE network usage and improper PFC configuration. Moreover, this detection strategy for the TCP protocol cannot accurately determine whether the packet loss is caused by a host failure or a communication network failure.

[0037] In addition, this detection strategy for the TCP protocol determines the RTT based on the time of the host's application layer. However, the RTT determined based on the application layer time includes the processing delay of the hosts at both ends of the communication and the network delay of the communication network. When an RTT anomaly occurs, it may be caused by the processing delay of the host or the network delay of the communication network. Therefore, this traditional detection strategy cannot accurately determine the cause of the delay. Furthermore, the traditional detection strategy cannot accurately determine whether the abnormal event will affect the business, and thus cannot give the processing priority of the abnormal event, which is not conducive to the operation and maintenance personnel to formulate solutions.

[0038] In view of this, according to an embodiment of the present disclosure, an improved scheme for host detection is provided. According to the scheme of the embodiment of the present disclosure, in response to the detection host receiving the detection task, a detection message is sent to the target host through the network card of the detection host; first operation information related to the sending of the detection message is added to the completion queue of the network card of the detection host, and the first operation information at least includes the sending time of the detection message; in the case of receiving at least one reply message from the target host via the network card of the detection host, second operation information related to the reception of at least one reply message is added to the completion queue of the network card of the detection host, and the second operation information indicates the receiving time of at least one reply message; and based on the first operation information and the second operation information, a first detection result is determined, and the first detection result indicates the network delay of the communication network between the detection host and the target host.

[0039] In this way, the sending time of the detection message and the receiving time of the response message can be obtained from the completion queue of the network card of the detection host. The sending time and the receiving time are respectively the time when the network card of the detection host sends the detection message and the time when the response message is received. Based on the sending time and the receiving time, the network delay of the communication network can be accurately determined, which is helpful to accurately determine the cause of the abnormality when an abnormal event occurs in the host cluster.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] Figure 2 FIG. 2 is a flowchart of a process 200 for host detection according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 The embodiments are described with reference to the environment 100 of FIG. 200. The process 200 may be implemented in Figure 1 At the host 130 or host 150, the process 200 will be described below by taking the process 200 being implemented at the host 130 as an example. In this case, the host 130 is the detection host, and the network card 131 of the host 130 is the network card of the detection host.

[0042] In block 210 of process 200 , in response to receiving the detection task, host 130 (ie, the detection host) sends a detection message to a target host (eg, host 150 ) through network card 131 (ie, the network card of the detection host).

[0043] For example, Figure 3 As shown, Figure 3 A schematic diagram of an example architecture 300 for host detection according to some embodiments of the present disclosure is shown. The host 130 may be deployed with an agent 310, and the agent 310 may receive a detection task from the controller 110. The detection task may indicate a transmission path of the detection message, and the transmission path may include a host and a network card that send the detection message, and may also indicate a host and a network card that receive the detection message. In response to receiving the detection task, the agent 310 may add a request to send the detection message to the task queue 320 of the network card 131 based on the detection task. The network card 131 is used to send the detection message to the target host based on the sending request taken out from the task queue 320. Alternatively or additionally, a UD QP (Unreliable Datagram Queue Pair) may be constructed as a task queue for each network card 131 of the host 130 by, for example, the agent 310. Using UD QP to implement the sending of the detection message can reduce the consumption of the connection resources of the network card 131, and because the process of establishing the communication connection is omitted, it is conducive to improving the accuracy of the detection result.

[0044] In some embodiments, the agent 310 may trigger the sending operation of the detection message from the application layer of the host 130. For example, the agent 310 may call the application program interface of the host 130 from the application layer to add a request to send the detection message to the task queue 320 of the network card 131.

[0045] In block 220 of process 200, the network card 131 of the host 130 may add first operation information related to the sending of the probe message to its completion queue. The first operation information at least includes the sending time of the probe message. Of course, the first operation information is not limited to the sending time, and the first operation information may also include, for example, the sending status of the probe message, address information, etc.

[0046] To illustrate the process of adding the first operation message to the completion queue by the network card 131, the following will be combined with Figure 4 For explanation. Figure 4 As shown, Figure 4 A flow chart of a signaling flow 400 for user host detection according to an embodiment of the present disclosure is shown. The signaling flow 400 involves a host 130 and a host 150, and the host 130 includes an RDMA network card 131 and a processor 132. The host 150 includes an RDMA network card 151 and a processor 152. In the signaling flow 400, the host 130 is a detection host, and the host 150 is a target host. It should be noted that the host 130 and the host 150 can be different hosts in a host cluster, or can be to the same host in a host cluster. The agent 310 here can be deployed on the host 130 and can be run by the processor 132 of the host 130.

[0047] As shown in the signaling flow 400, the processor 132 running the agent 310 can, in response to receiving the detection task, add (401) a request to send a detection message to the completion queue of the RDMA network card 131 to trigger the sending operation of the detection message. The RDMA network card 131 can take out the request to send the detection message from the task queue 320, and send (402) a detection message that complies with the RDMA protocol to the host 150 based on the sending request. In response to the completion of the detection message sending, the RDMA network card 131 adds the first operation information to the completion queue of the RDMA network card 131 itself. The processor 132 can take out (403) the first operation information from the completion queue, and determine the sending time T2 of the detection message by parsing the first operation information. For example, the RDMA network card 131 can add a completion queue element (Completion Queue Element, CQE) to the completion queue. The CQE may include a timestamp, a sending status, etc. The timestamp can be used to mark the time when the RDMA network card 131 sends the detection message. The processor 132 may take the CQE from the completion queue and parse the timestamp from the CQE.

[0048] In block 230 of process 200, when the host 130 receives at least one reply message from the target host (e.g., host 150) via the network card 131, the host 130 adds second operation information related to the reception of at least one reply message to the completion queue of the network card 131, and the second operation information indicates the reception time of at least one reply message. Here, the network card 131 may receive one reply message from the target host, may receive multiple reply messages from the target host, or may not receive a reply message from the target host.

[0049] In the case where the detection host does not receive the detection message from the target host, in some embodiments, the host 130 may generate an abnormal detection result (sometimes referred to as the fourth detection result in the text) in response to not receiving a response message fed back by the target host within a predetermined time after the detection message is sent. The abnormal detection result indicates that the communication between the host 130 and the target host is abnormal.

[0050] In some embodiments, the host 130 may receive a first response message sent by the target host in response to receiving the detection message, and a second response message sent by the target host in response to sending the first response message, via the network card 131. Afterwards, the host 130 may use the network card 131 to add the second operation information of the first response message and the second operation information of the second response message to the completion queue.

[0051] Exemplarily, as shown in the signaling flow 400, the RDMA network card 151 of the host 150 can write the probe message into the receiving buffer area of ​​the host 150 in response to receiving the probe message from the RDMA network card 131. The RDMA network card 151 can add a CQE to the completion queue of the RDMA network card 151 in response to the completion of writing the probe message. The reception time of the probe message can be recorded as T3, and the CQE includes a timestamp identifying the reception time T3. The processor 152 of the host 150 can take out (404) the CQE from the completion queue of the RDMA network card 151, and the processor 152 can determine that the probe message is received by parsing the CQE. In response to receiving the probe message, the processor 152 can add (405) a request to send the first reply message to the task queue of the RDMA network card 151.

[0052] The RDMA network card 151 can take out the sending request of the first reply message from its own task queue, and send (406) the first reply message to the host 130. In response to the completion of sending the first reply message, the RDMA network card 151 can add a CQE to its own completion queue. The sending time of the first reply message can be recorded as T4, and the CQE includes a timestamp identifying T4. The processor 152 can take out (407) the CQE from the completion queue of the RDMA network card 151, and then the processor 152 adds (408) the sending request of the second reply message to the task queue of the RDMA network card 151 in response to determining that the sending of the first reply message is completed. The RDMA network card 151 takes out the sending request of the second reply message from its own task queue, and sends (409) the second reply message to the host 130. The sending time of the second reply message can be recorded as T5.

[0053] At block 240 of process 200, host 130 determines a first probing result based on the first operation information and the second operation information. The first probing result indicates a network delay of a communication network between the probing host and the target host.

[0054] Exemplarily, as shown in the signaling flow 400, the RDMA network card 131 of the host 130 writes the first response message into the receiving buffer area of ​​the host 130 in response to receiving the first response message, and adds a CQE to the completion queue of the RDMA network card 131 in response to the completion of writing the first response message. The reception time of the first response message can be recorded as T6. The processor 132 takes out (410) the CQE from the completion queue of the RDMA network card 131, and can obtain a timestamp indicating the reception time T6 of the first response message. After receiving the second response message, the RDMA network card 131 also adds a CQE to its own completion queue. The reception time of the second response message can be recorded as T7. The processor 132 parses (411) the CQE to obtain a timestamp indicating the reception time T7 of the second response message.

[0055] The processing process of the host 150 responding to the detection message includes the RDMA network card 151 adding a CQE to its own completion queue, the processor 152 parsing the CQE, the processor 152 adding a send request to the task queue of the RDMA network card 151, and the RDMA network card 151 sending a first reply message. The processing process of the host 150 responding to the sending of the first reply message includes the RDMA network card 151 adding a CQE to the completion queue of its own completion, the processor 152 parsing the CQE, the processor 152 adding a send request to the task queue of the RDMA network card 151, and the RDMA network card 151 sending a second reply message. Therefore, the processing process of the host 150 responding to the detection message is similar to the processing process of responding to the sending of the first reply message. The processing delay of the host 150 from receiving the detection message to sending the first reply message is close to the processing time of the host from sending the first reply message to sending the second reply message. That is, the time difference between T4 and T3 is close to the time difference between T5 and T4. Since the time difference between T7 and T6 is close to the time difference between T5 and T4, it can be deduced that T7-T6≈T4-T3. In view of this, the host 130 can determine the second detection result based on the reception time of the first response message and the reception time of the second response message. The second detection result indicates the processing delay of the target host (that is, T4-T3). In this way, not only the failure of the host and the communication network 140 can be detected, but also the performance defects of the destination host can be detected.

[0056] In some embodiments, the host 130 can determine the RTT of the communication network 140 between the host 130 and the host 150 based on the sending time T2 of the detection message, the receiving time T6 of the first response message, and the receiving time T7 of the second response message. Specifically, the RTT of the communication network 140≈(T6-T2)-(T7-T6). In this way, the processing delay of the host 130 and the processing delay of the host 150 can be eliminated, and the network delay of the communication network 140 can be accurately determined without performing clock synchronization of the transmitting and receiving end network cards.

[0057] In some embodiments, the host 130 may also obtain the trigger time of sending the application layer trigger detection message, and obtain the incoming time of at least one response message delivered to the application layer of the host 130 when at least one response message is received from the target host via the network card of the detection host. Based on the trigger time of the detection message, the sending time of the detection message, the receiving time of the at least one response message, and the incoming time of the at least one response message. A third detection result is determined, and the third detection result indicates the processing delay of the host 130. In this way, without the need to synchronize the clock between the processor 132 and the network card 131 of the host 130, the processing delay of the host 130 itself can be accurately determined, and then the performance defect of the host 130 itself can be determined.

[0058] Exemplarily, the host 130 may determine the time T1 when the processor 152 adds a request to send a probe message to the task queue of the RDMA network card 131, and use T1 as the trigger time of the probe message. The host 130 may also determine the time T8 when the processor 152 completes parsing the second operation information (i.e., CQE) of the first response message, and use T8 as the incoming time when the first response message is delivered to the application layer of the host 130. On this basis, the processing delay of the host 130 is equal to (T8-T1)-(T6-T2).

[0059] In some embodiments, the detection task includes a first detection task, the first detection task indicates a first transmission path of the detection message, and in the first transmission path, the network card of the detection host is connected to the network card of the target host through a single network node. The network node here can be a switch 141 in the communication network 140. Since the first detection path only includes the network card of the detection host, the single network node and the network card of the target host, the fault of the single network node or the single network card is located based on the detection result of the first detection task. Specifically, if multiple first detection tasks pointing to the network card of the same target host are abnormal, it indicates that the network card of the target host may be faulty. If multiple first detection tasks passing through the single network node are abnormal, it indicates that the single network node may be faulty. By locating the fault of a single network card or a single network node, when an abnormal event occurs in the host cluster, the abnormal event caused by the communication network 140 can be eliminated, and the specific network node or specific network card that caused the abnormal event can be accurately located.

[0060] It is understandable that, for each network card in the host cluster, by controlling the time granularity of the detection message, the running state of the network card can be determined when a certain number of detection messages point to the network card. Further, the running state of each network card in the host cluster can be determined.

[0061] In some embodiments, the detection task includes a second detection task, the second detection task indicates a second transmission path of the detection message, and in the second transmission path, the network card of the detection host is connected to the network card of the target host through multiple network nodes. Since the second detection path passes through multiple network nodes, the network status of the communication network 140 can be detected to the greatest extent. For example, the first detection task and the second detection task can be combined. When the first detection task is used to exclude abnormal events caused by the network card, it can be determined that the abnormal event is caused by the communication network 140.

[0062] In some embodiments, the detection task includes a third detection task, and the third detection task includes a third transmission path of the detection message, and the third transmission path is the transmission path of the business traffic of the target business. The target business here may include various businesses performed by the host cluster, such as distributed training tasks, etc. By sending the detection message according to the transmission path of the business traffic of the target business, the operating status of the host related to the target business and the network node related to the target business in the communication network 140 can be determined. If the detection result of the third detection task indicates an abnormality, there is a greater probability that the abnormal event will affect the target business, and can provide guidance for operation and maintenance personnel.

[0063] In some embodiments, the host 130 can obtain the five-tuple information of the target path in response to establishing a communication connection with the target host based on the target business. The five-tuple information here includes the source IP address, the destination IP address, the source port, the destination port and the protocol number. Afterwards, the host 130 can construct a detection message that matches the service flow of the target business based on the five-tuple information, and periodically send the constructed detection message to the target host through the network card 131. In this way, during the execution of the target business, the operating status of the host, network card and network node related to the target business can be periodically detected. The communication connection here can be a communication connection based on the RDMA protocol.

[0064] Alternatively or additionally, the host 130 may also stop sending the detection message to the target host in response to the host 130 disconnecting the communication connection with the target host. When the communication connection between the host 130 and the target host is disconnected, the target service stops or the target service stops using the transmission path to transmit service traffic, and there is no need to detect the transmission path. Stopping sending the detection message to the target host can avoid unnecessary occupation of network resources.

[0065] Exemplarily, the host 130 can establish an RC QP (Reliable Connection Queue Pair) communication connection through the modify_qp function in the kernel, and can terminate the RC QP communication connection through the destroy_qp function. If the modify_qp function is called, it indicates that the host 130 is about to establish an RC QP communication connection, and if the destroy_qp function is called, it indicates that the host 130 is about to terminate the RC QP communication connection.

[0066] The five-tuple information of the RC QP communication connection to be established can be obtained by, for example, an extended Berkeley Packet Filter (eBPF) in response to the call of the modify_qp function. Then, a detection message matching the RCQP communication connection can be constructed based on the obtained five-tuple information, and the operating status of the hosts and the communication link at both ends of the RC QP communication connection can be detected. For example, Figure 3 As shown, each time the modify_qp function is called, quintuple information 331-1, 331-2, 331-3, ..., 331-N, etc. can be obtained respectively. The quintuple information 331-1, 331-2, 331-3, ..., 331-N can be added to the quintuple information set 340, and a detection message can be constructed based on the quintuple information in the quintuple information set 340, so as to achieve comprehensive coverage of hosts and network nodes related to the target service.

[0067] When the destroy_qp function is called, the eBPF may also respond to the destroy_qp function to obtain the quintuple information of the RC QP communication connection to be terminated, stop constructing the detection message matching the RC QP communication connection, and stop detecting the RC QP communication connection. For example, the corresponding quintuple information may be removed from the quintuple information set 340.

[0068] In some embodiments, Figure 3 As shown, the host 130 can also use the agent 310 to send the detection result 350 to the analyzer, and the analyzer 350 analyzes the detection result 350 to determine the operating status of the host cluster. For example, the processing delay of the host, the network delay of the communication network 140, the operating status of the network card, the operating status of each network node of the communication network 140, etc. are determined.

[0069] To sum up, according to the embodiments of the present disclosure, the sending time of the detection message and the receiving time of the response message can be obtained from the completion queue of the network card of the detection host, and the sending time and the receiving time are respectively the time when the network card of the detection host sends the detection message and the time when the response message is received. Based on the sending time and the receiving time, the network delay of the communication network can be accurately determined, thereby providing a network delay detection method with low resource consumption and high detection accuracy.

[0070] Figure 5 A schematic structural block diagram of an apparatus 500 for host detection according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the host 130. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0071] like Figure 5As shown, the device 500 includes a sending module 510, a first adding module 520, a second adding module 530 and a determining module 540. The sending module 510 is configured to send a detection message to the target host through the network card of the detection host in response to the detection host receiving the detection task. The first adding module 520 is configured to add first operation information related to the sending of the detection message to the completion queue of the network card of the detection host, and the first operation information at least includes the sending time of the detection message. The second adding module 530 is configured to add second operation information related to the reception of at least one reply message to the completion queue of the network card of the detection host when at least one reply message is received from the target host via the network card of the detection host, and the second operation information indicates the receiving time of at least one reply message. The determining module 540 is configured to determine a first detection result based on the first operation information and the second operation information, and the first detection result indicates the network delay of the communication network between the detection host and the target host.

[0072] In some embodiments, the detection task includes a first detection task, the first detection task indicates a first transmission path of the detection message, in which the network card of the detection host is connected to the network card of the target host through a single network node.

[0073] In some embodiments, the detection task includes a second detection task, the second detection task indicates a second transmission path of the detection message, and in the second transmission path, the network card of the detection host is connected to the network card of the target host through multiple network nodes.

[0074] In some embodiments, the detection task includes a third detection task, the third detection task includes a third transmission path of the detection message, and the third transmission path is a transmission path of the service traffic of the target service.

[0075] In some embodiments, the sending module 510 is further configured to: in response to the detection host establishing a communication connection with the target host based on the target service, obtain five-tuple information of the target path, the five-tuple information including the source IP address, destination IP address, source port, destination port and protocol number; based on the five-tuple information, construct a detection message that matches the service traffic of the target service; and periodically send the constructed detection message to the target host through the network card of the detection host.

[0076] In some embodiments, the apparatus 500 may further include: a stopping module configured to stop sending the detection message to the target host in response to the detection host disconnecting the communication connection with the target host.

[0077] In some embodiments, the sending module 510 is further configured to: in response to the detection host receiving a detection task, add a sending request for the detection message to the task queue of the network card of the detection host based on the detection task; and use the network card of the detection host to send the detection message to the target host based on the sending request taken out from the task queue.

[0078] In some embodiments, the second adding module 530 is further configured to: receive, via the network card of the detection host, a first reply message sent by the target host in response to the reception of the detection message, and a second reply message sent by the target host in response to the sending of the first reply message; and use the network card of the detection host to add the second operation information of the first reply message and the second operation information of the second reply message to the completion queue.

[0079] In some embodiments, the determination module 540 is further configured to determine a second detection result based on a reception time of the first response message and a reception time of the second response message, the second detection result indicating a processing delay of the target host.

[0080] In some embodiments, the determination module 540 is further configured to: obtain the trigger time of the sending of the detection message triggered by the application layer of the detection host; when at least one reply message is received from the target host via the network card of the detection host, obtain the incoming time of at least one reply message passed to the application layer of the detection host; and determine a third detection result based on the trigger time of the detection message, the sending time of the detection message, the receiving time of at least one reply message and the incoming time of at least one reply message, the third detection result indicating the processing delay of the detection host.

[0081] In some embodiments, the determination module 540 is further configured to generate a fourth detection result in response to the detection host not receiving a response message fed back by the target host within a predetermined time after the detection message is sent, and the fourth detection result indicates that the communication between the detection host and the target host is abnormal.

[0082] The units and / or modules included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 500 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0083] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 1 Host 130 or Figure 5 The apparatus 500 may be implemented as or included in one or more electronic devices 600 .

[0084] like Figure 6 As shown, the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 600.

[0085] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 600.

[0086] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0087] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0088] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0089] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0090] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0093] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0094] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for host detection, comprising: In response to the detection host receiving the detection task, sending a detection message to the target host through the network card of the detection host; Adding first operation information related to the sending of the detection message to the completion queue of the network card of the detection host, wherein the first operation information at least includes the sending time of the detection message; In the case where at least one response message is received from the target host via the network card of the detection host, second operation information related to the reception of the at least one response message is added to the completion queue of the network card of the detection host, wherein the second operation information indicates a reception time of the at least one response message; as well as Based on the first operation information and the second operation information, a first detection result is determined, the first detection result indicating a network delay of a communication network between the detection host and the target host.

2. The method according to claim 1, wherein the detection task includes a first detection task, the first detection task indicates a first transmission path of the detection message, and in the first transmission path, the network card of the detection host is connected to the network card of the target host through a single network node.

3. The method according to claim 1, wherein the detection task includes a second detection task, the second detection task indicates a second transmission path of the detection message, and in the second transmission path, the network card of the detection host is connected to the network card of the target host through multiple network nodes.

4. The method according to claim 1, wherein the detection task includes a third detection task, the third detection task includes a third transmission path of the detection message, and the third transmission path is a transmission path of the service traffic of the target service.

5. The method according to claim 4, wherein sending a detection message to a target host through a network card of the detection host comprises: In response to the detection host establishing a communication connection with the target host based on the target service, acquiring quintuple information of the target path, the quintuple information including a source IP address, a destination IP address, a source port, a destination port, and a protocol number; Based on the five-tuple information, construct a detection message matching the service flow of the target service; as well as The constructed detection message is periodically sent to the target host through the network card of the detection host.

6. The method according to claim 5, further comprising: In response to the detection host disconnecting the communication connection with the target host, stopping sending the detection message to the target host.

7. The method according to claim 1, wherein sending a detection message to a target host through a network card of the detection host comprises: In response to the detection host receiving the detection task, adding a request for sending the detection message to a task queue of a network card of the detection host based on the detection task; as well as The network card of the detection host is used to send the detection message to the target host based on the sending request taken out from the task queue.

8. The method according to claim 1, wherein adding the second operation information to the completion queue comprises: Receiving, via the network card of the detection host, a first response message sent by the target host in response to receiving the detection message, and a second response message sent by the target host in response to sending the first response message; as well as The network card of the detection host is used to add the second operation information of the first response message and the second operation information of the second response message to the completion queue.

9. The method according to claim 8, further comprising: Based on the reception time of the first response message and the reception time of the second response message, a second detection result is determined, and the second detection result indicates a processing delay of the target host.

10. The method according to claim 1, further comprising: Obtaining the trigger time at which the application layer of the detection host triggers the sending of the detection message; In the case where at least one response message is received from the target host via the network card of the detection host, acquiring the incoming time of the at least one response message being delivered to the application layer of the detection host; as well as A third detection result is determined based on the triggering time of the detection message, the sending time of the detection message, the receiving time of the at least one reply message, and the incoming time of the at least one reply message, and the third detection result indicates a processing delay of the detection host.

11. The method according to claim 1, further comprising: In response to the detection host not receiving a response message fed back by the target host within a predetermined time after the detection message is sent, a fourth detection result is generated, and the fourth detection result indicates that the communication between the detection host and the target host is abnormal.

12. A device for host detection, comprising: A sending module, configured to send a detection message to a target host through a network card of the detection host in response to the detection host receiving the detection task; A first adding module is configured to add first operation information related to the sending of the detection message to the completion queue of the network card of the detection host, wherein the first operation information at least includes the sending time of the detection message; A second adding module is configured to, when at least one reply message is received from the target host via the network card of the detection host, add second operation information related to the reception of the at least one reply message to the completion queue of the network card of the detection host, wherein the second operation information indicates a reception time of the at least one reply message; as well as The determination module is configured to determine a first detection result based on the first operation information and the second operation information, wherein the first detection result indicates a network delay of a communication network between the detection host and the target host.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Citation Information

Cited By

  • Time delay determination method and device based on intelligent agent and computer program product

    CN120186056A

  • Agent-based delay determination method, device, and computer program product

    CN120186056B

  • Method and device for monitoring and diagnosing a network of intelligent centers based on programmable switches

    CN122891494A

  • Method and Device for Network Monitoring and Diagnosis of Intelligent Computing Centers Based on Programmable Switches

    CN122891494B