Network monitoring method and device of server cluster, electronic equipment and storage medium

Abstract

This disclosure relates to the field of computer technology and discloses a method, apparatus, electronic device, and storage medium for network monitoring of server clusters. The method includes acquiring network topology information of the server cluster to be monitored, whereby the network topology information characterizes the connection relationships between network devices in the server cluster; generating a first type of in-band network telemetry probe based on the network topology information to obtain the network quality of ports in the network devices; generating a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic to obtain the service traffic telemetry path, wherein the transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe; and querying the port network quality based on the ports in the service traffic telemetry path to determine the network monitoring result of the server cluster to be monitored. This method ensures the real-time performance and accuracy of network monitoring.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to network monitoring methods, devices, electronic equipment, and storage media for server clusters. Background Technology

[0002] With the development of network applications, such as large model training and distributed storage, the collaborative operation of massive servers in server clusters places higher demands on network stability. It is necessary not only to ensure efficient network transmission but also to guarantee high network quality. Therefore, accurate network monitoring capabilities are required. Summary of the Invention

[0003] In view of this, the present disclosure provides a network monitoring method, apparatus, electronic device and storage medium for server clusters to solve the network monitoring problem.

[0004] In a first aspect, this disclosure provides a network monitoring method for a server cluster, the method comprising:

[0005] Obtain the network topology information of the server cluster to be monitored, wherein the network topology information is used to characterize the connection relationship between network devices in the server cluster to be monitored;

[0006] Based on the network topology information, a first type of in-band network telemetry probe is generated to obtain the network quality of the ports in the network device.

[0007] A second type of in-band network telemetry probe is generated based on the traffic quintuple information of the current service traffic to obtain the service traffic telemetry path. The transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe.

[0008] Based on the ports in the telemetry path of the service traffic, the network quality of the ports is queried to determine the network monitoring results of the server cluster to be monitored.

[0009] Secondly, this disclosure provides a network monitoring device for a server cluster, the device comprising:

[0010] The network topology information acquisition module is used to acquire the network topology information of the server cluster to be monitored, wherein the network topology information is used to characterize the connection relationship between network devices in the server cluster to be monitored.

[0011] A first-type in-band network telemetry probe generation module is used to generate a first-type in-band network telemetry probe based on the network topology information in order to obtain the network quality of the ports in the network device.

[0012] The second type of in-band network telemetry probe generation module is used to generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic in order to obtain the service traffic telemetry path. The transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe.

[0013] The network quality query module is used to query the network quality of ports based on the ports in the service traffic telemetry path, and determine the network monitoring results of the server cluster to be monitored.

[0014] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the network monitoring method for a server cluster described in the first aspect or any corresponding embodiment.

[0015] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the network monitoring method for a server cluster described in the first aspect or any of its corresponding embodiments.

[0016] The network monitoring method for server clusters provided in this disclosure generates a first type of in-band network telemetry probe based on the network topology information of the server cluster to be monitored. This probe then performs network quality detection on the ports of network devices within the server cluster. Specifically, the detection by the first type of in-band network telemetry probe is independent of actual service traffic. Next, a second type of in-band network telemetry probe is generated based on the traffic quintuple information of the current service traffic to detect the service traffic telemetry path. Since the second type of in-band network telemetry probe is related to the actual service traffic, the service traffic telemetry path of the current service traffic can be obtained through it. The network quality of each port can already be obtained from the detection results of the first type of in-band network telemetry probe. Since the service traffic telemetry path includes multiple ports, combining the network quality of multiple ports allows for accurate determination of the network quality corresponding to each service traffic telemetry path, thereby ensuring the real-time performance and accuracy of network monitoring. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1This is a schematic diagram of a fault location system according to an embodiment of the present disclosure;

[0019] Figure 2 This is a flowchart illustrating a network monitoring method for a server cluster according to an embodiment of the present disclosure;

[0020] Figure 3 This is a flowchart illustrating a network monitoring method for a server cluster according to an embodiment of the present disclosure;

[0021] Figure 4 This is a schematic diagram of a network topology according to an embodiment of the present disclosure;

[0022] Figures 5a-5d This is a schematic diagram of the update of the heap to be screened according to an embodiment of the present disclosure;

[0023] Figure 6 This is a flowchart illustrating a network monitoring method for a server cluster according to an embodiment of the present disclosure;

[0024] Figure 7 This is a schematic diagram of a fault path according to an embodiment of the present disclosure;

[0025] Figure 8 This is a structural block diagram of a network monitoring device for a server cluster according to an embodiment of the present disclosure;

[0026] Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] In related technologies, active network measurement is used to locate network faults in server clusters. However, when a network failure occurs, it is difficult to directly pinpoint the specific port where the fault occurred. Traffic sampling is required to locate port-level connectivity issues. Furthermore, when multiple failures occur simultaneously, the massive amounts of data analyzed and processed simultaneously also complicate the localization of network problems.

[0034] In other related technologies, network fault location for server clusters is achieved through passive network measurement. However, while this method can achieve switch-based network monitoring, it cannot obtain instantaneous network status information when data packets pass through, and therefore cannot directly correlate service traffic issues with network problems. Specifically, it troubleshoots by comparing whether the time points of network faults coincide, but it cannot directly determine whether the fluctuation in service traffic is caused by this problem.

[0035] Other related technologies employ in-band network telemetry (INT)-based network measurement schemes. These schemes involve inserting port-level telemetry information into probe packets via switches. This includes information such as the ingress and egress ports, queue depth, and hop-by-hop forwarding delays of the data packets, enabling the detection of more granular network conditions. However, since INT only provides a basic source domain, it is difficult to directly achieve full-port and full-traffic network awareness. Furthermore, the number of telemetry probes is directly proportional to the size and volume of traffic, leading to significant bandwidth consumption and server resource overhead.

[0036] Among them, network measurement methods based on INT cannot simultaneously meet the network awareness requirements of full ports and full traffic. For example, network telemetry schemes combined with segment routing (SR) technology achieve lightweight full port telemetry with non-overlapping coverage through path planning algorithms, but it is difficult to meet the requirements of Remote Direct Memory Access (RDMA) networks for full service traffic measurement. Another example is network telemetry schemes based on data compression, which achieve the requirement of full traffic measurement by statistically analyzing network traffic, but cannot collect network information such as queue depth and hop-by-hop time delay.

[0037] Because INT-based network measurement methods require switches to insert telemetry data and transmit it in the data plane, they consume significant network bandwidth and CPU resources, making it difficult to lightweightly address the needs of full-port, full-traffic network measurement. Related INT technologies need to simultaneously support fine-grained network monitoring and efficient network congestion control; relying solely on switch-based network telemetry information reporting leads to excessive southbound overhead. Furthermore, network measurement schemes struggle to directly pinpoint the specific port where a network fault occurs, requiring network administrators to spend considerable time analyzing massive amounts of network telemetry data to troubleshoot the problem.

[0038] Based on this, embodiments of this disclosure provide a network monitoring method for server clusters. This method, without affecting server and network performance, can detect the network status of all ports, measure real-time network information of all traffic, and accurately locate the source of network problems. Furthermore, the network monitoring method for server clusters provided in this disclosure is based on a network monitoring system. This system is a lightweight network measurement architecture that provides a highly real-time, fine-grained, full-port, full-traffic measurement solution for RDMA networks.

[0039] The network monitoring system provided in this disclosure embodiment, such as Figure 1As shown, the INT controller, INT analyzer, and INT database are deployed in a distributed server cluster to ensure that the telemetry system will not malfunction due to a single point of failure and to enhance the data processing capabilities of these components. INT agents and INT collectors are deployed on each server, ensuring that each device supports INT probe sending and receiving capabilities. The RDMA network interface card (NIC) automatically sends INT probe packets to the appropriate network application by recognizing the packet header. The data plane needs to support hash-based equal-cost multi-path routing (ECMP) technology and INT transmission functionality. To achieve a versatile, robust, and scalable in-band full-network telemetry system, the INT agent actively queries the INT controller to send INT probes, avoiding the INT controller's active sending of INT probes, thus achieving a loosely coupled design.

[0040] Specifically, the INT controller, based on network topology information, employs a full-port, full-traffic network measurement scheme. It achieves lightweight, high-performance network-wide telemetry by filtering the five-tuple information of probe packets to obtain the target probe packet five-tuple. The INT agent, used in the INT controller-based telemetry scheme, periodically sends INT probes according to the set probe sending frequency and other telemetry requirements. The RDMA network card, by recognizing the packet header of the INT probe packet, transmits the INT probe packet to the designated network application. The INT collector, deployed on each server, collects telemetry data directly written to memory by the RDMA network card. The INT collector periodically forwards the telemetry data to the INT analyzer; that is, the data is directly written to memory by the RDMA network card without CPU processing, thus avoiding additional overhead in data collection. The INT analyzer is responsible for parsing the received telemetry data and storing it in the INT database as needed. Because it is deployed on a server cluster, the INT analyzer is highly efficient in parsing large amounts of telemetry data. The INT database is responsible for storing telemetry data and can provide network telemetry information in real time. It separates and stores telemetry data into port data obtained by the first type of in-band network telemetry probe and path data obtained by the second type of in-band network telemetry probe.

[0041] When the network monitoring system is in operation, it performs network measurements at a first frequency on each port using a first-type in-band network telemetry probe, acquires the telemetry paths of all service traffic using a second-type in-band network telemetry probe corresponding to the second frequency of the service traffic, performs lightweight full-network measurements using a data plane telemetry data acquisition scheme, and achieves high-performance network telemetry data processing based on RDMA network cards and network telemetry data. Finally, it achieves accurate network monitoring by combining the service traffic telemetry paths obtained from the second-type in-band network telemetry probe with the port network quality obtained from the first-type in-band network telemetry probe.

[0042] According to an embodiment of this disclosure, a network monitoring method for a server cluster is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] This embodiment provides a network monitoring method for a server cluster, which can be used in the aforementioned servers. Figure 2 This is a flowchart of a network monitoring method for a server cluster according to an embodiment of this disclosure, such as... Figure 2 As shown, the process includes the following steps:

[0044] Step S201: Obtain the network topology information of the server cluster to be monitored.

[0045] Among them, network topology information is used to characterize the connection relationships between network devices in the server cluster to be monitored.

[0046] The server cluster to be monitored includes multiple network devices, including but not limited to switches, servers, and network interface cards (NICs) within the servers. For example, servers can be connected via switches, and multi-level connections can be deployed between switches according to network deployment requirements. Each server has at least one NIC, and different NICs within the same server can connect to the same network device or to different network devices, and so on.

[0047] Network topology information is obtained after network deployment is complete through the connections between network devices. Network topology information represents the connectivity between network devices, and each network device is represented by its own corresponding identifier to distinguish between different network devices.

[0048] Step S202: Generate a first type of in-band network telemetry probe based on network topology information to obtain the network quality of ports in network devices.

[0049] A single network device includes multiple physical ports. During communication between these devices, in addition to determining the network addresses of the communicating parties, it is also necessary to obtain their respective physical ports. That is, network detection information is obtained through probing using a first-type in-band network telemetry probe. Further analysis based on this network detection information yields the network quality of the port, which corresponds to the physical port's quality. Since network topology information represents the connections of all network devices in the server cluster to be monitored, a first-type in-band network telemetry probe is generated to probe each port of the network device and obtain the network quality of each port. It should be noted that the methods for determining the network quality of each port include, but are not limited to, using the network device's own monitoring results or combining relevant log aggregation results, etc. No limitations are placed on the method of determining network quality here. For the first-type in-band network telemetry probe, each port adds its own network quality to the probe's detection data, thereby obtaining the network detection information of each port in the network device, which is then further analyzed to obtain the port's network quality.

[0050] For the generation method of the first type of in-band network telemetry probe, it can be randomly generated based on network topology information to cover all ports; or, based on random generation, it can be filtered to ensure that each port can be covered multiple times, thereby avoiding a large number of duplicate or redundant probes.

[0051] Step S203: Generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic to obtain the service traffic telemetry path.

[0052] Among them, the transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe.

[0053] Methods for obtaining traffic 5-tuple information from current service traffic include, but are not limited to, methods based on the extended Berkeley Packet Filter (eBPF). After obtaining the traffic 5-tuple information of the current service traffic, a second type of in-band network telemetry probe is generated. The traffic 5-tuple information includes the source IP (Internet Protocol) address, destination IP address, source port, destination port, and protocol number. Once the traffic 5-tuple information is determined, the communicating parties corresponding to the service traffic can be identified. Based on this, a second type of in-band network telemetry probe is generated based on the traffic 5-tuple information to obtain the service traffic telemetry path of the current service traffic.

[0054] Since the first type of in-band network telemetry probe is independent of service traffic and has no impact on it, and the second type of in-band network telemetry probe, which is related to service traffic, has a lower transmission frequency than the first type, network telemetry ensures that it does not affect normal service traffic. For example, the first type of in-band network telemetry probe has a transmission frequency of 100PPS, while the second type has a transmission frequency of 1PPS, achieving lightweight full-traffic and full-port network monitoring with extremely low network bandwidth resource consumption (e.g., 200Kb / s).

[0055] Step S204: Query the network quality of ports based on the ports in the business traffic telemetry path to determine the network monitoring results of the server cluster to be monitored.

[0056] The service traffic telemetry path passes through at least one network device, and each network device includes an ingress port and an egress port. By analyzing the service traffic telemetry path, the ports corresponding to the service traffic telemetry path are obtained. Combined with the port network quality obtained in step S202 above, the network quality of each port in the service traffic telemetry path is obtained, thereby enabling full traffic and full port network monitoring and obtaining the network monitoring results of the system to be processed.

[0057] The network monitoring method for server clusters provided in this embodiment generates a first type of in-band network telemetry probe based on the network topology information of the server cluster to be monitored. This probe detects the network quality of the ports of network devices in the server cluster. Specifically, the detection of the first type of in-band network telemetry probe is unrelated to actual service traffic. Then, a second type of in-band network telemetry probe is generated based on the traffic quintuple information of the current service traffic to detect the service traffic telemetry path. Since the second type of in-band network telemetry probe is related to the actual service traffic, the service traffic telemetry path of the current service traffic can be obtained through it. The network quality of each port can already be obtained from the detection results of the first type of in-band network telemetry probe. Since the service traffic telemetry path includes multiple ports, combining the network quality of multiple ports allows for accurate determination of the network quality corresponding to each service traffic telemetry path, thereby ensuring the real-time performance and accuracy of network monitoring.

[0058] This embodiment provides a network monitoring method for a server cluster, which can be used in the aforementioned servers. Figure 3 This is a flowchart of a network monitoring method for a server cluster according to an embodiment of this disclosure, such as... Figure 3 As shown, the process includes the following steps:

[0059] Step S301: Obtain the network topology information of the server cluster to be monitored.

[0060] Network topology information is used to characterize the connectivity relationships between network devices in the server cluster to be monitored. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0061] Step S302: Generate a first type of in-band network telemetry probe based on network topology information to obtain the network quality of ports in network devices.

[0062] Specifically, step S302 includes:

[0063] Step S3021: Generate random probe packets based on network topology information to obtain the correspondence between probe packet quintuples and telemetry paths.

[0064] The source port in the random probe packet is randomly generated.

[0065] Random probe packets are generated based on network topology information. Since network topology information represents the connection relationships between network devices, it is used to form a 5-tuple to generate random probe packets. Specifically, a random probe packet includes a source IP address, a destination IP address, a source port, a destination port, and a protocol number. Because the source and destination IP addresses are fixed, the communicating parties are also fixed, the protocol number is fixed, and the destination port is specified by the network interface card (NIC), therefore, to obtain a random probe packet, only a randomly generated source port can be used to ensure that the random probe packet covers all ports of all network devices in the system being processed.

[0066] In some optional implementations, step S3021 above includes:

[0067] Step a1: Label the network devices in the server cluster to be monitored based on the network topology information.

[0068] Step a2: Based on the network cluster in which the network devices are located, all network devices are grouped to obtain the unique number of the network card in the server cluster to be monitored. The second cluster number corresponding to the network devices in the same group is different, while the other cluster numbers are the same. The second cluster number is the number of the cluster that has a connection relationship except for the top-level network device.

[0069] Step a3: Generate random probe packets for transmission between network interface cards within each group based on the unique number, so as to obtain the correspondence between the probe packet quintuple and the telemetry path.

[0070] All network devices are labeled based on network topology information, and each network interface card (NIC) of each server has a unique label. For example, combining... Figure 4As shown, the switches are labeled S0, S1, and S2 according to their distance from the nearest server, from closest to furthest. The S0 level switches include switch 0-1, switch 0-2, switch 0-3, and switch 0-4; the S1 level switches include switch 1-1, switch 1-2, switch 1-3, and switch 1-4; and the S2 level switches include switch 2-1 and switch 2-2. The lowest level servers include server1 to server4, each with two network interface cards (NICs) eth1 and eth2.

[0071] certainly, Figure 4 This is just one example and can be further expanded based on network layers. The cluster containing all network devices (including switches, servers, and network interface cards) is called the first cluster, Bigpod, excluding the top-level switch (…). Figure 4 In the three-layer network shown, the multiple interconnected clusters following S2 are called the second cluster, Minipods. Server clusters connected to the same switch are called the third cluster, Cluster. When a server has multiple network interface cards (NICs), all servers within the same Cluster are connected to the same uplink switch, meaning the network topology is equivalent. Based on this, all NICs (Eth) on all servers have unique network IDs (Bigpod id, Minipod id, Cluster id, Server id, Eth id). All ID values ​​are automatically generated based on the order in which the network topology information is obtained.

[0072] For example, Figure 4 The labels corresponding to eth1 in server1 are (Bigpod 1, Minipod 1, Cluster1, Server 1, Eth 1). Figure 4 The labels corresponding to eth1 of server3 are (Bigpod 1, Minipod2, Cluster 1, Server 1, Eth 1).

[0073] After obtaining the unique identifier of each network device, all network cards are grouped according to the following principle: network cards with the same Bigpod ID, Cluster ID, Server ID, and Eth ID, but different Minipod IDs are grouped together. For example, (Bigpod 1, Minipod 1, Cluster 1, Server 1, Eth 1) and (Bigpod 1, Minipod 2, Cluster 1, Server 1, Eth 1) are grouped together.

[0074] Taking (Bigpod 1, Minipod 1, Cluster 1, Server 1, Eth 1) as an example, random probe packets are generated using the address of this network interface as the source IP address, a random source port, and all other network interfaces in the same group as targets. The overall transmission rate of these random probe packets is limited to below 100 PPS. Because the source port is randomized, the 5-tuple information of each random probe packet is different. The data plane enables an equivalent multipath mechanism (selecting different paths based on the hash value of the 5-tuple), allowing network telemetry to detect more paths. For example, when the source port is 65534, the corresponding random probe packet passes through switch 0-1, switch 1-1, switch 2-1, switch 1-3, and switch 0-3. When the source port is 65535, the corresponding random probe packet passes through switch 0-1, switch 1-2, switch 2-2, switch 1-4, and switch 0-3. Because each server's network interface card participates in network telemetry and receives ample random probe packets, it is possible to cover all ports, including the last hop in the system under processing (the server and switch link).

[0075] To further illustrate the advantages of the aforementioned grouping, network interfaces with the same Minipod ID are grouped together during grouping. That is, probing is performed on network interfaces with the same Minipod ID, resulting in a random probe packet traversing fewer ports. For example, in the configuration (Bigpod 1, Minipod 1, Cluster 1, Server 1, Eth 1), if communicating with other Minipods within the same Minipod 1 (Bigpod 1, Minipod 1, Cluster 1, Server 2, Eth 1), the random probe packet can only obtain information about switches 0-1. However, if communicating with different Minipod IDs (Bigpod 1, Minipod 2, Cluster 1, Server 1, Eth 1), a single random probe packet can probe all switches and network ports along the entire path (switch 0-1, switch 1-1, switch 2-1, switch 1-3, switch 0-3). Furthermore, because the random source port generates different random probe packets, the resulting telemetry probes can cover more switches and ports, such as switch 0-1, switch 0-3, switch 1-1, switch 1-2, switch 1-3, switch 1-4, switch 2-1, and switch 2-2. Therefore, random probe packets across Minipod IDs can carry more telemetry information and cover more network ports.

[0076] Based on network topology information, network devices in the server cluster to be monitored are labeled, and then grouped to obtain unique IDs for the network interface cards (NICs) within the servers. Since devices in the same group are different in the second cluster but identical in the remaining clusters, the different labels of the second clusters ensure that the same random probe packet can probe as many ports as possible. Simultaneously, this grouping method ensures that all servers and NICs participate, thereby guaranteeing that all network ports are covered.

[0077] Step S3022: Based on the correspondence between the probe packet quintuple and the telemetry path, the probe packet quintuple corresponding to the random probe packet is filtered to obtain the target probe packet quintuple, so as to determine the port corresponding to the target probe packet quintuple.

[0078] Among them, the five-tuple of all target probe packets is used to cover all ports at least a preset number of times, and the preset number of times is the length of the longest telemetry path corresponding to the random probe packet.

[0079] Because of the randomness of the source port in the probe packet quintuple corresponding to the random probe packet, it is possible to ensure that as many ports as possible are covered, and the grouping method provides full coverage of all ports. Furthermore, to reduce redundant probe packet quintuples, the probe packet quintuples corresponding to the random probe packets are filtered. The filtering principle is that the target probe packet quintuples obtained after filtering must cover all ports at least a certain number of predictions, and the number of predictions is equal to the length of the longest telemetry path corresponding to the random probe packet. For example, if the length of the longest telemetry path is 5, then all target probe packet quintuples must ensure that all ports are covered at least 5 times. Of course, the length of the longest telemetry path is obtained based on network topology information.

[0080] The filtering of the probe packet quintuple corresponding to the random probe packet can be achieved by recording the number of times each port is covered after each random probe packet is sent. If the predicted number is reached, subsequent random probe packets targeting that port are discarded. Alternatively, an optimization algorithm can be used to filter the probe packet quintuple corresponding to the random probe packet to obtain the target probe packet quintuple.

[0081] In some optional implementations, step S3022 above includes:

[0082] Step b1: Construct a heap to be filtered. The heap to be filtered includes heap elements and the connection relationships between heap elements. Each heap element in the heap to be filtered corresponds one-to-one with a probe packet quintuple, and the value of the heap element is used to represent the number of effective ports that the probe packet quintuple can cover. The number of effective ports is obtained based on the telemetry path.

[0083] Step b2: Determine the probe packet quintuple corresponding to the top element of the pair to be filtered as the target probe packet quintuple, and delete the top element.

[0084] Step b3 updates the remaining heap element with the largest value to the new top heap element, thereby updating the heap elements and their values ​​in the heap to be filtered.

[0085] Step b4: Based on the updated heap to be screened, the target probe packet quintuple is determined by screening the probe packet quintuples.

[0086] The target probe packet quintuple is obtained by filtering the quintuples corresponding to the random probe packets using a heap optimization algorithm. It should be noted that the final target probe packet quintuple is multiple, not just one.

[0087] Specifically, a filter heap is constructed. The initial filter heap includes all heap elements and the connections between them. Each heap element corresponds one-to-one with a probe packet quintuple, and the value of each heap element represents the number of valid ports that the probe packet quintuple can cover. The ports that the probe packet quintuple can cover are obtained through the telemetry path corresponding to the probe packet quintuple. If a port has already been covered a preset number of times, it is considered an invalid port and is not included in the statistics.

[0088] For example, the initial heap to be screened is like Figure 5a As shown, the preset number of iterations is 2, and a total of 5 probe packet quintuples participate in the filtering, namely quintuple 1 to quintuple 5. Initially, each quintuple can cover 2 ports. The initial top element of the heap to be filtered is quintuple 1, which can cover 2 valid ports. Each time, the probe packet quintuple corresponding to the top element is selected as the target probe packet quintuple, and the remaining top element with the largest value is updated as the new top element. For example, Figure 5b As shown, Figure 5a Remove the top element of the heap and update the 5-tuple with the new top element, resulting in: Figure 5b The updated heap to be filtered is shown below. Similarly, after the heap to be filtered is updated, the coverage count of each port also needs to be updated so that subsequent filtering can continue. Through iteration, the following results are obtained: Figure 5c The pile to be screened shown and Figure 5d The pile to be screened is shown.

[0089] Each time, select the heap element with the most valid ports at the top of the heap, identify this 5-tuple as the target probe packet 5-tuple, and remove it from the top of the heap. Update the port information involved in the removed 5-tuples. If a port is covered more than twice, mark it as an invalid port and update the number of valid ports corresponding to the relevant 5-tuples. Repeat the above process until all ports are covered twice.

[0090] By using the above processing method, the target probe packet quintuples are selected, thus ensuring that the minimum number of target probe packet quintuples is used to cover all ports at least a preset number of times.

[0091] By constructing a heap to filter probe packet quintuples, the minimum number of target probe packet quintuples is obtained, minimizing the overhead of full-port network telemetry. This method significantly reduces the number of target probe packet quintuples and the amount of data to be analyzed. The method first uses random network telemetry across Minipods to obtain high-value target probe packet quintuples, and then uses a heap-optimized approach to select the fewest target probe packet quintuples to achieve full-port telemetry coverage. This method achieves lightweight full-port network measurement and can subsequently pinpoint the location of network faults.

[0092] Step S3023: Using the correspondence between the probe packet quintuple and the telemetry path, the target telemetry path corresponding to the target probe packet quintuple is obtained, so as to determine the set of target probe packet quintuples corresponding to each port in the target telemetry path.

[0093] The correspondence between the probe packet quintuple and the telemetry path is obtained through step S3021 above. After obtaining the target probe packet quintuple, the target telemetry path corresponding to the target probe packet quintuple is obtained by querying the correspondence. Accordingly, all ports corresponding to each target probe packet quintuple can be obtained. One target probe packet quintuple corresponds to multiple ports. Of course, different target probe packet quintuples may pass through the same port. Therefore, one port corresponds to multiple target probe packet quintuples. For ease of description, a set of target probe packet quintuples corresponding to the same port is generated based on the multiple target probe packet quintuples corresponding to the same port.

[0094] In some optional implementations, step S3023 above includes:

[0095] Step c1: Based on the target probe packet quintuple query correspondence, obtain the target telemetry path, so as to obtain the target probe packet quintuple corresponding to each port in the target telemetry path, and the target telemetry path includes multiple ports.

[0096] Step c2: Integrate the target probe packet quintuples corresponding to the same port to obtain the target probe packet quintuple set corresponding to each port.

[0097] Based on the correspondence between the probe packet quintuple and the telemetry path, the target telemetry path corresponding to the target probe packet quintuple is obtained. Since the target telemetry path includes multiple ports, the target probe packet quintuple corresponds to these multiple ports. Accordingly, the target probe packet quintuples corresponding to the same port are integrated to obtain the set of target probe packet quintuples corresponding to each port. This integration simply means placing multiple target probe packet quintuples into a single set for representation, without performing further processing on these multiple target probe packet quintuples.

[0098] Since there is a correspondence between the probe packet quintuple and the telemetry path, and the probe packet quintuple includes multiple ports, by integrating the target probe packet quintuple corresponding to the same port, a set of target probe packet quintuples corresponding to each port can be obtained, which can be used for subsequent fault location.

[0099] Step S3024: Generate a first type of in-band network telemetry probe based on the five-tuple set of target probe packets corresponding to each port, and obtain the network quality of each port in the network device.

[0100] Since the set of target probe packet quintuples corresponding to each port is obtained, multiple Type I in-band network telemetry probes are generated using these sets of target probe packet quintuples to obtain the network quality of each port. Specifically, each Type I in-band network telemetry probe corresponds one-to-one with a target probe packet quintuple. Sending each Type I in-band network telemetry probe allows the acquisition of the network quality of all ports along its corresponding target telemetry path. These port network quality values ​​are then stored accordingly to obtain the network quality of each port.

[0101] Step S303: Generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic to obtain the service traffic telemetry path.

[0102] The first type of in-band network telemetry probe transmits at a higher frequency than the second type. See details below. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0103] Step S304: Query the network quality of ports based on the ports in the service traffic telemetry path to determine the network monitoring results of the server cluster to be monitored. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0104] The network monitoring method for server clusters provided in this embodiment ensures that all obtained probe packet quintuples cover as many ports as possible due to the randomness of the source ports in the random probe packets. Based on this, a selection process is performed to obtain the target probe packet quintuple, guaranteeing that all ports are covered at least a few prediction counts. This reduces redundant and duplicate probe packet quintuples, thereby increasing the number of first-type in-band network telemetry probes that can be processed and analyzed, and ultimately improving the efficiency of network monitoring.

[0105] This embodiment provides a network monitoring method for a server cluster, which can be used in the aforementioned servers. Figure 6 This is a flowchart of a network monitoring method for a server cluster according to an embodiment of this disclosure, such as... Figure 6 As shown, the process includes the following steps:

[0106] Step S601: Obtain the network topology information of the server cluster to be monitored.

[0107] Network topology information is used to characterize the connectivity relationships between network devices in the server cluster to be monitored. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0108] Step S602: Generate a first type of in-band network telemetry probe based on network topology information to obtain the network quality of ports in network devices.

[0109] For a detailed implementation of obtaining network quality using a first-type in-band network telemetry probe, please refer to [link to relevant documentation]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0110] In some implementations, the sending and receiving of probes is achieved through a preset network interface card (NIC) configured in the server. This preset NIC can be an RDMA NIC. After receiving the probe data corresponding to the probe, the RDMA NIC directly stores it in memory without CPU processing. Therefore, step S602 includes:

[0111] Step d1: Identify the packet header information of the first type of in-band network telemetry probe by using a preset network card to determine the target network device corresponding to the first type of in-band network telemetry probe.

[0112] Step d2: Send the first type of in-band network telemetry probe to the target network device through the preset network card.

[0113] Step d3: Receive network telemetry information from the first type of in-band network telemetry probe through the preset network card, and parse the network telemetry information to obtain the value of the target field.

[0114] Step d4: Based on the value of the target field, target processing is performed on the network telemetry information through a preset network card. Target processing includes sending feedback data packets to the target network device and storing the network telemetry information in memory.

[0115] Combination Figure 1 The network monitoring system shown, based on the detection requirements, has the INT agent retrieve the first type of in-band network telemetry probe from the INT controller and send it to the RDMA network card. The RDMA network card identifies the data packets, determines the target network device, and sends the first type of in-band network telemetry probe to the target network device. Correspondingly, the RDMA network card receives the detection data from the first type of in-band network telemetry probe and parses it to obtain the value of the target field. Different values ​​of the target field indicate different target processing for the detection data. After parsing the target field value, corresponding target processing is performed. This target processing includes sending a feedback data packet (ACK) to the target network device and storing the detection data in memory. This method sets all switches as transmission nodes, treats the server's network card as a tail hop node, and collects telemetry data through the RDMA network card, which can reduce the soft interrupts caused by network telemetry to the server CPU and provide more accurate telemetry data.

[0116] In some implementations, if the value of the target field indicates the need for real-time telemetry data feedback, then for telemetry data requiring real-time sensing and feedback, such as congestion control and routing, the RDMA network card re-encapsulates the data, flipping the source IP address and destination IP address, encapsulating the telemetry data into a payload to obtain a feedback data packet, which is then fed back to the target network device. If the value of the target field indicates the need to directly store network telemetry information in memory, then the RDMA directly writes the network telemetry information into the designated server's memory, and the server periodically reports it to the telemetry database.

[0117] By offloading the telemetry data processing capability originally implemented by the switch to the RDMA network card, the server's soft interrupts are reduced, thereby reducing the CPU resource consumption. Furthermore, the switch does not need to actively report data, reducing southbound overhead. At the same time, the problem of inaccurate tail hop time delay in INT technology is also solved.

[0118] The probe is sent, received, and analyzed by a preset network card. By identifying network telemetry information, the probe data packets can be directly transmitted to the corresponding network applications. The information based on network monitoring will be directly transmitted to the designated memory through RDMA technology, thereby reducing CPU soft interrupts and indicating the accuracy of telemetry data.

[0119] Step S603: Generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic to obtain the fault path.

[0120] Among them, the transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe.

[0121] Because the second type of in-band network telemetry probe can obtain the telemetry path of all service traffic in real time, it can identify the fault path corresponding to the abnormal service traffic in the event of abnormal service traffic. Service traffic anomalies are detected through traffic monitoring, or other methods can be used; no restrictions are placed here, and the specific settings depend on actual needs.

[0122] It should be noted that the current abnormal service traffic may be caused by port failure or host-side failure. If the above analysis determines that there is no port failure, then the abnormal service traffic is caused by a host-side problem.

[0123] Specifically, step S603 includes:

[0124] Step S6031: If network fluctuations occur in the current service traffic, obtain the traffic quintuple information of the current service traffic.

[0125] After detecting network fluctuations in current service traffic, the path information of the service traffic before the failure is obtained by acquiring the traffic 5-tuple information of the current service traffic. By default, the path information of the same service traffic remains unchanged.

[0126] Step S6032: Generate a second type of in-band network telemetry probe based on the traffic quintuple information to obtain the service traffic telemetry path and determine the service traffic telemetry path as the fault path.

[0127] The fault path includes multiple ports.

[0128] When network fluctuations occur, the path information of current service traffic changes. These failures are referred to as path change events. The path prior to the occurrence of the path change event is used as the fault path. The fault path is obtained through a second-type in-band network telemetry probe generated from the traffic 5-tuple information of the current service traffic.

[0129] For example, such as Figure 7 As shown, when performing network telemetry on service traffic, a second-type in-band network telemetry probe, consistent with the traffic 5-tuple information, is first sent. While this second-type in-band network telemetry probe is transmitted in the network, it collects the service traffic telemetry path (switch 1 inbound port, switch 1 outbound port, switch 2 inbound port, switch 2 outbound port, switch 4 inbound port, switch 4 outbound port). By querying the network quality obtained for each port based on the first-type in-band network telemetry probe, and then combining all port states, the network quality of the service traffic is obtained.

[0130] Step S604: Based on the ports in the business traffic telemetry path, query the port network quality to determine the fault location result of the server cluster to be monitored.

[0131] Specifically, step S604 includes:

[0132] Step S6041: Analyze the fault path to obtain the ports and fault timestamps in the fault path.

[0133] The fault path includes multiple ports, and the time point corresponding to the fault path is called the fault timestamp. Since the transmission frequency of the second type of in-band network telemetry probe is lower than that of the first type of in-band network telemetry probe, in order to ensure the reliability of the fault information corresponding to the fault timestamp, a corresponding fault query time period is subsequently formed based on the fault timestamp.

[0134] Step S6042: Determine the fault query time period based on the fault timestamp.

[0135] The time period corresponding to a preset duration before and after the fault timestamp is determined as the fault query time period. For example, based on the fault timestamp, the time period corresponding to 5 seconds before and after it is determined as the fault query time period.

[0136] Step S6043: Query the network quality of each port in the fault path during the fault query period.

[0137] Because the first type of in-band network telemetry probe has a high transmission frequency, the network quality of each port can be obtained multiple times during the fault query period. These network quality data are then aggregated to obtain the network quality during the fault query period.

[0138] Step S6044: Based on the queried network quality, determine the fault location result of the server cluster to be monitored.

[0139] The network quality data is used to characterize the network status of each port. If the network quality indicates an abnormality, the port is identified as abnormal; otherwise, it can be considered an abnormality on the host side. Further, after identifying a port as abnormal, the fault type is determined by combining the detailed network quality information to obtain detailed fault information.

[0140] In some optional implementations, step S6044 above includes:

[0141] Step e1: Obtain network telemetry information detected by each first-type in-band network telemetry probe corresponding to the port in the fault path. Network quality is obtained based on the analysis of network telemetry information.

[0142] Step e2: If all network telemetry information detected by the first type of in-band network telemetry probes of the target port indicates that there is a network quality fault, then the target port is determined to be a faulty port and the fault type is determined based on the network telemetry information detected by the five-tuple of all target probe packets.

[0143] As analyzed above, each port corresponds to multiple Type I in-band network telemetry probes, and each Type I in-band network telemetry probe can obtain corresponding network telemetry information. Therefore, for each port in the fault path, the multiple Type I in-band network telemetry probes of each port can obtain multiple network telemetry information. By further processing the network telemetry information, the network quality is obtained. If the network telemetry information of each Type I in-band network telemetry probe indicates that there is a network quality fault, it is determined to be a port fault, and the port is identified as the target port. Based on this, the fault type is determined based on the network telemetry information detected by the five-tuple of all target probe packets of the target port.

[0144] Since the same port contains multiple target detection packet quintuples, it can be detected by multiple Type I in-band network telemetry probes. Only when the network telemetry information of all Type I in-band network telemetry probes on the same port indicates that there is a network quality fault, the port is determined to be a faulty port, which further ensures the accuracy of fault location.

[0145] The network monitoring method for server clusters provided in this embodiment generates a second type of in-band network telemetry probe based on the current service traffic's five-tuple information when network fluctuations occur, and then detects the fault path to obtain an accurate fault path. Since there is a difference in transmission frequency between the first type and the second type of in-band network telemetry probe, the fault timestamp is extended to a query time period, and the network quality is queried using this query time period, ensuring the accuracy of the obtained network quality information.

[0146] In some optional implementations, the above-described network monitoring method for server clusters further includes:

[0147] Step f1: Store the network telemetry information of each port obtained by the first type of in-band network telemetry probe into the first area of ​​the target storage location, so as to obtain the network quality based on the network telemetry information.

[0148] Step f2: Store the service traffic telemetry path obtained by the second type of in-band network telemetry probe to the second area of ​​the target storage location.

[0149] The first type of in-band network telemetry probe obtains network telemetry information for each port. Further analysis based on this telemetry information yields the port's network quality, which is then stored in the first area of ​​the target storage location. That is, the first area stores the obtained network telemetry information and the analyzed network quality. The second type of in-band network telemetry probe obtains the service traffic telemetry path and stores it in the second area of ​​the target storage location. For example, the target storage location is... Figure 1 The INT database shown is divided into two areas: the first area is used to store network data, and the second area is used to store path data.

[0150] Since the network telemetry information obtained by the first type of in-band network telemetry probe has a different processing method than the service traffic telemetry path obtained by the second type of in-band network telemetry probe, they are stored in different areas to ensure that their respective processing methods can be processed normally.

[0151] The aforementioned network monitoring method for server clusters proposes a high-performance, lightweight in-band network telemetry framework based on RDMA networks. A high-performance packet identification scheme, designed based on the programmability of RDMA network cards, offloads telemetry data processing to the RDMA network card, achieving compatibility between INT-based monitoring and INT-based congestion control. Combined with RDMA technology, probe data is directly written into memory. Finally, through data analysis techniques, real-time and fine-grained full-traffic, full-port measurement is achieved. Specifically, by obtaining the unique target probe packet quintuple set corresponding to each port, a first-type in-band telemetry probe set is obtained. Next, service traffic measurement acquires the port information traversed by service traffic when a network fault occurs. Finally, by combining the port information with the detection results of the first-type in-band network telemetry probes corresponding to each port, rapid location of network faults and network performance bottlenecks in service traffic is achieved. This scheme reduces the total amount of data required for network fault location and supports real-time fault location.

[0152] As a specific application embodiment of this disclosure, the current service traffic telemetry path is port <1,1>, port <2,1>, port <3,1>, wherein the target probe packet quintuple set corresponding to port <1,1> is: quintuple 1, quintuple 4, and quintuple 5; the target probe packet quintuple set corresponding to port <2,1> is: quintuple 1, quintuple 3, and quintuple 4; and the target probe packet quintuple set corresponding to port <3,1> is: quintuple 1, quintuple 4, and quintuple 6.

[0153] When network fluctuations occur, the fault path is identified using the second type of in-band network telemetry probe, thus obtaining the faulty port. Based on the faulty port, the corresponding target probe packet quintuple set is obtained, and the corresponding first type of in-band network telemetry probe is used to detect and obtain network telemetry information. Specifically, if a fault exists at that moment, the network telemetry information of the probe corresponding to the nth quintuple of the faulty port indicates a network quality fault. Based on this, the detection results of each quintuple probe are as follows: quintuple 1 indicates a fault, quintuple 2 indicates no fault, quintuple 3 indicates no fault, quintuple 4 indicates a fault, quintuple 5 indicates a fault, and quintuple 6 indicates no fault. Current port fault determination:

[0154] Port <1,1>: The probe results of each probe corresponding to the quintuple are: the probe results corresponding to quintuple 1, quintuple 4 and quintuple 5 are faults.

[0155] Port <2,1>: The probe results of each probe corresponding to the quintuple are as follows: the probe corresponding to quintuple 1 has a fault detection result, while the probes corresponding to quintuple 3 and quintuple 4 have a non-fault detection result.

[0156] Port <3,1>: The probe results of each probe corresponding to the quintuple are as follows: The probe results corresponding to quintuple 1, quintuple 4 and quintuple 6 are non-faulty.

[0157] Since all probes corresponding to the quintuple of port <1,1> show fault results, port <1,1> is a faulty port.

[0158] If all probes corresponding to any 5-tuple set show a fault, the problem is located at that port. Otherwise, it is determined to be a host-side problem. The fault type can be identified as at least one of the following: network congestion, network jitter, network packet loss, link failure, or switch failure.

[0159] This embodiment also provides a network monitoring device for a server cluster, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0160] This embodiment provides a network monitoring device for a server cluster, such as... Figure 8 As shown, it includes:

[0161] The network topology information acquisition module 801 is used to acquire the network topology information of the server cluster to be monitored. The network topology information is used to characterize the connection relationship between network devices in the server cluster to be monitored.

[0162] The first type of in-band network telemetry probe generation module 802 is used to generate a first type of in-band network telemetry probe based on network topology information in order to obtain the network quality of ports in network devices.

[0163] The second type of in-band network telemetry probe generation module 803 is used to generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic in order to obtain the service traffic detection path. The transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe.

[0164] The network quality query module 804 is used to query the network quality of ports based on the ports in the business traffic detection path, and to determine the network monitoring results of the server cluster to be monitored.

[0165] In some alternative implementations, the first type of in-band network telemetry probe generation module 802 includes:

[0166] The probe packet quintuple generation unit is used to generate random probe packets based on network topology information to obtain the correspondence between the probe packet quintuple and the telemetry path. The source port in the random probe packet is randomly generated.

[0167] The probe packet quintuple filtering unit is used to filter the probe packet quintuples corresponding to random probe packets based on the correspondence between the probe packet quintuples and the telemetry path to obtain the target probe packet quintuples, so as to determine the port corresponding to the target probe packet quintuples. All target probe packet quintuples are used to cover all ports for at least a preset number of times, and the preset number of times is the length of the longest telemetry path corresponding to the random probe packet.

[0168] The target telemetry path determination unit is used to obtain the target telemetry path corresponding to the target probe packet quintuple by utilizing the correspondence between the probe packet quintuple and the telemetry path, so as to determine the set of target probe packet quintuples corresponding to each port in the target telemetry path.

[0169] The port network quality determination unit is used to generate a first-type in-band network telemetry probe based on the five-tuple set of target probe packets corresponding to each port, and to obtain the network quality of each port in the network device.

[0170] In some optional implementations, the probe packet quintuple generation unit includes:

[0171] The labeling sub-unit is used to label network devices in the server cluster to be monitored based on network topology information.

[0172] The grouping subunit is used to group all network devices based on the network cluster in which they belong, and obtain the unique number of the network card in the server in the server cluster to be monitored. The second cluster number corresponding to the network devices in the same group is different, while the other cluster numbers are the same. The second cluster number is the number of the cluster that has a connection relationship except for the top-level network device.

[0173] The random probe packet generation subunit is used to generate random probe packets for transmission between network interface cards within each group based on a unique number, so as to obtain the correspondence between the probe packet quintuple and the telemetry path.

[0174] In some alternative implementations, the probe packet quintuple screening unit includes:

[0175] The filter stack construction sub-unit is used to construct the filter stack. The filter stack includes stack elements and the connection relationships between stack elements. The stack elements in the filter stack correspond one-to-one with the probe packet quintuple, and the value of the stack element is used to represent the number of effective ports that the probe packet quintuple can cover. The number of effective ports is obtained based on the telemetry path.

[0176] The first target probe packet quintuple determination subunit is used to determine the probe packet quintuple corresponding to the top element of the heap to be screened as the target probe packet quintuple, and delete the top element.

[0177] The heap to be filtered update sub-unit is used to update the remaining heap element with the largest value as the new top element of the heap, so as to update the heap elements and the values ​​of the heap elements in the heap to be filtered.

[0178] The second target detection packet quintuple determination subunit is used to determine the target detection packet quintuple based on the updated target heap.

[0179] In some optional implementations, the target telemetry path determination unit includes:

[0180] The correspondence query subunit is used to query the correspondence based on the target probe packet quintuple to obtain the target telemetry path, so as to obtain the target probe packet quintuple corresponding to each port in the target telemetry path. The target telemetry path includes multiple ports.

[0181] The integration subunit is used to integrate the target probe packet quintuples corresponding to the same port to obtain the target probe packet quintuple set corresponding to each port.

[0182] In some alternative implementations, the second type of in-band network telemetry probe generation module 803 includes:

[0183] The Traffic Probe Packet Five-Tube Acquisition Unit is used to acquire the traffic probe packet five-tuple of the current service traffic if network fluctuations occur in the current service traffic.

[0184] The fault path acquisition unit is used to generate a second type of in-band network telemetry probe based on the traffic probe packet quintuple, so as to obtain the service traffic telemetry path and determine the service traffic telemetry path as the fault path, which includes multiple ports.

[0185] In some alternative implementations, the network quality query module 804 includes:

[0186] The fault timestamp determination unit is used to analyze the fault path to obtain the ports and fault timestamps in the fault path.

[0187] The fault query time period determination unit is used to determine the fault query time period based on the fault timestamp.

[0188] The network quality query unit is used to query the network quality of each port in the fault path during the fault query period.

[0189] The fault location unit is used to determine the fault location results of the server cluster to be monitored based on the queried network quality.

[0190] In some optional implementations, the fault location unit includes:

[0191] The network telemetry information acquisition subunit is used to acquire network telemetry information detected by each first-type in-band network telemetry probe corresponding to the port in the fault path. Network quality is obtained based on the analysis of network telemetry information.

[0192] The fault type determination subunit is used to determine the target port as a faulty port and determine the fault type based on the network telemetry information detected by all in-band network telemetry probes of the first type of target port.

[0193] In some alternative implementations, the first type of in-band network telemetry probe generation module 802 further includes:

[0194] The packet header information identification unit is used to identify the packet header information of the first type of in-band network telemetry probe through a preset network card, and to determine the target network device corresponding to the first type of in-band network telemetry probe.

[0195] The first type of in-band network telemetry probe sending unit is used to send the first type of in-band network telemetry probe to the target network device through a preset network card.

[0196] The probe data receiving unit is used to receive network telemetry information from the first type of in-band network telemetry probe through a preset network card, and to parse the network telemetry information to obtain the value of the target field.

[0197] The target processing unit is used to perform target processing on network telemetry information based on the value of the target field through a preset network card. The target processing includes sending feedback data packets to the target network device and storing the probe data in memory.

[0198] In some optional implementations, the network monitoring device for the server cluster further includes:

[0199] The first storage module is used to store the network telemetry information of each port obtained by the first type of in-band network telemetry probe to the first area of ​​the target storage location, so as to obtain the network quality based on the network telemetry information.

[0200] The second storage module is used to store the service traffic telemetry path obtained by the second type of in-band network telemetry probe to the second area of ​​the target storage location.

[0201] In this embodiment, the network monitoring device for the server cluster is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0202] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0203] This disclosure also provides an electronic device having the above-described features. Figure 8 The network monitoring device shown is for the server cluster.

[0204] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure, such as... Figure 9 As shown, the electronic device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0205] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0206] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0207] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0209] The electronic device also includes a communication interface 30 for communicating with other devices or communication networks.

[0210] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0211] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A network monitoring method for a server cluster, characterized in that, The method includes: Obtain the network topology information of the server cluster to be monitored, wherein the network topology information is used to characterize the connection relationship between network devices in the server cluster to be monitored; Based on the network topology information, a first type of in-band network telemetry probe is generated to obtain the network quality of the ports in the network device. A second type of in-band network telemetry probe is generated based on the traffic quintuple information of the current service traffic to obtain the service traffic telemetry path. The transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe. Based on the ports in the telemetry path of the business traffic, the network quality of the ports is queried to determine the network monitoring results of the server cluster to be monitored.

2. The method according to claim 1, characterized in that, The step of generating a first type of in-band network telemetry probe based on the network topology information to obtain the network quality of ports in the network device includes: Random probe packets are generated based on the network topology information to obtain the correspondence between the probe packet quintuple and the telemetry path. The source port in the random probe packet is randomly generated. Based on the correspondence between the probe packet quintuple and the telemetry path, the probe packet quintuple corresponding to the random probe packet is filtered to obtain the target probe packet quintuple, so as to determine the port corresponding to the target probe packet quintuple. All the target probe packet quintuples are used to cover all the ports at least a preset number of times, where the preset number of times is the length of the longest telemetry path corresponding to the random probe packet. By utilizing the correspondence between the probe packet quintuple and the telemetry path, the target telemetry path corresponding to the target probe packet quintuple is obtained, thereby determining the set of target probe packet quintuples corresponding to each port in the target telemetry path; The first type of in-band network telemetry probe is generated based on the set of five-tuples of target probe packets corresponding to each port, and the network quality of each port in the network device is obtained.

3. The method according to claim 2, characterized in that, The step of generating random probe packets based on the network topology information to obtain the correspondence between probe packet quintuples and telemetry paths includes: The network devices in the server cluster to be monitored are labeled based on the network topology information. Based on the network cluster in which the network devices are located, all the network devices are grouped to obtain the unique number of the network card in the server in the server cluster to be monitored. The network devices in the same group have different second cluster numbers and the other cluster numbers are the same. The second cluster number is the number of the cluster that has a connection relationship except for the top-level network device. Based on the unique number, random probe packets are generated for transmission between network interface cards within each group to obtain the correspondence between the probe packet quintuple and the telemetry path.

4. The method according to claim 2, characterized in that, Based on the correspondence between the probe packet quintuple and the telemetry path, the probe packet quintuple corresponding to the random probe packet is filtered to obtain the target probe packet quintuple, including: Construct a filter stack, which includes stack elements and the connection relationships between the stack elements. Each stack element in the filter stack corresponds one-to-one with the probe packet quintuple, and the value of the stack element is used to represent the number of effective ports that the probe packet quintuple can cover. The number of effective ports is obtained based on the telemetry path. The probe packet quintuple corresponding to the top element of the pile to be screened is determined as the target probe packet quintuple, and the top element is deleted. The remaining heap element with the largest value is updated as the new top element of the heap, thereby updating the heap elements in the heap to be filtered and the values ​​of the heap elements; The target probe packet quintuple is determined by filtering the probe packet quintuples based on the updated heap to be filtered.

5. The method according to claim 2, characterized in that, The step of utilizing the correspondence between the probe packet quintuple and the telemetry path to obtain the target telemetry path corresponding to the target probe packet quintuple, and determining the set of target probe packet quintuples corresponding to each port in the target telemetry path, includes: Based on the target detection packet quintuple, the correspondence is queried to obtain the target telemetry path, so as to obtain the target detection packet quintuple corresponding to each port in the target telemetry path, and the target telemetry path includes multiple ports; The target probe packet quintuples corresponding to the same port are integrated to obtain the target probe packet quintuple set corresponding to each port.

6. The method according to claim 1, characterized in that, The generation of a second type of in-band network telemetry probe based on the traffic 5-tuple information of the current service traffic to obtain the service traffic telemetry path includes: If network fluctuations occur in the current service traffic, obtain the traffic quintuple information of the current service traffic; Based on the traffic quintuple information, a second type of in-band network telemetry probe is generated to obtain the service traffic telemetry path and determine the service traffic telemetry path as a fault path, wherein the fault path includes multiple ports.

7. The method according to claim 6, characterized in that, The process of querying port network quality based on the ports in the service traffic telemetry path to determine the network monitoring results of the server cluster to be monitored includes: Based on the fault path, the ports and fault timestamps in the fault path are obtained through analysis. The fault query time period is determined based on the fault timestamp. Query the network quality of each port in the fault path during the fault query time period; Based on the queried network quality, the fault location results of the server cluster to be monitored are determined.

8. The method according to claim 7, characterized in that, The process of determining the fault location result of the server cluster to be monitored based on the queried network quality includes: The network telemetry information detected by each first-type in-band network telemetry probe corresponding to the port in the fault path is obtained, and the network quality is obtained based on the analysis of the network telemetry information. If all network telemetry information detected by the first type of in-band network telemetry probes at the target port indicates that there is a fault in the network quality, then the target port is determined to be a faulty port and the fault type is determined based on the network telemetry information detected by all the first type of in-band network telemetry probes.

9. The method according to claim 1, characterized in that, The step of generating a first type of in-band network telemetry probe based on the network topology information to obtain the network quality of ports in the network device further includes: The packet header information of the first type of in-band network telemetry probe is identified by a preset network card to determine the target network device corresponding to the first type of in-band network telemetry probe. The first type of in-band network telemetry probe is sent to the target network device via the preset network card; The network telemetry information of the first type of in-band network telemetry probe is received through the preset network card, and the network telemetry information is parsed to obtain the value of the target field; Based on the value of the target field, the network telemetry information is processed by the preset network card. The target processing includes sending feedback data packets to the target network device and storing the network telemetry information in memory.

10. The method according to claim 1, characterized in that, The method further includes: The network telemetry information of each port obtained by the first type of in-band network telemetry probe is stored in the first area of ​​the target storage location, so as to obtain the network quality based on the network telemetry information; The service traffic telemetry path obtained by the second type of in-band network telemetry probe is stored in the second area of ​​the target storage location.

11. A network monitoring device for a server cluster, characterized in that, The device includes: The network topology information acquisition module is used to acquire the network topology information of the server cluster to be monitored, wherein the network topology information is used to characterize the connection relationship between network devices in the server cluster to be monitored. A first-type in-band network telemetry probe generation module is used to generate a first-type in-band network telemetry probe based on the network topology information in order to obtain the network quality of the ports in the network device. The second type of in-band network telemetry probe generation module is used to generate a second type of in-band network telemetry probe based on the traffic quintuple information of the current service traffic in order to obtain the service traffic telemetry path. The transmission frequency of the first type of in-band network telemetry probe is higher than that of the second type of in-band network telemetry probe. The network quality query module is used to query the network quality of ports based on the ports in the service traffic telemetry path, and determine the network monitoring results of the server cluster to be monitored.

12. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the network monitoring method for a server cluster as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the network monitoring method for the server cluster as described in any one of claims 1 to 10.

Images