Method and Device for Network Monitoring and Diagnosis of Intelligent Computing Centers Based on Programmable Switches
By using the P4 programmable switch for intelligent computing center network monitoring, the system automatically generates probe messages and combines them with a feedback mechanism. This solves the problems of resource dependence and poor scalability in existing technologies, achieving high-precision network performance monitoring and rapid fault location. It is suitable for large-scale intelligent computing center networks.
Patent Information
- Application Number
- CN202611375425.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-09-07
AI Technical Summary
Existing intelligent computing center network monitoring methods rely on host resources, resulting in high resource consumption, poor scalability, difficulty in distinguishing between network-side and host-side performance bottlenecks, and difficulty in adapting to the monitoring needs of large-scale high-performance network environments.
By using a network monitoring method for intelligent computing centers based on P4 programmable switches, the management platform, P4 programmable rack top switches, and host-side agents work together to automatically generate probe messages, achieving high-precision measurement and anomaly diagnosis of communication paths within and between racks, reducing dependence on host resources, and improving monitoring real-time performance and system scalability.
It achieves low resource consumption, high monitoring accuracy and good scalability, and can perceive network status in real time and quickly locate anomalies, making it suitable for large-scale high-performance network environments.
Smart Images

Figure CN122891494B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent computing center network monitoring and fault diagnosis technology, specifically to a method and device for intelligent computing center network monitoring and diagnosis based on a programmable switch, applicable to high-performance network environments represented by RoCE. Background Technology
[0002] With the rapid development of applications such as cloud computing, artificial intelligence, and large-scale model training, the scale of intelligent computing centers continues to expand, and the impact of network communication performance on overall system throughput and Quality of Service (QoS) is becoming increasingly significant. Intelligent computing centers typically handle high-performance computing, distributed training, and data-intensive tasks, placing more stringent demands on network low latency, high throughput, and high stability. Especially in high-speed network environments built on Remote Direct Memory Access (RDMA) technology, such as lossless networks represented by RoCE (RDMA over Converged Ethernet), the sensitivity of services to communication performance is further enhanced. Once link congestion, path imbalance, or host-side RNIC (RDMA Network Interface Card) performance bottlenecks occur in the network, they often trigger the "weakest link" effect, significantly reducing the overall efficiency of critical services such as distributed training. Therefore, high-precision, low-overhead real-time performance monitoring and fault diagnosis of large-scale intelligent computing center networks are of great importance.
[0003] In existing technologies, mainstream network performance monitoring and diagnosis methods mostly rely on deploying agent programs on the host side. These agents periodically send probe packets through an end-to-end probing mechanism, collecting metrics such as round-trip time (RTT) and packet loss rate, and then uploading this data to a centralized controller for unified analysis. While these methods can visualize network status to some extent, they still have significant shortcomings: First, the generation and processing of probe packets depend on host CPU resources, which can interfere with normal host operations and limit system scalability in large-scale deployments or high-frequency probing scenarios. Second, measurement data must be transmitted over the network to the controller for centralized processing, making it susceptible to network congestion or control channel bottlenecks, thus reducing measurement accuracy and increasing diagnostic latency. Furthermore, these methods often struggle to distinguish between network-side and host-side performance issues, resulting in coarse-grained fault location.
[0004] In recent years, with the development of P4 (Programming Protocol-independent Packet Processors) programmable switches, online monitoring methods that offload some measurement logic to network devices have gradually emerged. These methods can reduce reliance on host resources and improve real-time monitoring performance to some extent. However, existing switch-based data plane measurement schemes mostly focus on passive statistics or simple path sampling, lacking active detection capabilities and struggling to fully cover complex multi-path network environments. Furthermore, their scalability in multi-rack scenarios is limited, often requiring high probe traffic overhead, making them unsuitable for monitoring the massive inter-host communication needs in large-scale intelligent computing centers.
[0005] Therefore, there is an urgent need for a performance monitoring and fault diagnosis method for large-scale intelligent computing center networks to solve the above-mentioned deficiencies. Summary of the Invention
[0006] The purpose of this invention is to address the problems in existing intelligent computing center networks, such as reliance on host participation for communication performance monitoring, high resource consumption, poor scalability, and difficulty in distinguishing performance bottlenecks on the network side and the host side. This invention provides a method and device for monitoring and diagnosing intelligent computing center networks based on a programmable switch. This invention enables high-precision measurement and anomaly diagnosis of communication paths within and between racks using a P4 programmable top-of-rack switch without requiring the host to actively generate probe messages. This significantly improves the real-time performance, accuracy, and system scalability of monitoring, and is suitable for low-latency, high-throughput network environments such as RoCE.
[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a method for monitoring and diagnosing intelligent computing center networks based on programmable switches, applicable to RoCE intelligent computing center networks. Through the collaborative work of a management platform, a P4 programmable top-of-rack switch, and a host-side agent, it automatically monitors and diagnoses communication performance within and between racks. The method specifically includes: Pre-configuration phase: The management and control platform configures the detection task information of the P4 programmable top-of-rack switch; Probe message generation phase: Probe messages are automatically generated by the data plane on a periodic basis; Data plane processing and distribution phase: The data plane selects the corresponding distribution strategy according to the type of probe packet to distribute the probe packet to the target host, and realizes multi-path traversal in cross-rack scenarios; Response collection and delay decomposition phase: The target host-side agent automatically responds to the probe packets, and the source P4 programmable top-of-rack switch collects the response packets and records delay-related information; Anomaly detection and reporting phase: Based on latency-related information analysis, when latency anomalies, packet loss, or unreachable paths are detected, the source P4 programmable top-of-rack switch records the anomaly information and reports it to the management and control platform. The management and control platform summarizes and analyzes the measurement data of the entire network to locate network performance bottlenecks and faults. Closed-loop operation phase: The above phases are executed cyclically according to the preset measurement cycle, forming a continuous monitoring and diagnosis closed-loop process.
[0008] Furthermore, the detection task information includes rack-based host address list information, inter-rack address list information, 5-tuple information, and multicast group information generated from the rack-based host address list information. The 5-tuple information includes source IP address, source port number, destination IP address, destination port number, and transport layer protocol.
[0009] Furthermore, the probe message generation stage specifically includes: Within each measurement cycle, the control plane of the P4 programmable top-of-rack switch configures the message generation module to automatically generate probe message templates in RDMA unreliable datagram mode at a given cycle and injects them into the data plane pipeline. The data plane pipeline fills in the fields of the probe message templates based on the pre-configuration information of the management and control platform to form probe messages for in-rack and inter-rack measurements. The probe message includes a key field for controlling the measurement process, an identification field for distinguishing the measurement range, a control field for triggering multicast replication, a round identification field for identifying the measurement round, and a sequence identification field for identifying the measurement sequence.
[0010] Furthermore, the distribution strategy specifically includes: For rack-mounted detection scenarios, the management platform generates a multicast group based on the host address list information within the rack and sends the multicast group information to the P4 programmable top-of-rack switch, enabling it to achieve communication coverage of other hosts within the rack through a single detection message; For inter-rack probing scenarios, a set of cross-rack probing paths is generated based on the inter-rack address list information, and combined with the five-tuple information maintained by the management and control platform, to achieve communication measurement between multiple racks.
[0011] Furthermore, the data plane processing and distribution stage specifically includes: The data plane pipeline of the P4 programmable top-of-rack switch parses the probe packets and determines the type of the probe packet based on the identification field carried in the probe packet to distinguish the measurement range: If it is a probe message within the rack, the probe message is distributed to each host within the rack through multicast replication based on the multicast group information pre-configured during the pre-configuration phase. If it is an inter-rack probe packet, the management platform will determine the source IP address, source port number, and destination IP address fields based on the inter-rack address list to generate and distribute five-tuple information. Based on the five-tuple information, the probe packet will be forwarded to the remote P4 programmable top-of-rack switch to achieve traversal of multiple equal-cost multi-path forwarding paths. At the same time, the remote P4 programmable top-of-rack switch will forward the probe packet to the corresponding target host in turn based on the sequence identifier information carried in the probe packet.
[0012] Furthermore, the response collection and delay decomposition stage specifically includes: The P4 programmable top-of-rack switch records the time when sending probe packets. After the probe packet arrives at the target host, the host-side agent reads the completion queue entry generated by the RNIC to obtain the time when the host received the probe packet. and the time when the host sends back the response message and will The value is written as the host-side latency into the response message; the source P4 programmable top-of-rack switch records the time when it receives the response message. And read the host-side latency carried in the response message, in order to... Calculate network round-trip latency .
[0013] Furthermore, communication within the rack specifically includes the following processes: After identifying a probe message within the rack, the P4 programmable top-of-rack switch updates the current measurement round information and records the transmission timestamp. The probe messages are simultaneously distributed to all hosts within the rack via multicast replication. After receiving the response message, the receiving timestamp is recorded and the network round-trip delay between each host is calculated by combining the host-side delay carried in the response message. If no response message is received within a given time, it is determined to be a packet loss event. By utilizing the packet generation and distribution mechanism of the P4 programmable top rack switch, probe packets are generated and distributed by the switch side, enabling full interconnection measurement without the host actively generating probe packets.
[0014] Furthermore, the communication between racks specifically includes the following processes: The source P4 programmable top-of-rack switch sends probe packets to the remote P4 programmable top-of-rack switch. The remote P4 programmable top-of-rack switch determines the target host address based on the sequence identifier information carried in the probe packet using a mapping method based on counting and modulo, and forwards the probe packets to the corresponding target host in turn. Within multiple measurement cycles, the five-tuple adjustment mechanism sequentially traverses and covers all hosts in the remote rack and different equivalent multipath forwarding paths.
[0015] A second aspect of this invention provides a network monitoring and diagnostic device for intelligent computing centers based on programmable switches, including a management platform, a host-side agent, and one or more P4 programmable top-of-rack switches. The management platform is a host connected to the network, used to store and update host communication information and network topology information, summarize and analyze network-wide measurement data, and send probe task information to the P4 programmable top-of-rack switches, including rack-side host address list information, inter-rack address list information, 5-tuple information, and multicast group information generated from the rack-side host address list information. The host-side agent is used to obtain host-side timestamps and respond to probe packets. The P4 programmable top-of-rack switches and the hosts form a leaf-spine network architecture. The P4 programmable top-of-rack switches, the management platform, and the host-side agent are used to collaboratively execute the above-mentioned intelligent computing center network monitoring and diagnostic method based on programmable switches to realize probe packet generation, path traversal, latency calculation, and anomaly detection functions.
[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon. When executed by a P4 programmable top-of-rack switch, a management platform, and / or a host-side agent, the program is used to implement the above-described intelligent computing center network monitoring and diagnosis method based on a programmable switch.
[0017] Compared with the prior art, the beneficial effects of the present invention are:
[0018] (1) This invention can make full use of the data plane processing capability of the P4 programmable top rack switch to achieve efficient generation and distribution of probe packets, and combined with the feedback mechanism, complete the fine monitoring of network path and host performance; at the same time, this invention has low resource overhead, high monitoring accuracy and good scalability, and effectively realizes real-time perception and rapid anomaly location of the network operation status of the intelligent computing center.
[0019] (2) This invention achieves network monitoring without requiring the host to actively generate probe messages by sinking the probe message generation and monitoring processing functions to the data plane of the P4 programmable rack top switch, significantly reducing host resource overhead and improving system scalability; it utilizes the switch hardware-level message generation and multicast replication method to achieve high throughput and low latency probe message distribution, improving measurement efficiency and real-time performance; through the deterministic mapping mechanism of the management platform, it achieves effective coverage of multiple paths within and between racks, ensuring measurement integrity while reducing probe complexity; it decomposes round-trip delay based on multi-timestamp information to distinguish between network transmission delay and host processing delay, realizing fault diagnosis and improving fault location accuracy; through the architecture combining distributed measurement and centralized analysis, it achieves rapid detection and global analysis of network anomalies, has good engineering application value, and is suitable for large-scale high-performance network scenarios such as intelligent computing centers. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the system architecture of a network monitoring and diagnosis method for intelligent computing centers based on programmable switches, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the in-rack network performance monitoring process provided in an embodiment of the present invention. Detailed Implementation
[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0023] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0024] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.
[0025] Figure 1 This is a schematic diagram of the overall architecture of a system corresponding to a network monitoring and diagnosis method for intelligent computing centers based on programmable switches provided in an embodiment of the present invention. Figure 1 As shown, the system architecture of this method includes a management and control platform, a P4 programmable top-of-rack switch, and a host-side agent. The management and control platform is used to store and update host communication information and network topology information, summarize and analyze network-wide measurement data, and send probe task information to the P4 programmable top-of-rack switch, including in-rack host address list information, inter-rack address list information, five-tuple information, and multicast group information generated from the in-rack host address list information. The P4 programmable top-of-rack switch, as the execution core, is responsible for generating, copying, distributing, timestamp recording, and obtaining results of probe packets. The host-side agent is used to obtain host-side timestamps and respond to probe packets. This method, through the collaborative work of the management and control platform, the P4 programmable top-of-rack switch, and the host-side agent, achieves in-rack probe, inter-rack probe, and service-oriented performance monitoring functions, and supports the decomposition and analysis of network latency and host processing latency.
[0026] This invention relates to a network monitoring and diagnosis method for intelligent computing centers based on programmable switches, applicable to RoCE intelligent computing center networks. In RoCE intelligent computing center networks, service data flows utilize Reliable Connection (RC) mode of RDMA, while probe packets utilize Unreliable Datagram (UD) mode. This method, through the collaborative work of a management platform, a P4 programmable top-of-rack switch, and host-side agents deployed on the hosts, automatically monitors and diagnoses network communication performance within and between racks. The host-side agent refers to the monitoring agent program deployed on each host node, responsible for executing probe tasks on the host side, including receiving probe task configurations, collecting host operating status information, and generating and sending probe packets. The method specifically includes the following steps:
[0027] (1) Pre-configuration stage: Configure the probe task information of the P4 programmable top rack switch, deploy the host-side agent on the host, and configure the management and control platform to maintain, distribute and collect and analyze global information, so that the management and control platform can save and update host communication information and network topology information, summarize and analyze the measurement data of the whole network, and distribute probe task information to the P4 programmable top rack switch, including the rack-side host address list information, the rack-to-rack address list information, the five-tuple information, and the multicast group information generated by the rack-side host address list information.
[0028] Specifically, during the pre-configuration phase, based on the network topology of the intelligent computing center, each P4 programmable top-of-rack switch is initialized and configured, specifically, the probe task information for the P4 programmable top-of-rack switches is configured. Simultaneously, a host-side agent is deployed on the host. This agent, upon receiving a probe packet from the target host, reads the Completion Queue Entry (CQE) generated by the RNIC to obtain the time the host received the probe packet and the time the host sent back the response packet. The host-side latency is then written into the response packet to support subsequent decomposition and calculation of network-side latency and host-side latency.
[0029] Furthermore, a management and control platform is configured. This platform acts as a host connected to the network, storing and updating host communication information and network topology information, summarizing and analyzing network-wide measurement data, and maintaining the five-tuple information of probe packets based on network topology relationships and probe requirements. Simultaneously, it distributes probe task information to the P4 programmable top-of-rack switches, including rack-based host address list information, inter-rack address list information, five-tuple information, and multicast group information generated from the rack-based host address list information. The rack-based host address list information describes the set of hosts currently connected to the P4 programmable top-of-rack switch, the inter-rack address list information identifies other P4 programmable top-of-rack switch nodes in the network and their topology relationships, and the five-tuple information includes the source IP address, source port number, destination IP address, destination port number, and transport layer protocol. By distributing the aforementioned probe task information, the management and control platform provides the basic configuration for probe packet generation, distribution, and path selection. During cross-rack communication measurements, it adjusts the five-tuple information to traverse Equal-Cost Multi-Path (ECMP) forwarding paths. The management platform primarily pre-deploys the necessary probing configurations to each P4 programmable top-of-rack switch, including host / RNIC information, probing period, multicast group configurations generated based on the rack's host address list, and inter-rack address list and 5-tuple information required for cross-rack and service flow probing. These configurations are then updated when network topology or service information changes.
[0030] It should be noted that intelligent computing center networks typically consist of multiple racks, each containing multiple hosts and P4 programmable top-of-rack switches connected to them. In a leaf-spine network architecture, cross-rack communication paths are not unique; multiple paths with the same forwarding cost exist. The ECMP routing mechanism is typically employed, which uses hash mapping on the data flow 5-tuple to allocate different data flows to different forwarding paths to achieve load balancing. Since the ECMP routing mechanism usually maintains a consistent hash result for data flows with the same 5-tuple, different probe flows can be generated by controlling the 5-tuple information of probe packets, enabling traversal coverage of multiple ECMP forwarding paths across racks.
[0031] (2) Probe message generation stage: Probe messages are automatically generated periodically by the data plane of the P4 programmable top rack switch.
[0032] Specifically, during the probe packet generation phase, within each measurement cycle, the P4 programmable top-of-rack switch's internal control plane configures PktGen to automatically generate probe packet templates for a specific RDMA UD mode at a given cycle and injects them into the data plane pipeline. The data plane pipeline fills in the probe packet templates with fields based on host / RNIC information, probe cycles, and other relevant information pre-configured by the management platform, forming probe packets for both in-rack and inter-rack measurements. These probe packets include key fields for controlling the measurement process, identification fields to distinguish the measurement range (i.e., in-rack and inter-rack), control fields to trigger multicast replication, round identifier fields to identify the measurement rounds, and sequence identifier fields to identify the measurement sequence. These fields enable fine-grained control of the probe tasks in subsequent data plane processing, supporting subsequent data plane processing and path traversal.
[0033] It should be noted that PktGen in the data plane of the P4 programmable top rack switch automatically generates probe message templates with built-in logic, without relying on the internal control plane to send them one by one.
[0034] (3) Data plane processing and distribution stage: The data plane pipeline of the P4 programmable top rack switch selects the corresponding distribution strategy according to the type of probe message and distributes the probe message to the target host. In the cross-rack scenario, multi-path traversal is realized. That is, in the intra-rack probe scenario, the probe message is forwarded to the target host through multicast replication. In the inter-rack probe scenario, multi-path traversal is realized based on the five-tuple information pre-configured by the management and control platform.
[0035] Furthermore, the distribution strategy specifically includes: For in-rack probing scenarios, the management platform generates multicast groups based on the in-rack host address list information and distributes the multicast group information to the P4 programmable top-of-rack switch. This allows a single probe packet to cover all hosts within the same rack after multicast replication, achieving efficient probe coverage of all hosts within the rack with a single probe packet. For inter-rack probing scenarios, a cross-rack probe path set is generated based on the inter-rack address list information and combined with the five-tuple information maintained by the management platform to achieve communication measurement and ECMP coverage between multiple racks.
[0036] It should be understood that a multicast group is equivalent to a forwarding address table. Once configured, it can be used to copy and distribute the same packet to each target host in the table.
[0037] Specifically, the data plane pipeline of the P4 programmable top-of-rack switch parses the probe packets to determine their type. This is achieved by identifying the type of probe packet based on the identifier field carried in the packet to distinguish the measurement range, thus enabling intra-rack multicast forwarding or inter-rack forwarding. For intra-rack probe packets, based on the multicast group information pre-configured during the pre-configuration phase, the probe packet is distributed to each host within the same rack via multicast replication. For inter-rack probe packets, the management platform deterministically modifies the source IP address, source port number, and destination IP address fields based on the inter-rack address list to generate and distribute five-tuple information. The probe packet is then forwarded to the remote P4 programmable top-of-rack switch based on this five-tuple information, enabling traversal of multiple ECMP forwarding paths. Simultaneously, the remote P4 programmable top-of-rack switch, based on the sequence identifier information carried in the probe packet, calculates (i.e., a mapping method based on counting and modulo) and forwards the probe packets to the corresponding target hosts in turn.
[0038] Specifically, in the data plane processing and distribution phase, after the probe packets enter the data plane, they are parsed (i.e., their type is determined) and processed by programmable logic in the data plane pipeline: If it is an intra-rack probe packet, the P4 programmable top-of-rack switch uses multicast replication to spread the probe packet of a single probe packet template to all target hosts within the rack, achieving full coverage measurement of the communication path within the rack; if it is an inter-rack probe packet, the P4 programmable top-of-rack switch sends the probe packet to a remote P4 programmable top-of-rack switch, and the management platform performs deterministic modification of the packet fields (directly modifying the five-tuple that will be filled into the probe packet header and affect forwarding), so that the probe packet can traverse different forwarding paths in different monitoring cycles, thereby achieving traversal of different equivalent multipaths and coverage of multipath network environments. Furthermore, in cross-rack scenarios, after receiving a probe packet, the remote P4 programmable top-of-rack switch determines the target host address based on the sequence identifier information (i.e., the sequence identifier field) carried in the probe packet using a counting and modulo-based mapping method. It then forwards the probe packet to the target host at the corresponding address, enabling round-robin probing of all hosts within the target rack. This method allows probe tasks in different monitoring cycles to cover all host nodes in the remote rack, achieving comprehensive coverage of the remote host set across multiple measurement cycles and reducing overall probe complexity. It ensures monitoring integrity while reducing probe traffic complexity.
[0039] Furthermore, in the data plane, the generation, processing, and distribution of probe messages specifically include the following processes: PktGen in the data plane of the P4 programmable top-of-rack switch automatically generates probe message templates using built-in logic, without relying on the internal control plane to distribute them one by one; within the measurement cycle, PktGen injects the generated probe message templates into the data plane pipeline in the P4 programmable top-of-rack switch; the data plane pipeline modifies the fields of the probe message templates based on the information issued by the management platform according to the probe task requirements to generate corresponding probe messages, and re-injects them into the data plane pipeline after copying the probe message templates to maintain continuous monitoring; the corresponding distribution strategy is selected based on the identification field carried in the probe message to distinguish the measurement range, realizing intra-rack multicast or inter-rack forwarding.
[0040] (4) Response collection and delay decomposition stage: The agent on the target host side automatically responds to the probe message, and the source P4 programmable top rack switch collects the response message and records the delay-related information.
[0041] Specifically, the source P4 programmable top-of-rack switch records the time when sending probe packets. After the probe packet arrives at the target host, the probe application in the host-side agent deployed on the target host reads the CQE generated by the RNIC to obtain the time when the host received the probe packet. and the time when the host sends back the response message and will The value is written as the host-side latency in the response message. Subsequently, the response message is returned to the source P4 programmable top-of-rack switch, which records the time upon receiving the response message. And read the host-side latency information carried in the response message, and finally according to Calculate network round-trip latency This enables real-time monitoring of network communication performance.
[0042] Specifically, in the feedback phase, namely the response collection and latency decomposition phase, after the target host receives the probe packet, the target host's RNIC generates a CQE related to the probe packet. The probe application in the agent deployed on the target host reads the corresponding timestamp information to obtain the time when the host received the probe packet. The time when the host sends back the response message and will The host-side delay is written into the response message and returned to the source P4 programmable top-of-rack switch. The source P4 programmable top-of-rack switch records the time during both the probe message transmission and response message reception processes. and time Therefore, after receiving the response message, the source P4 programmable top-of-rack switch can, based on , The network round-trip time is calculated using the host-side latency carried in the response message. Through the decomposition and processing of the timestamp information and host-side latency information, the overall measurement results can be distinguished into network transmission-related latency and host-side processing-related latency. This allows for the differentiation of network-side congestion, link anomalies, and host-side processing bottlenecks, enabling fine-grained location of performance bottlenecks. Furthermore, when the source P4 programmable top-of-rack switch does not receive a response message within a preset time threshold, it is classified as a packet loss event, and relevant anomaly information is recorded.
[0043] It should be understood that the source P4 programmable top-of-rack switch records timestamps when sending probe packets and receiving response packets. The target host obtains these timestamps by reading the corresponding CQEs generated by the RNIC during the receiving and sending of probe packets through the probe application in the proxy. Based on these timestamps, the host-side latency can be obtained, and the source P4 programmable top-of-rack switch can further calculate the network round-trip latency, thereby distinguishing the performance bottlenecks on the network side and the host side.
[0044] Furthermore, the communication measurement process within the rack specifically includes: after the P4 programmable top-of-rack switch identifies a probe packet within the rack, it updates the current measurement round information and records the sending timestamp; it simultaneously distributes the probe packet to each host within the rack via multicast replication; upon receiving an acknowledgment packet, it records the receiving timestamp and calculates the network round-trip latency with each host based on the host-side latency carried in the acknowledgment packet, and determines a packet loss event if no acknowledgment packet is received within a given time; through the P4 programmable top-of-rack switch's packet generation and distribution mechanism, the switch generates and distributes probe packets, achieving full interconnection measurement without requiring the host to actively generate probe packets, thus reducing the computational resource overhead on the host side caused by the probe packet generation process.
[0045] Furthermore, the communication measurement process between racks specifically includes: the source P4 programmable top-of-rack switch sends probe packets to the remote P4 programmable top-of-rack switch; the remote P4 programmable top-of-rack switch calculates the target host address based on the sequence identifier information carried in the probe packets, using a mapping method based on counting and modulo, and then forwards the probe packets to the corresponding target hosts in turn; within multiple measurement cycles, the five-tuple adjustment mechanism is used to sequentially traverse and cover all hosts in the remote rack and different ECMP forwarding paths.
[0046] (5) Anomaly detection and reporting stage: Based on the analysis of latency-related information, when an anomaly, packet loss or unreachable path is detected, the source P4 programmable top rack switch records the anomaly information and reports it to the management and control platform. The management and control platform summarizes and analyzes the measurement data of the entire network to locate network performance bottlenecks and faults.
[0047] It should be noted that by deterministically adjusting the five-tuple information of the probe packets, different probe packets are matched with different ECMP forwarding paths, thereby achieving full coverage of multi-path communication across racks. Furthermore, path status differentiation and fault location are achieved based on the measurement indicators corresponding to each path. Secondly, when probe packet data packets are returned, they can be broken down into network transmission latency and host processing latency based on the timestamp. This allows for the determination of whether the fault lies on the network side or the host side based on anomalies in different latency components, thus enabling the location of network performance bottlenecks and faults.
[0048] Specifically, in the result processing phase (i.e., the anomaly detection and reporting phase), each P4 programmable top-of-rack switch statistically summarizes and aggregates local measurement results. When anomalies in latency (e.g., latency exceeding a threshold, significant increases, etc.), packet loss, or path unreachability (i.e., no response to probe packets received or near 100% packet loss) are detected, an anomaly reporting mechanism is triggered. The management platform periodically collects data reported by each P4 programmable top-of-rack switch (i.e., local measurement results and anomalies such as latency anomalies, packet loss, and path unreachability), comprehensively analyzes round-trip latency, packet loss rate, and their fluctuations, and constructs a global network communication performance view. This allows the display of the distribution of anomalies in the network, changes in transmission latency across links (performance distribution across different links), and changes in processing latency of each host, thereby enabling the location and diagnosis of network faults and performance bottlenecks.
[0049] (6) Closed-loop operation phase: The above steps are executed cyclically according to the preset measurement cycle, forming a continuous monitoring and diagnosis closed-loop process of detection message generation, path traversal, response collection and result analysis.
[0050] Specifically, the above steps are executed cyclically according to a preset measurement cycle. Through the periodic operation mechanism, a closed-loop process of probe message generation, distribution, response collection and result analysis is formed, realizing automated and continuous network performance monitoring and fault diagnosis.
[0051] For example, Figure 2 This is a schematic diagram of an in-rack network performance monitoring process provided by an embodiment of the present invention. Figure 2 As shown, during the rack-mounted probing process, the management platform pre-configures a host address list and multicast group information in the P4 programmable top-of-rack switch. Subsequently, the packet generation module in the data plane of the P4 programmable top-of-rack switch periodically generates probing packet templates. After the probing packet templates are filled with the corresponding fields in the data plane pipeline, they are propagated to each target host in the rack via multicast replication. Upon receiving the probing packet, each host's RNIC generates a response packet and returns it to the source P4 programmable top-of-rack switch. The source P4 programmable top-of-rack switch calculates the round-trip time of the corresponding host by recording the probing packet sending time, the response packet receiving time, and the host-side latency carried in the response packet. If no response is received, it is determined as a packet loss event, thereby achieving real-time monitoring of the network status and host-side performance within the rack. Furthermore, by integrating packet generation and replication / distribution functions into the data plane of the P4 programmable top-of-rack switch, the monitoring task is decentralized to the network side, eliminating the need for the host to actively generate probing packets during the entire monitoring process, significantly reducing system resource overhead. Meanwhile, the multi-timestamp decomposition mechanism enables refined analysis of network latency and host processing latency, improving the accuracy of fault location.
[0052] Through the above structure and process, this invention realizes real-time monitoring and anomaly diagnosis of communication performance within the P4 programmable rack top switch. While ensuring measurement accuracy, it reduces system overhead and has high real-time performance, low invasiveness and good scalability. It is suitable for low-latency, high-throughput intelligent computing center network environments represented by RoCE.
[0053] In summary, the method described in this invention supports collaboration with remote direct memory access network protocols, is applicable to RDMA network environments represented by RoCE, and enables fully interconnected measurements without requiring the host to actively generate probe messages, thereby reducing monitoring overhead and improving measurement accuracy.
[0054] Corresponding to the aforementioned embodiments of the intelligent computing center network monitoring and diagnosis method based on programmable switches, the present invention also provides embodiments of an intelligent computing center network monitoring and diagnosis device based on programmable switches.
[0055] This invention provides a network monitoring and diagnostic device for intelligent computing centers based on programmable switches, comprising a management platform, a host-side agent deployed on a host, and one or more P4 programmable top-of-rack switches. The management platform is a host connected to the network, used to store and update host communication information and network topology information, summarize and analyze network-wide measurement data, and send probe task information to the P4 programmable top-of-rack switches, including rack-based host address list information, inter-rack address list information, 5-tuple information, and multicast group information generated from the rack-based host address list information. The P4 programmable top-of-rack switches and the host constitute a leaf-spine network architecture, and the method described in this invention is applicable to other data center network topologies. The P4 programmable top-of-rack switches, the management platform, and the host-side agent deployed on the host are used to collaboratively execute the aforementioned network monitoring and diagnostic method for intelligent computing centers based on programmable switches, to achieve probe packet generation, path traversal, latency calculation, and anomaly detection functions, automatically monitoring and diagnosing communication performance within and between racks.
[0056] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0057] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0058] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a P4 programmable top-of-rack switch, a management platform, and / or a host-side agent, is used to implement the above-described intelligent computing center network monitoring and diagnosis method based on a programmable switch.
[0059] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for monitoring and diagnosing intelligent computing center networks based on programmable switches, characterized in that, Applicable to RoCE intelligent computing center networks, this method automates the monitoring and diagnosis of communication performance within and between racks through the collaborative work of a management and control platform, a P4 programmable top-of-rack switch, and a host-side agent; the method specifically includes: Pre-configuration phase: The management and control platform configures the detection task information of the P4 programmable top-of-rack switch; Probe message generation phase: Probe messages are automatically generated periodically by the data plane; the probe message generation phase specifically includes: Within each measurement cycle, the control plane of the P4 programmable top-of-rack switch configures the message generation module to automatically generate probe message templates in RDMA unreliable datagram mode at a given cycle and injects them into the data plane pipeline. The data plane pipeline fills in the fields of the probe message templates based on the pre-configuration information of the management and control platform to form probe messages for in-rack and inter-rack measurements. The probe message includes a key field for controlling the measurement process, an identification field for distinguishing the measurement range, a control field for triggering multicast replication, a round identification field for identifying the measurement round, and a sequence identification field for identifying the measurement sequence. Data plane processing and distribution phase: The data plane selects the corresponding distribution strategy according to the type of probe packet to distribute the probe packet to the target host, and realizes multi-path traversal in cross-rack scenarios; Response collection and delay decomposition phase: The target host-side agent automatically responds to probe packets, and the source P4 programmable top-of-rack switch collects the response packets and records delay-related information; the response collection and delay decomposition phase specifically includes: The P4 programmable top-of-rack switch records the time when sending probe packets. After the probe packet arrives at the target host, the host-side agent reads the completion queue entry generated by the RNIC to obtain the time when the host received the probe packet. and the time when the host sends back the response message and will The value is written as the host-side latency into the response message; the source P4 programmable top-of-rack switch records the time when it receives the response message. And read the host-side latency carried in the response message, in order to... Calculate network round-trip latency ; Anomaly detection and reporting phase: Based on latency-related information analysis, when latency anomalies, packet loss, or unreachable paths are detected, the source P4 programmable top-of-rack switch records the anomaly information and reports it to the management and control platform. The management and control platform summarizes and analyzes the measurement data of the entire network to locate network performance bottlenecks and faults. Closed-loop operation phase: The above phases are executed cyclically according to the preset measurement cycle, forming a continuous monitoring and diagnosis closed-loop process.
2. The intelligent computing center network monitoring and diagnosis method based on a programmable switch according to claim 1, characterized in that, The detection task information includes a list of host addresses within the rack, a list of addresses between racks, a 5-tuple, and multicast group information generated from the list of host addresses within the rack. The 5-tuple includes the source IP address, source port number, destination IP address, destination port number, and transport layer protocol.
3. The intelligent computing center network monitoring and diagnosis method based on a programmable switch according to claim 1, characterized in that, The distribution strategy specifically includes: For rack-mounted detection scenarios, the management platform generates a multicast group based on the host address list information within the rack and sends the multicast group information to the P4 programmable top-of-rack switch, enabling it to achieve communication coverage of other hosts within the rack through a single detection message; For inter-rack probing scenarios, a set of cross-rack probing paths is generated based on the inter-rack address list information, and combined with the five-tuple information maintained by the management and control platform, to achieve communication measurement between multiple racks.
4. The intelligent computing center network monitoring and diagnosis method based on a programmable switch according to claim 1, characterized in that, The data plane processing and distribution stage specifically includes: The data plane pipeline of the P4 programmable top-of-rack switch parses the probe packets and determines the type of the probe packet based on the identification field carried in the probe packet to distinguish the measurement range: If it is a probe message within the rack, the probe message is distributed to each host within the rack through multicast replication based on the multicast group information pre-configured during the pre-configuration phase. If it is an inter-rack probe packet, the management platform will determine the source IP address, source port number, and destination IP address fields based on the inter-rack address list to generate and distribute five-tuple information. Based on the five-tuple information, the probe packet will be forwarded to the remote P4 programmable top-of-rack switch to achieve traversal of multiple equal-cost multi-path forwarding paths. At the same time, the remote P4 programmable top-of-rack switch will forward the probe packet to the corresponding target host in turn based on the sequence identifier information carried in the probe packet.
5. The intelligent computing center network monitoring and diagnosis method based on a programmable switch according to claim 1, characterized in that, In-rack communication specifically includes the following processes: After identifying a probe message within the rack, the P4 programmable top-of-rack switch updates the current measurement round information and records the transmission timestamp. The probe messages are simultaneously distributed to all hosts within the rack via multicast replication. After receiving the response message, the receiving timestamp is recorded and the network round-trip delay between each host is calculated by combining the host-side delay carried in the response message. If no response message is received within a given time, it is determined to be a packet loss event. By utilizing the packet generation and distribution mechanism of the P4 programmable top rack switch, probe packets are generated and distributed by the switch side, enabling full interconnection measurement without the host actively generating probe packets.
6. The intelligent computing center network monitoring and diagnosis method based on a programmable switch according to claim 1, characterized in that, Inter-rack communication specifically includes the following processes: The source P4 programmable top-of-rack switch sends probe packets to the remote P4 programmable top-of-rack switch. The remote P4 programmable top-of-rack switch determines the target host address based on the sequence identifier information carried in the probe packet using a mapping method based on counting and modulo, and forwards the probe packets to the corresponding target host in turn. Within multiple measurement cycles, the five-tuple adjustment mechanism sequentially traverses and covers all hosts in the remote rack and different equivalent multipath forwarding paths.
7. A network monitoring and diagnostic device for intelligent computing centers based on a programmable switch, characterized in that, The system includes a management and control platform, a host-side agent, and one or more P4 programmable top-of-rack switches. The management and control platform is a host connected to the network, used to store and update host communication information and network topology information, summarize and analyze network-wide measurement data, and send probe task information to the P4 programmable top-of-rack switches, including rack-side host address list information, inter-rack address list information, 5-tuple information, and multicast group information generated from the rack-side host address list information. The host-side agent is used to obtain host-side timestamps and respond to probe packets. The P4 programmable top-of-rack switches and the host form a leaf-spine network architecture. The P4 programmable top-of-rack switches, the management and control platform, and the host-side agent are used to collaboratively execute the intelligent computing center network monitoring and diagnosis method based on programmable switches as described in any one of claims 1-6, to achieve probe packet generation, path traversal, latency calculation, and anomaly detection functions.
8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by the P4 programmable rack top switch, the management platform, and / or the host-side agent, is used to implement the intelligent computing center network monitoring and diagnosis method based on the programmable switch as described in any one of claims 1-6.
Citation Information
Patent Citations
Method, apparatus, device, medium and program product for host probing
CN118984289A
Method for verifying data center network performance
US20230139774A1
