Device and method for ROCE network exit polarization analysis and congestion detection

Abstract

The invention relates to the technical field of electronic communication, in particular to a device and a method for ROCE network exit polarization analysis and congestion detection, which comprise a switch and a matched method. The switch is provided with a CPU and a switch chip, the switch chip is interconnected with a high-speed bandwidth through a double CPI interface, the switch chip cuts off a service message into front 64 bytes and sends the front 64 bytes to the CPU, the CPU polls a network card queue to receive data in a user mode by using a DPDK module, analyzes the message through a flow analysis application and stores the message in an annular buffer area with a pre-distributed 512M memory, and the message is sent to the switch chip through the DPDK module. The device supports a local analysis and remote analysis dual-mode mode, can extract multi-dimensional flow information such as a physical port number and a QP number, and generates a flow thermodynamic diagram to visually display an RDMA flow state. According to the scheme, through microsecond-level low-delay acquisition, QP granularity monitoring and efficient memory management, the problems that a traditional scheme is high in delay, coarse in sampling and incapable of accurately detecting ROCE congestion are solved.

Description

Technical Field

[0001] The present invention relates to the field of electronic communications technology, and in particular to a device and method for polarization analysis and congestion detection in a ROCE network egress. Background Art

[0002] As data centers continue to expand, applications such as high-performance computing (HPC) and artificial intelligence (AI) training are placing higher demands on network performance. RDMA over Converged Ethernet (ROCE), as an efficient data transmission protocol, has been widely used in these scenarios. However, traditional traffic monitoring and analysis methods have exposed numerous shortcomings under the high-bandwidth and low-latency requirements, seriously impacting network stability and performance.

[0003] Traditional network traffic analysis methods in existing technologies and their shortcomings

[0004] Reliance on centralized servers for analysis:

[0005] Traditional network traffic analysis methods typically rely on switches to transmit traffic data to a server for centralized analysis. Specifically, switches transmit traffic information (such as flow tables and traffic statistics) to a traffic analysis server via protocols such as sFlow or NetFlow.

[0006] Although this solution can achieve traffic monitoring and performance analysis to a certain extent, it has significant shortcomings.

[0007] Latency is too high:

[0008] Traditional network traffic analysis methods typically forward sampled data from a switch to an analysis server after traffic occurs, where the server then parses, processes, and calculates the data. This data transmission process incurs significant latency, especially in high-bandwidth, high-traffic network environments, where data transmission delays can reach several milliseconds or even higher.

[0009] The Remote Direct Memory Access (RDMA) protocol, especially large, long-lasting flows, consumes significant network bandwidth. The latency of traditional solutions makes it impossible to monitor and analyze the impact of these flows in real time. This results in an inability to detect congestion caused by traffic bottlenecks in the RDMA network, impacting RDMA performance and throughput.

[0010] Sampling ratio is too low:

[0011] A low sampling ratio can prevent the accurate capture of critical network traffic information, especially in high-bandwidth, high-traffic scenarios. Insufficient traffic sampling can cause some large flows to be missed, preventing timely detection of network congestion, bottlenecks, or abnormal traffic, impacting network performance and service stability.

[0012] Unable to analyze RDMA congestion in real time:

[0013] RDMA is a protocol with extremely high latency requirements, and is widely used for high-speed data transmission, particularly in high-performance computing (HPC) environments and AI training clusters. Large traffic flows often put pressure on network loads, but due to the latency of traditional methods, traffic data sampled on switches must be transmitted to servers for analysis, making it difficult to detect ECMP (Equal Cost Multi-Path) egress traffic congestion or RDMA traffic bottlenecks in a timely manner.

[0014] This will lead to the continuation of network bottlenecks and the inability to adjust paths or optimize traffic in a timely manner, thus affecting the computing performance of the entire cluster.

[0015] Unable to match QP granularity:

[0016] Traditional sFlow technology has limitations in monitoring at the Queue Pair (QP) granularity, especially in high-performance computing (HPC) and remote direct memory access (RDMA) environments. sFlow packets only contain L2 / L3 / L4 header information but lack RDMA protocol-related fields, making it impossible to effectively monitor traffic at the QP granularity.

[0017] High traffic transmission pressure:

[0018] Traditional network traffic analysis methods typically transfer all sampled data from switches to servers for centralized processing. In AI clusters, the numerous switches and massive network traffic result in a massive amount of traffic data transmission. This is especially true when using RDMA networks or large data transmission, where the volume of sampled traffic increases dramatically, easily overloading server processing capacity.

[0019] The large number of switches and the large and diverse traffic mean that the server not only has to process the data packets sent by all switches, but also needs to perform complex analysis, statistics and calculations, which greatly increases the burden on the server.

[0020] sFlow traffic consumes service bandwidth:

[0021] In high-performance computing (HPC) scenarios, network ports typically operate at full bandwidth. The traditional sFlow protocol sends collected traffic data to the sFlow server for analysis, which consumes service bandwidth and affects network performance.

[0022] sFlow generates monitoring data by sampling data packets. However, in an HPC environment, additional sFlow traffic may compete with business traffic for bandwidth, increasing latency and packet loss, especially affecting HPC applications with strict low-latency requirements. Summary of the Invention

[0023] (1) Technical problems solved

[0024] In view of the shortcomings of the existing technology, the present invention provides a device and method for polarization analysis and congestion detection in a ROCE network egress.

[0025] (2) Technical solution

[0026] To achieve the above objectives, the present invention provides the following technical solutions: A device for ROCE network egress polarization analysis and congestion detection of the present invention includes a switch, the switch is configured with a CPU and a switching chip, the CPU is configured with a flow analysis application, the switch is configured with a CPI interface, the CPI interface is interconnected with the CPU via a high-speed bandwidth interface, the CPU is configured with a DPDK module for directly polling a network interface card receive queue in user mode, the switching chip is configured with a message truncation function for truncating an uploaded service message into bytes and sending them to the CPU, and the flow analysis application is used to parse the received service message.

[0027] Preferably, the DPDK module includes a ring buffer, which is used to store business messages parsed by the analysis flow analysis application.

[0028] Further preferably, at least two CPI interfaces are configured.

[0029] Again preferably, the message interception function is to cut the service message sent to the CPI interface into the first 64 bytes before sending it up.

[0030] Preferably, the traffic analysis application supports a local analysis mode and a remote analysis mode.

[0031] Further preferably, the ring buffer adopts a memory preprocessing mechanism, pre-allocates 512M memory, and divides it into 32-byte elements for storing flow information.

[0032] Again preferably, the traffic analysis application extracts and records flow information, including physical port number, protocol source port, protocol destination port, protocol source IP, protocol destination IP, protocol type and target QP number.

[0033] Preferably, a method for ROCE network egress polarization analysis and congestion detection specifically includes the following steps:

[0034] Step S1: Configure the CPI port of the switch chip and truncate the service message to 64 bytes;

[0035] Step S2: Obtain and process the truncated message directly from the network card receive queue through the DPDK module;

[0036] Step S3: Use the traffic analysis application AIFlowAnalyzer to parse the message and store it in the ring buffer;

[0037] Step S4: Select local analysis mode or remote analysis mode for data analysis as needed.

[0038] Further preferably, in step S4, the local analysis mode queries the data in the ring buffer through commands and performs analysis and processing; the remote analysis mode sends the locally stored data to the remote server through gRPC for analysis and processing.

[0039] Again preferably, step S5 is further included: generating a traffic distribution heat map, and visually displaying the real-time status, performance indicators and abnormal conditions of the RDMA flow through the user interface to help operation and maintenance personnel quickly locate problems and optimize network performance.

[0040] (3) Beneficial effects

[0041] Compared with the prior art, the present invention provides a device and method for ROCE network egress polarization analysis and congestion detection, which has the following beneficial effects:

[0042] Ultra-low latency data acquisition

[0043] Through high-speed interconnection with the CPU via the CPI interface, combined with the switch chip's 64-byte message truncation function, only critical flow identification information is transmitted, reducing data volume by approximately 90%. The CPU utilizes the DPDK module to directly poll the network card queue in user mode, avoiding kernel context switching overhead. Data collection latency is reduced to microseconds (<10μs), enabling real-time capture of congestion bursts of large flows in ROCE networks, addressing the detection lag caused by traditional solutions.

[0044] QP granularity precise monitoring

[0045] Flow analysis applications deeply analyze RDMA protocol fields, extracting key information such as the target QP number, which is not available with traditional sFlow technology, enabling queue-pair (QP)-level monitoring of ROCE traffic. By recording multi-dimensional flow information such as physical port numbers and source / destination IP addresses, the distribution of RDMA flows across ECMP multipaths can be accurately tracked, locating egress polarization issues caused by uneven load.

[0046] High-performance memory management

[0047] The ring buffer uses 512MB of memory pre-allocation and a 32-byte element partitioning mechanism, combined with a lock-free programming model, to avoid dynamic memory allocation latency and lock contention overhead. It can support a throughput of 1 million packets per second on a quad-core CPU, with memory access latency below 50ns, meeting the real-time analysis requirements of high-traffic scenarios such as AI training clusters, and improving memory utilization by 60%.

[0048] Bimodal analysis architecture

[0049] Local analysis mode: Command line query response time is less than 100ms, suitable for rapid on-site diagnosis.

[0050] Remote Analysis Mode: gRPC compresses data transmission (reducing bandwidth usage by 60%) and integrates with the cloud platform to enable historical trend analysis and machine learning modeling. These two modes work together to balance real-time edge performance with in-depth analysis capabilities at the center.

[0051] Low resource usage and high reliability

[0052] Dual CPI interface redundancy design: automatic switching when a single interface fails to ensure data collection continuity.

[0053] Memory pool reuse technology: The object pool reuses parsed objects, reducing garbage collection frequency by 75%, and the overall system resource usage rate is less than 5%, with negligible impact on the production environment.

[0054] Visual decision support

[0055] The traffic distribution heat map intuitively displays the export polarization status. Combined with anomaly marking and performance indicator analysis, the network congestion location time is shortened from hours to minutes, significantly improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the switch structure of the present invention;

[0057] Figure 2 This is a schematic diagram of the ring buffer of the present invention;

[0058] Figure 3 Schematic diagram of the system workflow of the present invention;

[0059] Figure 4 This is a schematic diagram of the local analysis process of the present invention;

[0060] Figure 5 This is a schematic diagram of the remote analysis process of the present invention;

[0061] Figure 6 This is a schematic diagram of the visual display process of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] See also Figure 1-6 , a device for ROCE network egress polarization analysis and congestion detection of the present invention includes a switch, the switch is configured with a CPU and a switching chip, the CPU is configured with a flow analysis application, the switch is configured with a CPI interface, the CPI interface is interconnected with the CPU through a high-speed bandwidth interface, the CPU is configured with a DPDK module for directly polling the network interface card receive queue in user mode, the switching chip is configured with a message truncation function for truncating the uploaded service message into bytes and sending them to the CPU, and the flow analysis application is used to parse the received service message.

[0064] This technical solution provides a device and method for RDMA network egress polarization analysis and congestion detection, aiming to address the high latency, low sampling rate, and inability to perform real-time analysis inherent in traditional traffic monitoring and analysis solutions. By configuring a CPU, switching chips, and a series of optimized modules on a switch, this device achieves efficient data collection, processing, and analysis.

[0065] Hardware-accelerated data acquisition

[0066] The switch chip truncates service packets to the first 64 bytes via the CPI interface before sending them to the CPU, avoiding the bandwidth pressure caused by transmitting the entire packet. The CPU uses a dedicated DPDK module to directly poll the NIC receive queue in user mode, bypassing the traditional kernel network stack and reducing data collection latency to microseconds.

[0067] The DPDK module runs primarily on Linux and is a collection of function libraries and drivers for fast packet processing. This can significantly improve data processing performance and throughput, and enhance the efficiency of data plane applications.

[0068] The DPDK module uses polling rather than interrupts to process packets. Upon receiving a packet, the network card driver, overloaded by the DPDK module, does not notify the CPU via an interrupt. Instead, it directly stores the packet in memory and passes it to the application layer software for direct processing through the DPDK interface. This saves a significant amount of CPU interrupt time and memory copy time.

[0069] Real-time stream information extraction

[0070] The flow analysis application AIFlowAnalyzer parses the truncated packets, extracts key flow information (such as physical port number, source / destination IP address, and QP number), and stores it in a ring buffer. Through a memory preprocessing mechanism, the ring buffer pre-allocates fixed-size memory, supports lock-free read and write operations, and ensures low-latency data storage at high throughput.

[0071] The AI Flow Analyzer flow analysis application, through its powerful traffic capture, parsing, and analysis capabilities, provides significant value in multiple key areas, including but not limited to high-performance computing, AI training, data center operations, enterprise network management, network security, and IoT device management. Its key advantages lie in its ability to provide real-time monitoring, precise analysis, and intelligent optimization, helping enterprises improve network performance, ensure service quality, and enhance security. The configuration and usage strategies of the AI Flow Analyzer flow analysis application can be flexibly adjusted to achieve optimal results based on the specific needs of different application scenarios.

[0072] Bimodal analysis architecture

[0073] The system supports two modes: local analysis and remote analysis:

[0074] Local analysis: Directly query the ring buffer data through the command line, suitable for quick diagnosis and temporary analysis

[0075] The local analysis process is as follows: the user initiates a local analysis command, reads data from the ring buffer, calculates traffic statistics, detects polarization and congestion, generates an analysis report, and outputs it to a user interface, such as the CLI or web interface.

[0076] Remote analysis: Transmit data to a remote server via gRPC and leverage cloud computing resources for complex analysis and visualization.

[0077] The remote analysis process is as follows: initialize the gRPC client - read data in batches from the ring buffer - serialize and compress the data - send the data to the remote server - in case of failure, the data is retained in the buffer for retry.

[0078] gRPC is a modern, open-source, high-performance remote procedure call (RPC) framework that can run in any environment. It can efficiently connect services within and across data centers, supporting load balancing, tracing, health checks, and authentication. It is also suitable for the last mile of distributed computing, connecting devices, mobile applications, and browsers to backend services.

[0079] Polarization analysis and congestion detection

[0080] By continuously monitoring the traffic distribution of each egress port, the system calculates traffic polarization indices (such as standard deviation and coefficient of variation) to identify abnormal traffic distribution. Combined with RDMA-specific metrics (such as QP queue depth and retry rate), it provides microsecond-level congestion warnings.

[0081] Various optimized technologies

[0082] CPI interface dual-channel design

[0083] It is preferred to configure at least two CPI interfaces and interconnect them with the CPU via high-speed bandwidth (such as 2×100Gbps) to ensure message transmission bandwidth and redundancy.

[0084] Message truncation optimization

[0085] The switching chip intercepts the first 64 bytes of the service message (including the ROCE header, IP and transport layer key information), retaining the necessary fields while avoiding invalid data transmission.

[0086] Efficient ring buffer management

[0087] 512MB of memory is pre-allocated and divided into 32-byte elements. Each element stores the key information of a single flow (such as physical port number, protocol source / destination IP, QP number, etc.) and supports lock-free read and write operations.

[0088] Dual-mode analysis adaptation

[0089] Local analysis mode: parse the ring buffer data in real time and query traffic statistics directly through commands.

[0090] Remote analysis mode: Push data to a remote server via the gRPC protocol, supporting large-scale historical data analysis and AI modeling.

[0091] Visualization and anomaly detection

[0092] Generates traffic distribution heat maps based on parsed data, and displays RDMA flow status, throughput, latency, and congestion events in real time through the user interface (UI), helping operations personnel quickly locate network bottlenecks.

[0093] The visualization process is: Get analysis structure - generate flow thermal Figure 1 Mark abnormal traffic points - calculate performance indicators - update implementation monitoring interface - provide interactive query function.

[0094] Detailed workflow

[0095] Step S1: Configure the switch chip CPI interface

[0096] Operation: Configure the message truncation function on the CPI interface of the switch chip to truncate the sent service message to the first 64 bytes.

[0097] Purpose: To reduce the amount of data processed by the CPU and improve processing efficiency.

[0098] Step S2: directly obtain and process the truncated message from the network card receiving queue through the DPDK module: directly obtain and process the truncated message from the network card receiving queue through the DPDK module.

[0099] Purpose: To reduce data transmission delay and improve data processing speed through direct user-mode polling.

[0100] Step S3: Use the traffic analysis application AIFlowAnalyzer to parse the message and store it in the ring buffer

[0101] Operation: Run the traffic analysis application AIFlowAnalyzer to parse the received service packets, extract key flow information, and store it in the ring buffer.

[0102] Purpose: To parse business messages in detail, extract key information and store it for subsequent analysis.

[0103] Step S4: Select local analysis mode or remote analysis mode for data analysis as needed

[0104] Local analysis mode:

[0105] Operation: Use commands to query the data in the ring buffer and analyze and process it.

[0106] Purpose: Complete data analysis locally, suitable for scenarios that require rapid response.

[0107] Remote analysis mode:

[0108] Operation: Send locally stored data to a remote server via gRPC for analysis and processing.

[0109] Purpose: Utilize the powerful computing power of remote servers to meet the needs of large-scale data analysis.

[0110] Step S5: Generate a traffic distribution heat map and intuitively display the real-time status, performance indicators and abnormal conditions of the RDMA flow through the user interface

[0111] Operation: Generate a traffic distribution heat map based on the analysis results, and intuitively display the real-time status, performance indicators and abnormal conditions of the RDMA flow through the user interface.

[0112] Purpose: To help operation and maintenance personnel quickly locate problems and optimize network performance.

[0113] Reference application examples are:

[0114] To build an AI cluster environment, use 4 servers, each with 8 GPU cards and 8 RNIC cards. 2 switches are deployed at the leaf layer and 2 switches at the spine layer. All switches are designed using the above technical solution.

[0115] Deploy traffic analysis applications on the four switches above;

[0116] Configure the traffic analysis application sampling ratio to 1:20, the statistical granularity to 1ms, and the analysis mode to local.

[0117] Enable traffic analysis application flow sampling and sample the outbound traffic through the 32 ECMP upstream ports according to the above sampling ratio;

[0118] The traffic analysis application analyzes the sflow packets received by the DPDK module, extracts the flow (source IP, source port -> destination IP, destination port (4791)) and port map, counts each flow, and stores the statistical information in a local buffer.

[0119] You can use the show AIFlowAnalyzer 1all command to view traffic statistics for every 1 lm in the last second.

[0120] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A device for ROCE network egress polarization analysis and congestion detection, characterized in that: The invention comprises a switch, wherein the switch is equipped with a CPU and a switching chip, the CPU is equipped with a flow analysis application, the switch is equipped with a CPI interface, the CPI interface is interconnected with the CPU via a high-speed bandwidth interface, the CPU is equipped with a DPDK module for directly polling a network interface card receiving queue in user mode, the switching chip is equipped with a message truncation function for truncating uploaded service messages into bytes and sending them to the CPU, and the flow analysis application is used to parse received service messages.

2. The device for ROCE network egress polarization analysis and congestion detection according to claim 1, characterized in that: The DPDK module includes a ring buffer, which is used to store business messages parsed by the analysis flow analysis application.

3. The device for ROCE network egress polarization analysis and congestion detection according to claim 2, characterized in that: There are at least two CPI interfaces.

4. The device for ROCE network egress polarization analysis and congestion detection according to claim 3, characterized in that: The message interception function is to cut the service message sent to the CPI interface into the first 64 bytes before sending it up.

5. The device for ROCE network egress polarization analysis and congestion detection according to claim 4, characterized in that: The traffic analysis application supports local analysis mode and remote analysis mode.

6. The device for ROCE network egress polarization analysis and congestion detection according to claim 5, characterized in that: The ring buffer adopts a memory preprocessing mechanism, pre-allocates 512M memory, and divides it into 32-byte elements for storing flow information.

7. The device for ROCE network egress polarization analysis and congestion detection according to claim 6, characterized in that: The traffic analysis application extracts and records flow information, including physical port number, protocol source port, protocol destination port, protocol source IP, protocol destination IP, protocol type, and target QP number.

8. A method for ROCE network egress polarization analysis and congestion detection, characterized in that: The method is an application method of the device for ROCE network egress polarization analysis and congestion detection according to any one of claims 1 to 7, and specifically comprises the following steps: Step S1: Configure the CPI port of the switch chip and truncate the service message to 64 bytes; Step S2: Obtain and process the truncated message directly from the network card receive queue through the DPDK module; Step S3: Use the traffic analysis application AIFlowAnalyzer to parse the message and store it in the ring buffer; Step S4: Select local analysis mode or remote analysis mode for data analysis as needed.

9. The method for ROCE network egress polarization analysis and congestion detection according to claim 8, characterized in that: In step S4, the local analysis mode queries the data in the ring buffer through commands and performs analysis and processing; the remote analysis mode sends the locally stored data to the remote server through gRPC for analysis and processing.

10. The method for ROCE network egress polarization analysis and congestion detection according to claim 8, characterized in that: The process also includes step S5: generating a traffic distribution heat map and visually displaying the real-time status, performance indicators, and abnormal conditions of the RDMA flow through the user interface to help operation and maintenance personnel quickly locate problems and optimize network performance.

Images