A method for monitoring and evaluating performance of an intelligent network card system

Abstract

This invention relates to a method for performance monitoring and evaluation of intelligent network interface card (NIC) systems, applicable to the fields of computer network and performance monitoring technology. Addressing the technical problems of low transparency in the internal execution of regular expressions and data compression acceleration engines in intelligent NICs, and the difficulty in accurately quantifying contention for on-chip multi-dimensional system resources (CPU, cache, memory, etc.), this invention instrumentes and encapsulates the accelerator's task submission and retrieval interfaces, records high-precision timestamps of tasks entering and leaving the hardware engine to calculate accurate latency, and eliminates disk I / O blocking through a lock-free circular queue for asynchronous disk write-to-disk operation. Simultaneously, this invention proposes a performance counter monitoring method based on time-division multiplexing, mapping low-level hardware events to event selection registers for round-robin sampling, and combining this with a file system polling mechanism to obtain real-time micro and macroscopic states of the CPU, last-level cache, memory, and NIC. Through instrumentation, encapsulation, and polling techniques, this invention achieves fine-grained joint monitoring of intelligent NIC accelerators and global system resources.

Description

Technical Field

[0001] This invention relates to the field of smart network interface card (NIC) performance analysis technology, specifically to a method for monitoring the traffic of smart NIC hardware accelerators and the multi-dimensional system resource load. Background Technology

[0002] With the explosive growth of artificial intelligence (AI) services, network traffic in intelligent computing centers is becoming increasingly massive and complex. Traditional network data processing architectures centered around central processing units (CPUs) face severe computational and memory bandwidth bottlenecks when dealing with the demands of ultra-large-scale, low-latency networks. To alleviate the processing pressure on the host, smart network interface cards (SmartNICs), especially architectures represented by data processing units (DPUs), have become key infrastructure for next-generation intelligent computing centers.

[0003] As a typical System-on-a-Chip (SoC), a smart network interface card (NIC) not only integrates a multi-core general-purpose processor and a high-speed network switching module, but more importantly, it integrates a large number of domain-specific accelerators, such as the regular expression matching accelerator RegEx and the data compression / decompression accelerator Compress. These hardware accelerators implement complex network computing tasks through dedicated hardware circuits, enabling them to handle specific data-intensive loads with extremely high efficiency, significantly improving the overall system throughput and reducing processing latency.

[0004] However, in actual deployment and operation and maintenance of high-concurrency network environments, monitoring the system resource performance of smart network interface cards (NICs) faces severe challenges.

[0005] First, the black-box nature of hardware accelerators leads to significant observability gaps. Unlike general-purpose processing cores, the internal logic of hardware such as RegEx and Compress is completely opaque to upper-layer software. Current smart NIC hardware does not provide comprehensive hardware performance counters for these accelerators, nor does it have a standard API interface for directly reading their internal task queue depth, real-time processing latency, or concurrency status. This makes it impossible for monitoring programs to obtain the number of tasks being processed internally by the hardware accelerator, and system administrators and developers cannot easily perceive in real time whether the accelerator is reaching its throughput limit.

[0006] Secondly, the competitive effect of general-purpose system resources within smart NICs is difficult to quantify. In the smart NIC SoC architecture, multiple general-purpose processor cores share the final-level cache and memory controller bandwidth. When the network load is high-concurrency, the service processes running on different cores can cause severe resource competition. For example, frequent memory access may lead to memory bandwidth saturation or a surge in cache misses, which in turn can cause processor pipeline stalls. Although the underlying hardware performance monitoring unit exists, the number of physical registers is extremely limited, and there is a lack of proactive event reporting mechanisms, making it difficult to monitor multiple dimensions of underlying events simultaneously.

[0007] Finally, the silo effect of heterogeneous resource consumption makes it difficult to pinpoint performance bottlenecks. Existing monitoring tools often only obtain isolated application-layer statistics or macro-level operating system metrics, failing to align asynchronous API events of applications accessing hardware accelerators with underlying hardware performance counters and file system resource status at a unified microsecond-level time dimension. When system performance fluctuations occur, operations personnel cannot determine whether the bottleneck lies in resource contention within general-purpose processing cores or in specific hardware accelerators. Summary of the Invention

[0008] This invention aims to solve the problems in the prior art where the internal state of the smart network card hardware accelerator is not visible due to its black-box operation, the difficulty in accurately quantifying the resource contention of the on-chip multidimensional system, and the lack of temporal correlation of heterogeneous monitoring data, and provides a method for multidimensional performance monitoring of smart network card system resources.

[0009] This invention provides a method for multi-dimensional performance monitoring of intelligent network interface card (NIC) system resources, which specifically includes the following steps:

[0010] Step 1: Construct an accelerator traffic interception mechanism that integrates event-driven and asynchronous recording. A monitoring proxy layer is built between the application layer and the underlying hardware driver. For applications based on DPDK (Data Plane Development Kit), APIs for submitting tasks and retrieving results to the hardware accelerator are intercepted through code instrumentation, such as the enqueue and dequeue functions of the regular expression accelerator. For applications based on closed-source frameworks, the underlying task interaction APIs are intercepted through interface encapsulation. When the business program calls the interface, the current high-precision timestamp, thread identifier, and queue identifier are obtained, and the total byte size of the submitted or retrieved task is calculated. A globally unique sequence number range is assigned to each hardware processing batch, thereby non-intrusively obtaining the actual inflow and outflow status of each work queue.

[0011] Step two involves introducing a lock-free circular queue to decouple the physical threads of the monitoring data. To eliminate the blocking impact of high-frequency acquisition of high-precision timestamps and log serialization on the normal forwarding rate of network packets, the timestamps, data sizes, execution statuses, and globally unique sequence number ranges obtained in Step one are packaged into a monitoring event structure. This monitoring event structure is then written into a pre-allocated lock-free circular queue using a memory copy method. An independent background monitoring thread in the control plane is configured, which asynchronously polls the monitoring event structure from the lock-free circular queue and performs disk I / O operations on the logs.

[0012] Step 3: Implement time-division multiplexing-based sampling of the underlying hardware performance monitoring unit. Addressing the limitations of the number of system performance monitoring registers on smart network cards and the lack of an interrupt reporting mechanism, the underlying hardware events to be monitored are divided into several mutually exclusive groups according to physical resource domains. These include: a memory subsystem read / write event group for evaluating memory channel load; a last-level cache hit / miss event group for evaluating data affinity; and an I / O device read / write request group for evaluating on-chip data flow pressure. Within an independent monitoring thread, these groups are sequentially traversed at fixed time slices. The event code of the current group is written to the hardware event selection register. After enabling the register, the performance counter baseline value is read, and the register is read again after the sampling window ends to calculate the increment of the hardware event. This process is repeated to achieve time-division multiplexing monitoring of multi-dimensional underlying metrics.

[0013] Step four involves implementing dynamic collection of basic system resources based on virtual file system polling. This is done by periodically calling the operating system's underlying virtual file system interface to collect macroscopic system resource load data. Specifically, this includes: polling the processor status interface to obtain the overall CPU utilization and single-core load; polling the memory status interface to obtain the total system memory usage and the allocated and remaining number of huge pages; and polling the network status interface to obtain the receive and send byte rates, packet rates, and packet loss statistics for physical and virtual network interface cards (NICs).

[0014] Step 5: Perform time-series alignment and joint presentation of multi-dimensional heterogeneous resource states. The accelerator event stream output from Step 2, the underlying hardware event increments acquired in Step 3, and the macroscopic system resource data collected in Step 4 are uniformly abstracted and extracted into structured resource state vectors containing resource domain features, high-precision timestamps of sampling action triggers, and specific resource indicators. All resource state vectors are placed in the global clock domain, arranged in ascending order by timestamp, and fused to reconstruct a dynamic joint timeline of smart NIC task processing and multi-dimensional system resource consumption, used to assist in locating performance bottlenecks.

[0015] In summary, the innovative points of this invention are as follows:

[0016] (1) Breaking the black box limitation and controlled low-overhead monitoring: In response to the problem of lack of available performance registers in smart network card hardware accelerators, this invention intercepts interactive traffic by building a proxy layer in the upper-layer API and uses global sequence number ranges for batch aggregation and recording; at the same time, it innovatively introduces a lock-free ring queue to achieve complete physical decoupling between high-frequency data plane acquisition and control plane disk I / O. By binding the system resource monitoring program to run on an independent processor core, the interference to the processing performance of the main business program is greatly reduced, and online load monitoring is realized.

[0017] (2) Multi-dimensional sampling that overcomes the physical limitations of the underlying hardware: In response to the severe architectural defect of insufficient number of physical counters in the PMU of the smart network card, this invention designs a group polling scheduling mechanism based on time-division multiplexing. Without modifying the underlying hardware logic, by using time-slice round-robin multiplexing of limited physical registers, it successfully achieves concurrent and continuous sampling of multi-dimensional micro performance indicators such as the last-level cache, memory read / write bandwidth, and I / O bus pressure.

[0018] (3) Panoramic packet-resource joint diagnostic capability: This invention completely breaks down the silos of heterogeneous monitoring data, and achieves high-precision time-series fusion of discrete accelerator asynchronous processing micro-latency with the system-wide macro-resource status such as CPU full load and cache pollution on a unified time axis. This provides complete contextual data support for determining the cause of performance interference under complex heterogeneous network loads, so that bottleneck location no longer depends on prior experience. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture of the intelligent network card multi-dimensional system resource monitoring in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of the hardware accelerator performance monitoring mechanism in an embodiment of the present invention;

[0021] Figure 3 This is a diagram showing the monitoring results of the underlying hardware performance counter based on time-division multiplexing in an embodiment of the present invention;

[0022] Figure 4 This is a graph showing the results of polling and monitoring basic system resources in an embodiment of the present invention.

[0023] Figure 5 This is a diagram showing the traffic monitoring results of the physical network interface of the smart network card in an embodiment of the present invention. Detailed Implementation

[0024] This invention provides a method for multi-dimensional performance monitoring of intelligent network interface card (NIC) system resources, the process of which is as follows: Figure 1As shown. Hardware accelerators on smart network interface cards (NICs) typically operate in black-box mode, and on-chip system resource contention is complex. To achieve comprehensive performance evaluation with extremely low overhead, this embodiment relies on the Nvidia BlueField-2 smart NIC hardware platform and its supporting software stack. The specific implementation process includes the following steps:

[0025] Step 1: Build a proxy layer to intercept online traffic. Introduce a monitoring proxy layer between the application layer business logic and the underlying network card driver, such as... Figure 2 As shown. This embodiment provides customized interception for two typical smart NIC software development frameworks: For open-source applications based on DPDK, code instrumentation is employed. Taking the regular expression accelerator RegEx as an example, the `rte_regexdev_enqueue_burst` (task enqueueing) and `rte_regexdev_dequeue_burst` (task dequeueing) functions in the DPDK driver source code are directly modified. When the application submits a task to the work queue, a high-precision clock is called at the beginning of the driver function to obtain the initial timestamp. Subsequently, the length of the effective data payload to be processed in this batch is accumulated by traversing the passed `rte_mbuf` task list. At the same time, a global atomic counter is maintained to assign a continuous sequence number range (e.g., from ID 1000 to ID 1063) to the successfully submitted tasks in this batch. For closed-source applications based on DOCA (Data Center Infrastructure-on-a-Chip Architecture), since the underlying driver cannot be modified, interface secondary encapsulation technology is used. Developers call the re-encapsulated interface with the same name. Internally, this interface prioritizes recording the system's high-precision timestamp of the submission action trigger moment and extracts the actual length of the task buffer from the doca_buf structure. Then, it calls the underlying raw DOCA API to complete the actual hardware deployment. Through this mechanism, regardless of how the smart NIC accelerator schedules internally, the proxy layer can accurately capture the amount and timing of the task's entry data flow to the hardware circuit, as well as the output results and timing returned from the hardware circuit.

[0026] Step two involves asynchronously persisting monitoring data to disk using a lock-free circular queue. In extremely high-concurrency scenarios with 10 Gigabit or even 100 Gigabit network throughput, if the proxy layer directly calls the file system API for log printing while intercepting data, the resulting system call overhead and disk I / O blocking can cause a precipitous drop in packet forwarding rate. Therefore, this embodiment pre-allocates a contiguous shared memory pool in memory and constructs a lock-free circular queue with a single producer and single consumer. The data intercepted by the proxy layer in step one is quickly populated into a lightweight monitoring event structure, which only contains: accelerator type, operation direction (enqueue / dequeue), thread PID, queue ID, task start and end sequence numbers, high-precision timestamp, and total batch size. The proxy layer uses memory barriers and atomic operations to push the structure to the tail of the queue, taking only nanoseconds. In the control plane, a separate low-priority background monitoring thread is configured. This thread continuously monitors the head pointer of the lock-free circular queue in a polling manner. When a new monitoring event structure is detected, it is extracted in batches, and subsequent structured log format conversion and disk write operations are performed. This mechanism completely eliminates the blocking effect of log writing to disk on the ARM core packet processing pipeline of the smart network card.

[0027] Step 3: Implement monitoring of the underlying hardware performance counters based on time-division multiplexing. The results are as follows: Figure 3As shown. The smart network interface card (NIC) integrates a general-purpose ARM core and a high-performance memory subsystem, which are configured with hardware performance monitoring units. However, the BlueField-2 chip can typically only activate a limited number of physical performance counter registers simultaneously, far fewer than the types of performance metrics that need to be observed, and these counters do not support proactive event reporting of state changes. Therefore, this embodiment adopts a time-division multiplexing-based multithreaded sampling algorithm. First, the complex on-chip hardware events are divided into multiple mutually exclusive logical groups according to the analysis domain. For example, group A focuses on memory, including memory controller byte read and write events; group B focuses on cache affinity, including L3 cache (LLC) hit and miss counts; group C focuses on interconnects, including I / O device read and write request groups. In an independent PMU monitoring thread, the program executes time-slice round-robin scheduling. Taking group A as an example, the monitoring thread first writes the specific event configuration codes for memory read and memory write to the event selection register through direct memory access or dedicated instructions. After enabling the counters, the current value of the data register is immediately read as the base. Subsequently, the thread is put to sleep for a fixed sampling time interval (e.g., 100 milliseconds). After being awakened from hibernation, the current value of the data register, Current, is read again, and the hardware event increment within the sampling period is calculated using Current-Base. After completing several consecutive samplings for group A, the monitoring thread actively disables the current counter, clears the event selection register, and then writes the event configuration code for group B to begin the next round of time-slice sampling. This mechanism, combining proactive fetching with time-division multiplexing, ensures comprehensive sampling of micro-resource states while meeting the physical limitations of the underlying architecture.

[0028] Step four involves implementing dynamic collection of basic system resources based on virtual file system polling, with the results as follows: Figure 3As shown. The reception, classification, offloading, and transmission of data packets within the smart NIC are highly dependent on CPU computing power, system memory capacity, and NIC physical bandwidth. This embodiment collects macro-level resource metrics by periodically retrieving virtual file system nodes exposed by the Linux operating system kernel: For CPU resources, the ` / proc / stat` file is polled to analyze clock ticks for user mode, system mode, idle mode, and interrupt handling. This not only calculates the overall CPU utilization of the SoC but also monitors the single-core load of each individual ARM core in detail. This is crucial for detecting severe load imbalances in DPDK applications caused by improper thread binding to cores. For memory resources, the ` / proc / meminfo` file is polled to track the system's regular memory usage and remaining amount, with a focus on analyzing and monitoring the status of large page memory. Data plane applications on the smart NIC heavily rely on large page memory to reduce TLB misses. Real-time monitoring of large page memory availability can effectively prevent application crashes or sudden drops in forwarding performance caused by large page resource exhaustion. For NIC resources, the ` / proc / net / dev` file is polled to obtain macro-level performance metrics of the underlying physical network interface. The throughput of the physical network interface card (NIC) is obtained by extracting the number of received bytes and the number of sent bytes and dividing by the sampling time difference; the number of received data packets is extracted to calculate PPS (packets per second); and packet loss statistics are monitored simultaneously. These metrics objectively reflect the physical layer load limit of the smart NIC's interaction with the external network.

[0029] Step 5: Multi-dimensional Data Time-Series Fusion. After acquiring the massive monitoring data from the above three dimensions, the system uniformly formats them. Whether it's accelerator operation events based on lock-free queue disk writes, memory bandwidth increments obtained through PMU polling, or single-core CPU load obtained from the / proc file system, all are abstracted into a standard resource state vector: {Resource Domain R, Action Triggering System Timestamp TU, Specific Resource Measurement Indicator V}. The system globally sorts all heterogeneous vectors according to the absolute timestamp TU. When performing joint diagnosis of network performance bottlenecks, if it is found that the accelerator task dequeue rate intercepted by the agent layer drops sharply within a certain period (latency suddenly increases), the operations and maintenance personnel can horizontally compare the macroscopic system resource vectors at the same moment. If, under the same clock domain, PMU monitoring shows a surge in L3 cache miss rate, and CPU monitoring shows that the specific ARM core responsible for task distribution reaches 100% load, then a precise conclusion can be drawn: the root cause of the current performance degradation is not that the processing capacity of the accelerator hardware circuit itself has reached a bottleneck, but rather that software-level blocking is caused by system-wide computing resource contention and cache pollution.

[0030] To verify the availability and low-overhead characteristics of the above monitoring mechanism, this embodiment performed a full-load limit test on a real BlueField-2 platform.

[0031] For the regular expression accelerator, four sets of complex regular expression rules covering real-world scenarios such as network protocols and security defenses were used for high-concurrency random text matching. Experimental statistics show that in full-throughput tests with four different rule sets, the average performance monitoring overhead of the regular expression accelerator was 8.4%, with the highest additional performance overhead not exceeding 9.6% (the baseline runtime was approximately 23.67 seconds, which slightly increased to 25.59 seconds after monitoring was enabled). For the compression / decompression accelerator, multiple sets of randomly generated text of different sizes were used for compression throughput limit tests. Experiments show that the average performance monitoring overhead of the compression / decompression accelerator was only 4.2%, with the highest additional performance overhead not exceeding 4.8% (the baseline runtime was 25.74 seconds, which increased to 26.94 seconds after monitoring was enabled).

[0032] Furthermore, regarding the monitoring mechanism for general system resources, this embodiment conducted practical tests and verifications on multi-dimensional system resources during operation based on DPDK. Test results show that the system resource monitoring mechanism based on time-division multiplexing and polling in this invention can accurately capture the system's runtime state and characterize it as the dynamic changes of multi-dimensional resources over time. Specifically, in terms of performance counter monitoring, the system successfully captured and displayed the dynamic changes in the last-level cache hit count, cache hit rate, memory read count, and read bandwidth of a specific processor core block over time; in terms of CPU monitoring, the system accurately recorded the overall processor utilization rate and the time-series fluctuations in the utilization rate of each independent single core; in terms of memory and network interface monitoring, it achieved real-time plotting of large page memory utilization rate, overall memory utilization rate, and the byte rate, data packet rate, and packet loss rate of the underlying physical network interface for receiving and sending. This fully demonstrates the accuracy of the mechanism in obtaining performance fluctuations of the underlying architecture, providing detailed data for multi-dimensional system resource correlation analysis. In terms of overhead control, considering that high-frequency polling of hardware performance counters and operating file system interfaces will introduce additional computation and I / O overhead, this embodiment binds the system resource monitoring program to run on a dedicated CPU core, thereby effectively reducing its interference with the processing performance of the main business program.

[0033] In summary, detailed experimental data demonstrates that the innovative design of the proxy layer interception, time-division multiplexing polling, and lock-free circular queue in this invention not only effectively breaks the black-box operation limitations of hardware accelerators and achieves precise alignment of microsecond-level data packets with multi-dimensional resource timelines, but also keeps the introduced performance overhead within a reasonable and controllable range: the additional performance overhead of hardware accelerator monitoring is controlled within 10%, and system resource monitoring achieves physical isolation by running on an independent CPU core, maximizing the protection of normal business operations.

Claims

1. A method for performance monitoring and evaluation of an intelligent network interface card (NIC) system, characterized in that, Includes the following steps: (1) Construct a system resource monitoring architecture that integrates event-driven and polling, and separate the API-level non-blocking event recording logic and the time-series polling logic of basic system resources under the monitoring system; (2) By instrumenting the task enqueue function of the Data Plane Development Kit (DPDK) or encapsulating the task submission and retrieval interface of the Smart NIC Software Development Kit (DOCA), the current timestamp, thread identifier, queue identifier, and task byte size are obtained when the application submits a task to the hardware accelerator or retrieves a result, thereby obtaining the real-time inflow and outflow of data from the hardware accelerator; for the regular expression accelerator, the timestamp is obtained before the task is submitted, and the total length of the actually submitted task is extracted from the data packet buffer. The task submission information is recorded by setting a globally unique ID for each task. (3) For hardware performance monitoring units in on-chip systems that do not support active event reporting, a time-division multiplexing-based sampling algorithm is adopted to divide the set of hardware events to be monitored into multiple groups. In each sampling period, the groups are traversed sequentially and the event codes are written into the hardware event selection register. The underlying performance indicators of the last-level cache, memory read and write bandwidth and I / O device read and write are obtained by reading the increment of the data register. Among them, the hardware event groups include memory subsystem read and write events, last-level cache hit and miss events, and I / O read and write request events that reflect the pressure of data flow in the on-chip network. (4) By polling the file system interface of the operating system kernel, the overall and single-core utilization of the central processing unit, the system memory and large page memory usage, and the throughput and packet loss rate of the network card physical interface are collected in real time; wherein, the file system interface includes the / proc / stat interface for obtaining CPU status, the / proc / meminfo interface for obtaining memory information, and the / proc / net / dev interface for obtaining network card performance. (5) Extract the common features of resource status in each dimension in the above steps, construct a resource status vector containing resource domain, sampling timestamp and resource index, and sort all vectors in ascending order of time to reconstruct the dynamic change timeline of smart network card system resources; (6) Based on the amount of data flowing into and out of the hardware accelerator obtained in step (2), the throughput extreme value of the accelerator is calculated using the round-robin scheduling model, and the performance service margin of the hardware accelerator is derived by combining the current throughput, so as to realize the real-time evaluation of the performance status of the hardware accelerator.

Images