An eBPF-based multi-task network traffic isolation collection method

Abstract

The application discloses an eBPF-based multi-task network traffic isolation collection method, allocates a unique task slot identifier for each concurrent test task running in the same process, establishes a corresponding relationship between a target IP address and the task slot identifier, constructs a statistical mapping in the kernel state with a composite key containing the target IP address and the task slot identifier as an index, realizes kernel state physical isolation of multi-task traffic data, in response to a network connection establishment event, stores the current connection and the task slot identifier at the connection establishment time in the connection anchoring mapping, in response to a data transmission or retransmission event, queries the task slot identifier of the connection anchor from the connection anchoring mapping, and updates the corresponding statistical data in the statistical mapping. The application solves the network traffic statistical cross-contamination problem of the same target IP in the multi-coroutine concurrent scene in the same process, and ensures the statistical attribution consistency of the whole life cycle of the connection.

Description

Technical Field

[0001] This application relates to the field of computer network technology, and more specifically, to a multi-task network traffic isolation and collection method based on eBPF. Background Technology

[0002] As network service testing platforms evolve towards large-scale and high-concurrency operation, systems typically need to simultaneously schedule multiple independent test tasks to perform network probing, load testing, and traffic interception on the same or different target IPs. To ensure the accuracy of measurement results, each task must independently collect its own network traffic statistics, such as the number of packets sent and received, bytes, connections, and TCP retransmissions. Existing network traffic monitoring technologies, particularly eBPF-based tools, are primarily designed for global observability, with their collection logic often fixed at the network interface card (NIC) level or the global network stack, failing to provide fine-grained traffic isolation at the task level within the kernel. When multiple test tasks concurrently request the same target IP, the packets generated by these tasks cannot be distinguished in the kernel, leading to cross-contamination of traffic statistics and compromising the purity of individual task metrics. Furthermore, traditional process PID-based traffic isolation schemes can only distinguish between different processes. However, modern testing engines, which prioritize performance, often employ a single-process, multi-coroutine architecture, where multiple independent tasks actually run within the same process space. This renders the PID isolation mechanism completely ineffective, failing to separate traffic initiated by different coroutines.

[0003] To address the aforementioned issues, some solutions attempt a bypass acquisition method of "full kernel packet capture and asynchronous user-space filtering." This involves capturing all packets in the kernel, copying them to user space, and then having the upper-layer application filter and aggregate them based on task identifiers. However, in high-concurrency, high-throughput streaming scenarios, massive data packets frequently need to traverse between kernel and user spaces, accompanied by rule matching and inter-process synchronization operations, resulting in significant CPU overhead and system latency, failing to meet the real-time requirements of high-performance network testing. Furthermore, when multiple tasks concurrently access the same target IP, the state transitions of network connections are highly complex. Existing solutions generally lack a full lifecycle connection attribution mapping mechanism from connection establishment and handshake to disconnection and cleanup. This leads to events such as TCP retransmissions or byte fluctuations occurring on a connection established by a particular test task being easily misattributed to other concurrent tasks targeting the same IP, severely compromising the accuracy and rigor of the load testing results.

[0004] Furthermore, the statistical values ​​collected in eBPF Maps are typically monotonically increasing cumulative counters, while network testing algorithms actually require incremental metrics within a specific time window. Existing technologies lack a simple and reliable mechanism for converting cumulative values ​​to incremental values, especially in multi-task concurrent scenarios, requiring independent baseline maintenance for each task, further increasing implementation complexity. Simultaneously, different Linux distributions have significant differences in kernel versions and mountable eBPF hooks. Existing eBPF tools often employ a rigid "all probes must be successfully mounted before running" strategy; failure of any probe mount leads to overall system failure, failing to adapt to the differentiated kernel capabilities of domestic operating systems and cross-version deployment scenarios, and lacking graceful degradation mechanisms for mount failures. Therefore, how to achieve accurate, efficient, and robust task-isolated network traffic collection in multi-task, high-concurrency, and heterogeneous kernel environments has become a pressing technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide a multi-task network traffic isolation and collection method based on eBPF, which solves the problem of cross-contamination of network traffic statistics for the same target IP in the scenario of multiple coroutines running concurrently within the same process, and ensures the consistency of statistical attribution throughout the entire lifecycle of the connection.

[0006] The objective of this application is achieved through the following technical solution: Firstly, this application proposes a multi-task network traffic isolation and collection method based on eBPF, including: Assign a unique task slot identifier to each concurrent test task running in the same process, and establish a correspondence between the target IP address of each test task and the assigned task slot identifier; In kernel mode, a statistical mapping is built to store network traffic statistics for the corresponding task. The statistical mapping is indexed by a composite key, which includes the target IP address and the task slot identifier. In response to the network connection establishment event, the corresponding relationship is queried based on the target IP address of the current connection to obtain the current task slot identifier. The task slot identifier is associated with the connection's four-tuple information and stored in the connection anchoring mapping to anchor the task to which the connection belongs. In response to data transmission or retransmission events related to the connection, the task slot identifier of the connection anchor is obtained from the connection anchoring map, a composite key is constructed by combining the target IP address of the connection, and the network traffic statistics corresponding to the composite key are updated in the statistics map.

[0007] In one possible implementation, the step of assigning a unique task slot identifier to each concurrent test task running within the same process includes: A thread-safe slot allocator is maintained in user space, and slot resources within a preset value range are managed through the slot allocator; When a new test task is started, an unoccupied slot is assigned to the task as a task slot identifier through the slot allocator. When the test task ends, the task slot identifier corresponding to the task is reclaimed by the slot allocator for reallocation. Write the mapping between target IPs and slots into the target IP mapping table in kernel mode.

[0008] In one possible implementation, the step of querying the correspondence based on the target IP address of the current connection to obtain the current task slot identifier in response to a network connection establishment event includes: Capture the connection establishment function by attaching a kernel probe (kprobe) or a tracepoint; During the connection establishment phase, the target IP mapping table is queried to obtain the currently active task slot identifier for the target IP; Generate connection metadata and store it in the connection anchoring map. The connection metadata includes connection quadruple information and task slot identifier.

[0009] In one possible implementation, the step of querying the connection anchoring task slot identifier from the connection anchoring map in response to a connection-related data transmission or retransmission event includes: When tcp_sendmsg sends data, tcp_cleanup_rbuf receives data, or tcp_retransmit_skb retransmits data, extract the four-tuple information of the current packet; Use quadruple information to query join anchor mappings; If the query is successful, the original task slot identifier that was anchored when the connection was established is read and used as the attribution identifier for this traffic statistics. If the query fails to find a match, it will be used as a fallback logic to query the target_ip_map and use the slot corresponding to the current target IP for statistics.

[0010] In one possible implementation, the method further includes obtaining the traffic increment within a specific time window: At the start of the time window, record the baseline values ​​of various network traffic statistics indicators corresponding to each task slot in the current statistical mapping; At the end of the time window, the statistical value in the current statistical mapping is read and the difference is calculated with the baseline value to obtain the traffic increment data within the time window; Derived metrics are calculated using incremental traffic data, including retransmission rate and packet loss rate.

[0011] In one possible implementation, the method further includes: When performing the operation of subtracting the baseline value from the current statistic, check for underflow of unsigned integers; If the current statistical value is detected to be less than the baseline value, it is determined to be an abnormal situation, and the difference is safely returned to zero.

[0012] In one possible implementation, the method further includes: Define a BPF mapping of type circular buffer in kernel mode to store connection events; When a connection is established, closed, or a retransmission occurs, the event structure is submitted to the circular buffer in a zero-copy manner. The event structure contains a timestamp, a quadruple, an event type, and a task slot identifier. The user-space program consumes events in the circular buffer via polling and filters them based on the task slot identifier.

[0013] In one possible implementation, the method further includes: When the user-mode consumption rate is detected to be lower than the kernel-mode production rate, causing the circular buffer to fill up, newly generated connection events are discarded in non-blocking mode.

[0014] In one possible implementation, the method further includes: eBPF probes are classified into basic, enhanced, and complete levels; The system attempts to load probes of all levels in order from basic to full level. If any probe in the current level fails to load, the system stops attempting to load probes of higher levels and continues to run probes of lower levels that have been successfully loaded. Based on the probe level that was successfully loaded, declare the currently supported acquisition capabilities to the upper-layer application.

[0015] In one possible implementation, for network packets without a connection protocol, in response to a packet transmission event, the current task slot identifier is obtained by directly querying the corresponding relationship based on the destination IP address of the packet, and a composite key is constructed in combination with the destination IP address to update the corresponding network traffic statistics in the statistical mapping.

[0016] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.

[0017] This invention designs a global singleton eBPF collector, initialized when the test engine starts, and shared by all tasks. Probes are mounted in kernel space using kprobe and tracepoint to collect events such as TCP / UDP connection lifecycle, byte transmission, and retransmission. Multi-task statistical isolation is achieved in kernel space using a composite key (IP + TaskSlot), and statistical consistency is ensured in scenarios with multiple tasks sharing the same IP address through a connection anchoring mechanism. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The diagram shows a flowchart of a multi-task network traffic isolation and collection method based on eBPF proposed in an embodiment of this application.

[0020] Figure 2 A schematic diagram of the overall architecture of the global singleton eBPF collector is shown.

[0021] Figure 3 A schematic diagram of the core interaction flow of the user-mode slot allocator is shown.

[0022] Figure 4 A schematic diagram of the baseline-differential snapshot engine architecture is shown.

[0023] Figure 5 The diagram illustrates the data structure design of the BPF Map and its complete data flow and processing logic under kernel-mode driving. Detailed Implementation

[0024] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0025] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Existing network traffic monitoring and collection technologies have several shortcomings when facing multi-task, high-concurrency network testing scenarios: the kernel lacks a fine-grained isolation mechanism based on task dimensions, leading to mutual contamination of traffic data from multiple tasks under the same target IP; traditional isolation schemes based on process PIDs are too coarse-grained and cannot be applied to single-process, multi-coroutine architectures; and the method of using full kernel packet capture plus user-space post-processing filtering causes serious cross-state copying and synchronization overhead under high-throughput streaming, resulting in significant performance bottlenecks; at the same time, due to the lack of a full lifecycle connection attribution mapping from connection establishment to disconnection, events such as TCP retransmissions are easily misattributed to other concurrent tasks, compromising the accuracy of load testing; furthermore, there is a semantic gap between the cumulative counter of the eBPF Map and the window increment metric required by the testing algorithm, and existing solutions are highly dependent on kernel versions and lack degradation capabilities in case of mounting failure, making it difficult to deploy stably in heterogeneous domestic environments.

[0027] To address the aforementioned technical issues, this application proposes a multi-task network traffic isolation and collection method based on eBPF. By employing eBPF composite key (IP+TaskSlot) statistical isolation and connection anchoring (Flow Meta) mechanism, it resolves the problems of cross-contamination of traffic data from the same target IP, incorrect connection attribution, and deployment limitations caused by dependence on container environments in high-concurrency multi-task scenarios.

[0028] Please refer to Figure 1 , Figure 1 This paper presents a flowchart illustrating a multi-task network traffic isolation and collection method based on eBPF, as proposed in an embodiment of this application, including: Step S1: Assign a unique task slot identifier to each concurrent test task running in the same process, and establish the correspondence between the target IP address of each test task and the assigned task slot identifier.

[0029] To achieve fine-grained isolation of network traffic between multiple independent test tasks running concurrently within the same process space, a thread-safe slot allocator was designed in user space. This allocator uniformly manages a limited number of slot resources ranging from 0 to 254 and employs a round-robin strategy for allocation. When a new test task starts, the allocator assigns it a unique, currently unoccupied slot as a task identifier (task_slot) and establishes a binding relationship between the task's target IP address and the assigned task_slot, allowing traffic attribution to be distinguished throughout the task's lifecycle.

[0030] Step S2: Construct a statistical mapping in kernel mode. The statistical mapping uses a composite key containing the target IP address and task slot identifier as an index to store the network traffic statistics of the corresponding task.

[0031] A hash-based statistical mapping (BPF_MAP_TYPE_HASH) is built in kernel space to store and accumulate network traffic metrics. This mapping uses a composite key structure `stats_key`, which consists of the target IP address and the task slot identifier (`task_slot`). The corresponding value is a structure (`target_stats`) containing 10 metrics such as the number of packets sent / received, the number of bytes, and the number of retransmissions. Through this design, the statistical data for different tasks, or even different tasks targeting the same target IP, are independent entries at the kernel space physical storage level, achieving physical isolation between multiple tasks from the root.

[0032] Step S3: In response to the network connection establishment event, query the corresponding relationship based on the target IP address of the current connection to obtain the current task slot identifier, and associate the task slot identifier with the connection's four-tuple information and store it in the connection anchoring mapping to anchor the task to which the connection belongs.

[0033] When the kernel detects a network connection establishment event, the system queries the kernel-maintained `target_ip_map` based on the target IP address of the current connection to obtain the corresponding `task_slot`. Subsequently, the system extracts the four-tuple information of the connection to construct a `flow_key`, and writes the `flow_meta` structure, containing the aforementioned `task_slot` and the connection establishment timestamp, into the connection anchoring map (`active_conns_map`). This permanently anchors the connection to a specific test task at the very beginning of its creation.

[0034] Step S4: In response to data transmission or retransmission events related to the connection, query the connection anchoring mapping to obtain the task slot identifier of the connection anchor, construct a composite key by combining it with the target IP address of the connection, and update the network traffic statistics corresponding to the composite key in the statistics mapping.

[0035] In the subsequent lifecycle after a connection is established, when events such as data transmission or retransmission occur, the eBPF program no longer directly trusts the mapping relationships in the current `target_ip_map`. Instead, it prioritizes querying the connection anchoring mapping (`active_conns_map`) through the four-tuple of the current packet. If the query finds a match, it directly reads the original `task_slot` anchored when the connection was established, constructs a composite key `stats_key{ip, task_slot}` by combining it with the target IP, and atomically updates the metric value corresponding to this key in the statistical mapping. This mechanism ensures that all traffic changes on the connection are accurately attributed to the original task that established the connection, completely avoiding statistical attribution drift in multi-task concurrent scenarios.

[0036] The steps for assigning unique task slot identifiers to each concurrent test task running within the same process include: A thread-safe slot allocator is maintained in user space, and the slot allocator manages slot resources within a preset value range; When a new test task is started, the slot allocator assigns a currently unoccupied slot as the task slot identifier. When the test task ends, the slot allocator reclaims the task slot identifier corresponding to the task for reallocation. Write the mapping between target IPs and slots into the target IP mapping table in kernel mode.

[0037] The user-space side maintains a thread-safe slot allocator (SlotAllocator), which manages 255 pre-configured slots (0-254) using a round-robin approach for orderly allocation and reclamation, and avoids concurrency conflicts through thread-safe mechanisms. When a new test task starts, the slot allocator selects an available slot from the currently unoccupied slot set as the unique identifier for the task (task_slot). This task_slot, together with the target IP bound to the task, forms a kernel-space statistical composite key (stats_key{ip, task_slot}), used to store 10 network metrics for the task in the eBPFHash Map. During test task execution, the user space writes the target IP → task_slot mapping to the kernel-space target_ip_map, enabling the kernel to quickly find the currently available slot when traffic events occur. When a test task completes normally or is canceled, the slot allocator reclaims the task_slot occupied by that task. Simultaneously, user space notifies the kernel to delete the corresponding entry in the target_ip_map and cleans up the statistical records for that {ip, task_slot} in the eBPF statistics map, so that the slot can be reallocated to subsequent new tasks. Through this mechanism of centralized management in user space and slot-based isolation in kernel space, different tasks, even those targeting the same IP, have their own independent statistical entries in the kernel, fundamentally avoiding data pollution and resource contention in multi-task concurrent scenarios.

[0038] Figure 3 This diagram illustrates the core interaction flow of the user-space slot allocator, a mechanism that enables precise isolation and resource management of multi-task network traffic. First, during the slot allocation phase, the control plane, upon receiving a task request, allocates an idle slot from the range 0-254 in a round-robin fashion, ensuring concurrency safety through atomic operations and triggering subsequent registration processes via an incrementing counter. Second, in the registration and mapping phase, user space binds the task's target IP to the allocated slot, writing it to the kernel's `target_ip_map` via a system call to establish the basis for traffic attribution. Next, in the task deregistration phase, user space initiates a cleanup request upon task completion, prompting the kernel to delete the task's IP mapping record in `target_ip_map` and clear related statistical entries. Finally, in the slot reclamation phase, the kernel confirms the cleanup is complete and notifies user space, which then releases the slot, marking it as idle and allowing it to re-enter the round-robin allocation pool, thus completing the closed-loop management of the entire lifecycle.

[0039] In response to a network connection establishment event, the steps for querying the corresponding relationship based on the target IP address of the current connection to obtain the current task slot identifier include: Capture the connection establishment function by attaching a kernel probe (kprobe) or a tracepoint; During the connection establishment phase, the target IP mapping table is queried to obtain the currently active task slot identifier for the target IP; Generate connection metadata containing connection quadruple information and task slot identifiers, and store it in the connection anchoring map.

[0040] To ensure consistent connection-level statistics when multiple tasks concurrently access the same target IP, the kernel precisely captures TCP connection establishment events by mounting `tcp_v4_connect` via `kprobe` or `inet_sock_set_state` via `tracepoint`. When the connection establishment process is triggered, the eBPF program first queries the kernel's `target_ip_map` for the current `task_slot` corresponding to the target IP. Then, the kernel constructs a `flow_key` using the connection's four-tuple information as the key and generates corresponding connection metadata `flow_meta`, which includes at least the connection establishment timestamp and the `task_slot`. This `flow_meta` is written to a dedicated `active_conns_map`, thus anchoring the connection to the slot identifier of its assigned task from the very beginning of the connection's creation. From then on, regardless of whether the target IP is subsequently re-registered by other tasks or its slot is switched, all network events within the connection's lifecycle will strictly belong to the `task_slot` determined during connection establishment.

[0041] The step of querying the connection anchoring task slot identifier from the connection anchoring map in response to a connection-related data transmission or retransmission event includes: When tcp_sendmsg sends data, tcp_cleanup_rbuf receives data, or tcp_retransmit_skb retransmits data, extract the four-tuple information of the current packet; Use quadruple information to query join anchor mappings; If the query is successful, the original task slot identifier that was anchored when the connection was established is read and used as the attribution identifier for this traffic statistics. If the query fails to find a match, it will be used as a fallback logic to query the target_ip_map and use the latest slot corresponding to the current target IP for statistics.

[0042] When the kernel processes subsequent network events of an established connection—including tcp_sendmsg sending data, tcp_cleanup_rbuf receiving data, tcp_retransmit_skb retransmission, and tcp_close closing the connection—the eBPF program first extracts the four-tuple information (source IP, source port, destination IP, destination port) of the current packet or socket and constructs a query key, flow_key, based on this. The system then uses this key to search the kernel-mode active_conns_map connection anchoring mapping table: if the query matches, it indicates that the connection was anchored during the connection establishment phase, and the program will directly read and use the original task_slot anchored during connection establishment recorded in the flow_meta structure as the attribution identifier for this traffic statistics, thereby overriding the latest slot value in target_ip_map that may be updated due to other task registrations, ensuring that all events throughout the connection's lifecycle strictly belong to the task that initially created it; if the query does not match, it serves as a fallback logic to query target_ip_map and use the latest slot corresponding to the current target IP for statistics.

[0043] This method also includes obtaining traffic increments within a specific time window: At the start of the time window, record the baseline values ​​of various network traffic statistics indicators corresponding to each task slot in the current statistical mapping; At the end of the time window, the statistical value in the current statistical mapping is read and the difference is calculated with the baseline value to obtain the traffic increment data within the time window; Derived metrics are calculated using incremental traffic data, including retransmission rate and packet loss rate.

[0044] To transform the monotonically increasing cumulative counters in the eBPF Map into incremental time window data required for the test ladder, a baseline is specifically designed. The differential snapshot engine executes statistical loops independently at the task_slot level. Before the concurrent testing framework enters a certain stress level, the system calls BaselineSnapshot(task_slot), which reads all 10 current network metrics (including sent / received packets and bytes, established connections, active connections, retransmissions, RST packets, TCP / UDP packets, etc.) from the kernel-mode statistics map based on the composite key stats_key{ip, task_slot}, and records them as baseline values ​​saved in user mode. After the stress level is completed, the system calls DeltaSnapshot(task_slot) again, reads the current statistical value corresponding to the same stats_key, and sets the value as the current value. The differential operation of the baseline value yields the traffic increment data within this tier. Based on this, the upper-layer algorithm can directly calculate various derived indicators based on the increment, such as retransmission rate LossPct = retransmission increment / sent packet increment × 100%, byte rate, connection success rate, etc., providing quantitative basis for adaptive tiered concurrency and circuit breaker decision-making.

[0045] Figure 4 The diagram illustrates the baseline-differential snapshot engine architecture, a mechanism specifically designed to process cumulative network statistics stored in the eBPF Map to enable incremental data collection within arbitrary time tiers. The main control logic employs a hierarchical loading strategy, attempting to mount eBPF probes from Level 1 to Level 3 in priority order. The actual operating level (full / standard / basic) of the collector is dynamically determined based on the mounting results, and the configuration is then distributed to the kernel. The kernel-mode differential calculation logic on the right executes the specific incremental algorithm. At the start of each collection cycle, the current absolute value of each metric is safely obtained as a "baseline snapshot" via `bpf_probe_read_kernel`. At the end of the cycle, this snapshot is read again and subtracted from the baseline to calculate the actual network traffic increment and retransmission rate within that time tier. This effectively solves the problem that the cumulative counter cannot directly reflect instantaneous performance metrics.

[0046] The method further includes: When performing the operation of subtracting the baseline value from the current statistic, check for underflow of unsigned integers; If the current statistical value is detected to be less than the baseline value, it is determined to be an abnormal situation, and the difference is safely returned to zero.

[0047] A safeSub underflow protection mechanism is introduced to address the risk of unsigned integer underflow in eBPF Maps caused by abnormal scenarios such as task restarts, statistic clearing, or kernel counter resets. When user space reads the current statistic value of a certain stats_key{ip, task_slot} and attempts to subtract it from the previously recorded baseline value, the system first performs a numerical comparison: if the current statistic value is found to be less than the baseline value, it is considered an abnormal state. In this case, the normal subtraction operation is not performed, and unsigned integer wrapping that could produce a maximum value is not allowed. Instead, the safeSub function safely returns a difference of 0, thus ensuring that the calculation result of the derived statistic always stays within a reasonable range.

[0048] The method further includes: Define a BPF mapping of type circular buffer in kernel mode to store connection events; When a connection is established, closed, or a retransmission occurs, an event structure containing a timestamp, a quadruple, an event type, and a task slot identifier is submitted to the circular buffer using a zero-copy method. User-space programs consume events in the circular buffer via polling and filter them based on task slot identifiers.

[0049] Whenever a connection is established, closed, or retransmitted, the eBPF program constructs a `conn_event` event structure, which includes the timestamp of the event, connection quadruple information, event type, and the `task_slot` identifier to which the connection was anchored during connection establishment. The program then calls the `bpf_ringbuf_submit` interface to submit the event directly to the aforementioned circular buffer in a zero-copy manner, thus avoiding the additional memory copying overhead of traditional `perf_event` or user-space polling. The user-space program continuously consumes events from this circular buffer through a polling mechanism and can precisely filter and merge connection events for different test tasks based on the `task_slot` field, achieving task-level timing reconstruction and state analysis.

[0050] The method further includes: When the user-mode consumption rate is detected to be lower than the kernel-mode production rate, causing the circular buffer to fill up, newly generated connection events are discarded in non-blocking mode.

[0051] To prevent kernel buffer backlog due to insufficient user-space processing capacity in high-concurrency scenarios, which could then block the network protocol stack or affect the test engine itself, the system introduces a backpressure protection strategy: when it detects that the user-space consumption speed lags behind the kernel event output speed, it actively discards newly generated events in a select + default non-blocking mode, sacrificing some non-critical observable data in exchange for kernel-side stability and deterministic execution of the overall test task.

[0052] The method further includes: eBPF probes are classified into basic, enhanced, and complete levels; The system attempts to load probes of all levels in order from basic to full level. If any probe in the current level fails to load, the system stops attempting to load probes of higher levels and continues to run probes of lower levels that have been successfully loaded. Based on the probe level that was successfully loaded, declare the currently supported acquisition capabilities to the upper-layer application.

[0053] This application categorizes the required eBPF probes into three levels based on functional dependencies and importance: Level 1 (basic level), Level 2 (enhanced level), and Level 3 (full level). The system follows a loading strategy of "from low to high, trying each level sequentially, and downgrading upon failure": at startup, it prioritizes attempting to mount the Level 1 probe set (tcp_v4_connect, tcp_close, inet_sock_set_state). If any probe fails to mount, it is determined that basic capabilities are missing, and the collector fails to start directly. If all Level 1 probes are successfully mounted, it continues to attempt to mount the Level 2 probe set (tcp_sendmsg, tcp_cleanup_rbuf). If any Level 2 probe fails to mount, Level 2 and higher levels are abandoned, and the system operates only with Level 1 capabilities. Similarly, Level 3 (tcp_retransmit_skb, udp_sendmsg) is an optional enhancement and is only enabled if all probes are successfully mounted. The `activeLevel` field records the currently active probe level, and the `HasRetransmitSignal()` interface is provided to declare to the upper-layer application whether retransmission signal acquisition capability is available. This allows the upper-layer concurrent testing framework to dynamically adjust the range of collected metrics based on the current kernel support. Table 1 shows the three-level probe degradation mechanism: Table 1

[0054] For network packets without connection protocols, in response to the packet sending event, the current task slot identifier is obtained by directly querying the corresponding relationship based on the destination IP address of the packet, and a composite key is constructed by combining the destination IP address, and the corresponding network traffic statistics are updated in the statistical mapping.

[0055] Figure 2 This diagram illustrates the overall architecture of the global singleton eBPF data collector. The architecture employs a layered, decoupled design, consisting of a user-space control plane on the left and a kernel-space data acquisition plane on the right. The user-space control plane acts as the central hub, responsible for responding to requests from upper-layer concurrent test tasks, creating and managing the lifecycle of Maps, and continuously polling kernel data as a consumer. The kernel-space plane serves as the execution engine, capturing the entire lifecycle and transmission details of TCP / UDP by mounting kprobe and tracepoint probes, and utilizing composite keys (IP + TaskSlot) and connection anchoring mechanisms to achieve precise isolation and attribution statistics for multi-task traffic. The two planes communicate bidirectionally through efficient Ring Buffers and Map mappings, jointly supporting a high-concurrency, low-overhead network performance acquisition system.

[0056] Figure 5 This diagram illustrates the data structure design of the BPF Map and its complete data flow and processing logic under kernel-mode operation. The diagram reveals the system's control flow and data input sources, covering connection establishment and data transfer events captured through system calls and kernel probes, as well as configuration and initialization commands issued from user space. The core processing layer of the eBPF program includes the declaration and interaction logic of three key Maps: active_conns_map (connection anchoring), target_ip_map (slot mapping), and stats_map (statistical storage), responsible for transforming captured raw events into structured traffic metadata. The diagram also presents the differential calculation algorithm and TopN sorting logic implemented based on these Maps.

[0057] Compared with the prior art, the embodiments of this application have the following beneficial effects: First, by introducing a (IP, TaskSlot) composite key into the eBPF Map, physical isolation of multi-task statistics is achieved from the kernel space, eliminating the need for post-filtering in the user space. Compared to the traditional approach that requires traversing the entire dataset (O(n)), this approach directly queries (O(1)), completely resolving the problem of cross-contamination of traffic from multiple tasks under the same target IP.

[0058] Secondly, by utilizing a connection anchoring mechanism, the assigned task slot is recorded during connection establishment, and all subsequent transmission, retransmission, and closure events are statistically analyzed using this anchored slot. This capability eliminates statistical ambiguity in multi-task scenarios with the same IP address, ensuring correct attribution of events such as TCP retransmissions and byte fluctuations.

[0059] Third, it implements zero-copy event pushing from kernel space to user space based on BPF ring buffer, and supports independent connection timing reconstruction for each task in conjunction with task slot filtering. At the same time, it provides a backpressure protection mechanism, which can discard new events under high load to avoid kernel blocking and ensure the stability of the testing engine itself.

[0060] Fourth, probes are divided into three levels: basic, enhanced, and complete. When some probes fail to mount, they automatically downgrade to a lower level and continue running without manual configuration. Newer kernels can obtain all 10 metrics, while older kernels can still provide at least 3 core statistics, significantly improving deployment adaptability across Linux distributions and domestic operating systems.

[0061] Fifth, a "baseline recording + incremental difference" model is adopted to convert the monotonically increasing cumulative counter in the eBPF Map into an incremental indicator within a time window. Combined with safe subtraction to prevent underflow errors caused by counter resets, this provides accurate derived indicators for the upper-level adaptive ladder algorithm.

[0062] Sixth, it does not rely on container APIs such as Kubernetes and Docker, and can run independently in any Linux environment, including bare metal, physical machines, virtual machines, and KylinServer OS. This feature is particularly suitable for domestic operating systems and traditional non-containerized testing scenarios, expanding the applicability of the technology.

[0063] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-task network traffic isolation and collection method based on eBPF, characterized in that, include: Assign a unique task slot identifier to each concurrent test task running in the same process, and establish a correspondence between the target IP address of each test task and the assigned task slot identifier; In kernel mode, a statistical mapping is built to store network traffic statistics for the corresponding task. The statistical mapping is indexed by a composite key, which includes the target IP address and the task slot identifier. In response to the network connection establishment event, the corresponding relationship is queried based on the target IP address of the current connection to obtain the current task slot identifier. The task slot identifier is associated with the connection's four-tuple information and stored in the connection anchoring mapping to anchor the task to which the connection belongs. In response to data transmission or retransmission events related to the connection, the task slot identifier of the connection anchor is obtained from the connection anchoring map, a composite key is constructed by combining the target IP address of the connection, and the network traffic statistics corresponding to the composite key are updated in the statistics map.

2. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, The steps for assigning unique task slot identifiers to each concurrent test task running within the same process include: A thread-safe slot allocator is maintained in user space, and slot resources within a preset value range are managed through the slot allocator; When a new test task is started, an unoccupied slot is assigned to the task as a task slot identifier through the slot allocator. When the test task ends, the task slot identifier corresponding to the task is reclaimed by the slot allocator for reallocation. Write the mapping between target IPs and slots into the target IP mapping table in kernel mode.

3. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, In response to a network connection establishment event, the steps for querying the corresponding relationship based on the target IP address of the current connection to obtain the current task slot identifier include: Capture the connection establishment function by attaching a kernel probe (kprobe) or a tracepoint; During the connection establishment phase, the target IP mapping table is queried to obtain the currently active task slot identifier for the target IP; Generate connection metadata and store it in the connection anchoring map. The connection metadata includes connection quadruple information and task slot identifier.

4. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, The step of querying the connection anchoring task slot identifier from the connection anchoring map in response to a connection-related data transmission or retransmission event includes: When tcp_sendmsg sends data, tcp_cleanup_rbuf receives data, or tcp_retransmit_skb retransmits data, extract the four-tuple information of the current packet; Use quadruple information to query join anchor mappings; If the query is successful, the original task slot identifier that was anchored when the connection was established is read and used as the attribution identifier for this traffic statistics. If the query fails to find a match, it will be used as a fallback logic to query the target_ip_map and use the slot corresponding to the current target IP for statistics.

5. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, The method also includes obtaining the traffic increment within a time window: At the start of the time window, record the baseline values ​​of various network traffic statistics indicators corresponding to each task slot in the current statistical mapping; At the end of the time window, the statistical value in the current statistical mapping is read and the difference is calculated with the baseline value to obtain the traffic increment data within the time window; Derived metrics are calculated using incremental traffic data, including retransmission rate and packet loss rate.

6. The multi-task network traffic isolation and acquisition method as described in claim 5, characterized in that, The method further includes: When performing the operation of subtracting the baseline value from the current statistic, check for underflow of unsigned integers; If the current statistical value is detected to be less than the baseline value, it is determined to be an abnormal situation, and the difference is safely returned to zero.

7. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, The method further includes: Define a BPF mapping of type circular buffer in kernel mode to store connection events; When a connection is established, closed, or a retransmission occurs, the event structure is submitted to the circular buffer in a zero-copy manner. The event structure contains a timestamp, a quadruple, an event type, and a task slot identifier. The user-space program consumes events in the circular buffer via polling and filters them based on the task slot identifier.

8. The multi-task network traffic isolation and acquisition method as described in claim 7, characterized in that, The method further includes: When the user-mode consumption rate is detected to be lower than the kernel-mode production rate, causing the circular buffer to be full, newly generated connection events are discarded in non-blocking mode.

9. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, The method further includes: eBPF probes are classified into basic, enhanced, and complete levels; The system attempts to load probes of all levels in order from basic to full level. If any probe in the current level fails to load, the system stops attempting to load probes of higher levels and continues to run probes of lower levels that have been successfully loaded. Based on the probe level that was successfully loaded, declare the currently supported acquisition capabilities to the upper-layer application.

10. The multi-task network traffic isolation and acquisition method as described in claim 1, characterized in that, For network packets without connection protocols, in response to the packet sending event, the current task slot identifier is obtained by directly querying the corresponding relationship based on the destination IP address of the packet, and a composite key is constructed by combining the destination IP address, and the corresponding network traffic statistics are updated in the statistical mapping.

Images