High-speed traffic acquisition method based on DPDK and XDP optimization strategies

Abstract

The invention relates to the technical field of network traffic, in particular to a high-speed traffic collection method based on a DPDK and XDP optimization strategy, which comprises the following steps of: finishing environment deployment by adopting a DPDK script and a tool, loading a UIO module and mapping a network card register to a user space, configuring a 2MB / 1GB giant page memory through a Linux system starting parameter, applying for a continuous physical memory area in the DPDK, and collecting the traffic in the DPDK. An executable file is generated through a BPF compiler, a DPDK is configured to use a zero copy technology to receive a data packet to a user buffer area, an XDP program performs preliminary classification on a driving layer, traffic data is written into a disk by adopting DIRECT-IO and asynchronous IO, a compression algorithm is combined to reduce storage occupation, a time index is established to realize rapid retrieval, and an efficient storage and queriable traffic database is obtained. According to the invention, the problems of low real-time performance, easy packet loss, slow response speed and data redundancy during multi-domain information network flow collection in a high-speed network environment are solved.

Description

Technical Field

[0001] This invention belongs to the field of network traffic technology, specifically relating to a high-speed traffic collection method based on DPDK and XDP optimization strategies. Background Technology

[0002] As networks expand in scale, supporting increasingly diverse application systems, and becoming more complex, network services place ever higher demands on network resources. To improve packet processing performance on general-purpose servers (COTS), Intel introduced the DPDK technology for IA (Intel Architecture) systems. DPDK is a highly customized, high-performance network packet processing solution based on Intel processors, encompassing packet processing function libraries and network device drivers. Utilizing Linux's UIO (User Space I / O) technology, it establishes a transmission channel between kernel space and user space, reducing kernel data copying and interrupt occurrences, paving the way for high-performance network packet capture. Addressing the issues of low real-time performance, packet loss, slow response, and data redundancy in high-speed network traffic acquisition for multi-domain information networks, and to better meet the packet capture requirements of this project under high-speed networks, a high-performance traffic acquisition technology based on the Data Plane Development Kit (DPDK) and eXpress Data Path (XDP) was researched. By using zero-copy sockets AF_XDP to bypass the kernel protocol stack, data packets are directly transferred from kernel space to user space, effectively reducing the system overhead caused by frequent context switching between user space and kernel space, and improving data packet capture performance.

[0003] This invention addresses the shortcomings of existing traffic acquisition technologies in high-speed network environments, such as low real-time performance, susceptibility to packet loss, slow response speed, and data redundancy. It proposes a high-speed traffic acquisition technology based on DPDK and XDP optimization strategies, enabling comprehensive, high-speed, accurate, and complete network traffic acquisition on probe devices. Specifically, XDP technology integrates high-performance packet processing with the operating system kernel. By providing a secure execution environment and receiving support from the kernel community, it significantly reduces the cost of application packet processing, achieving efficient acquisition of full network traffic from layers L2 to L7. While ensuring the performance of the full traffic acquisition and tracing system, it enhances system compatibility and security, guaranteeing secure and efficient acquisition of full network traffic. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a high-speed traffic acquisition method based on DPDK and XDP optimization strategies. This method solves the problems of low real-time performance, packet loss, slow response, and data redundancy encountered when acquiring traffic from multi-domain information networks in high-speed network environments. To achieve the above objectives, this invention adopts the following technical solution: The high-speed traffic acquisition method based on DPDK and XDP optimization strategies includes the following steps: Deploying the environment using DPDK scripts and tools, loading the UIO module and mapping the network interface card (NIC) registers to user space, configuring the PMD driver to reduce interrupt overhead, and obtaining an initialized DPDK runtime environment; configuring 2MB / 1GB huge page memory through Linux system boot parameters, allocating contiguous physical memory regions in DPDK, reducing the number of page table entries and improving TLB hit rate, resulting in optimized memory access performance; utilizing DPDK's CPU affinity API to bind packet processing threads to specific CPU cores, avoiding CAC errors caused by cross-core switching. After the he failed, a stable task processing unit was obtained; eBPF bytecode was written to implement packet filtering logic, and an executable file was generated by the BPF compiler. The executable file was loaded into the kernel driver layer using the bpf system call to obtain a high-efficiency kernel-mode processing program; DPDK was configured to receive packets to the user buffer using zero-copy technology, and the XDP program performed preliminary classification at the driver layer. Metadata was passed through BPF mapping to obtain high-speed captured and preprocessed traffic data; DIRECT-IO and asynchronous IO were used to write traffic data to disk, combined with compression algorithms to reduce storage usage, and a time index was established to achieve fast retrieval, resulting in a high-efficiency and queryable traffic database.

[0005] Furthermore, the process of deploying the environment using DPDK scripts and tools, loading the UIO module and mapping the network card registers to user space, configuring the PMD driver to reduce interrupt overhead, and obtaining an initialized DPDK runtime environment includes the following steps: The DPDK environment is built on a general-purpose server using an installation and configuration method; device memory management and hardware interrupt handling are implemented by loading the UIO module; mmap technology is used to obtain management capabilities for physical memory, virtual memory, and logical memory; network card hardware registers are mapped to user space; and PMD polling mode driver technology is used to replace the traditional interrupt operation mode, reducing interrupt overhead and obtaining an initialized DPDK runtime environment.

[0006] Furthermore, the method of configuring 2MB / 1GB huge page memory through Linux system boot parameters, allocating contiguous physical memory regions in DPDK, reducing the number of page table entries, and improving TLB hit rate to obtain optimized memory access performance includes the following steps: 1. Configuring huge pages by adding specific kernel parameters during Linux system boot, setting default_hugepagesz=2MB, hugepagesz=2MB, hugepages=N, hugepagesz=1GB, and hugepages=M, where N is the specific number of pages and M is the number of 1GB pages; 2. Allocating contiguous huge page physical memory regions by calling the memory management API during the DPDK initialization phase; 3. Directly mapping virtual addresses to physical addresses to obtain an optimized memory performance environment that reduces memory access latency and improves data processing efficiency.

[0007] Furthermore, the method of using the CPU affinity API of DPDK to bind the packet processing thread to a specified CPU core, avoiding cache invalidation caused by cross-core switching, and obtaining a stable task processing unit includes the following steps: using the CPU affinity API of DPDK to accurately allocate the packet processing thread to a specified physical core by parsing the binding relationship parameters between the thread and the CPU core; tightly associating the thread execution context with the CPU cache level, fixing the thread to execute on a specific core, and avoiding cache invalidation during task migration; and ensuring that instructions and data continuously reside in the core's private cache, reducing cache line filling overhead, and obtaining a stable task processing unit with low latency and high hit rate.

[0008] Furthermore, the process of writing eBPF bytecode to implement packet filtering logic, generating an executable file using a BPF compiler, and loading it into the kernel driver layer using the bpf system call to obtain a high-efficiency kernel-mode processing program includes the following steps: Writing a packet filtering program in C language combined with eBPF, specifying L2-L7 layer field matching rules and processing actions; converting the source code into eBPF bytecode using an LLVM compiler to generate a binary file conforming to kernel verification specifications; injecting the bytecode into the kernel using the bpf system call, checking the legality of the instructions through a verifier, and attaching the bytecode to the XDP hook point in the network card driver layer; and deeply integrating the filtering logic with the hardware packet receiving path to obtain a high-efficiency kernel-mode packet processing program that does not require context switching.

[0009] Furthermore, the configuration of DPDK uses zero-copy technology to receive data packets to the user buffer. The XDP program performs preliminary classification at the driver layer and transmits metadata through BPF mapping to obtain high-speed captured and preprocessed traffic data. This includes the following steps: configuring a zero-copy receive queue using the rte_pktmbuf library of DPDK to directly map the network card DMA descriptor to the user-space memory pool; parsing the five-tuple information in the packet header through the XDP program at the driver layer and performing flow classification according to preset rules; encapsulating the classification result and packet pointer into a BPF mapping entry, which is then transmitted through the kernel-user space shared BPF_MAP_TYPE_PERF_EVENT_ARRAY mapping table to obtain traffic data that skips kernel protocol stack processing and carries pre-classified tags.

[0010] Furthermore, the method of using DIRECT-IO and asynchronous IO to write traffic data to disk, combined with compression algorithms to reduce storage usage, and establishing a time index to achieve fast retrieval, resulting in a traffic database with efficient storage and queryability, includes the following steps: Using DIRECT-IO mode to bypass the kernel buffer, directly writing the traffic data captured by DPDK to the disk block device; submitting batch write requests through asynchronous IO to reduce context switching; using the LZ4 compression algorithm to perform lossless compression on the original packets, reducing storage space usage by more than 60%; constructing a time-series index structure based on a B+ tree by extracting the packet timestamp as the index key; and synchronously writing the compressed data and index metadata to obtain a traffic database with both high compression ratio and millisecond-level retrieval response.

[0011] Furthermore, the step of configuring a zero-copy receive queue using the DPDK rte_pktmbuf library to directly map the network card DMA descriptor to the user-space memory pool includes the following steps: initializing the memory pool using the DPDK rte_pktmbuf library, pre-allocating a contiguous physical memory region and constructing a user-space buffer; configuring the receive queue through the rte_eth_rx_queue_setup function to directly bind the network card DMA descriptor to the mbuf structure in the memory pool; and using memory alignment optimization technology to ensure that the starting address of each mbuf is aligned with the cache line boundary, reducing CPU prefetch overhead and obtaining a zero-copy data receive path without kernel intervention.

[0012] Furthermore, the step of parsing the five-tuple information in the packet header at the driver layer using the XDP program and performing flow classification actions according to preset rules includes the following steps: embedding processing logic in the network card driver layer using the XDP program architecture, obtaining the five-tuple information by parsing the first 128 bytes of the packet header; performing hash matching between the source / destination IP, port, and protocol type and a preset rule table to quickly determine the category to which the data flow belongs; and performing differentiated processing actions based on the classification results to obtain data packets carrying flow classification tags. The processing actions include allowing the flow to be directly forwarded to user space, discarding illegal flows, and marking suspicious flows.

[0013] In the technical solution provided by this invention, DPDK scripts and tools are used to complete the environment deployment, load the UIO module and map the network card registers to user space, configure the PMD driver to reduce interrupt overhead, and obtain an initialized DPDK runtime environment; 2MB / 1GB huge page memory is configured through Linux system boot parameters, and a contiguous physical memory region is allocated in DPDK to reduce the number of page table entries and improve TLB hit rate, resulting in optimized memory access performance; by utilizing DPDK's CPU affinity API, the packet processing thread is bound to a specified CPU core to avoid cache invalidation caused by cross-core switching, thus achieving stable operation. The invention comprises a task processing unit; eBPF bytecode is written to implement packet filtering logic, an executable file is generated by the BPF compiler, and the BPF system call is used to load it into the kernel driver layer to obtain a high-efficiency kernel-mode processing program; DPDK is configured to receive packets to the user buffer using zero-copy technology, the XDP program performs preliminary classification at the driver layer, and metadata is passed through BPF mapping to obtain high-speed captured and preprocessed traffic data; DIRECT-IO and asynchronous IO are used to write the traffic data to disk, combined with compression algorithms to reduce storage usage, and a time index is established to achieve fast retrieval, resulting in a high-efficiency and queryable traffic database. This invention solves the problems of low real-time performance, easy packet loss, slow response speed, and data redundancy faced in multi-domain information network traffic collection in high-speed network environments. Attached Figure Description

[0014] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0015] Figure 1 This is a schematic diagram of the first embodiment of a high-speed traffic acquisition method based on DPDK and XDP optimization strategies in this invention.

[0016] Figure 2 This is a schematic diagram of a second embodiment of a high-speed traffic acquisition method based on DPDK and XDP optimization strategies in this invention.

[0017] Figure 3 This is a schematic diagram of a third embodiment of a high-speed traffic acquisition method based on DPDK and XDP optimization strategies in this invention.

[0018] Figure 4 This is a schematic diagram of the fourth embodiment of a high-speed traffic acquisition method based on DPDK and XDP optimization strategies in this invention.

[0019] Figure 5 This is a schematic diagram of the fifth embodiment of a high-speed traffic acquisition method based on DPDK and XDP optimization strategies in this invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] A high-speed traffic acquisition method based on DPDK and XDP optimization strategies, such as Figure 1As shown, the process includes the following steps: Deploying the environment using DPDK scripts and tools, loading the UIO module and mapping the network card registers to user space, configuring the PMD driver to reduce interrupt overhead, and obtaining an initialized DPDK runtime environment; configuring 2MB / 1GB huge page memory through Linux system boot parameters, allocating contiguous physical memory regions in DPDK, reducing the number of page table entries and improving TLB hit rate, resulting in optimized memory access performance; utilizing DPDK's CPU affinity API to bind packet processing threads to specific CPU cores, avoiding cache invalidation caused by cross-core switching, and achieving stable operation. The task processing unit writes eBPF bytecode to implement packet filtering logic, generates an executable file using the BPF compiler, loads it into the kernel driver layer using the bpf system call, and obtains a high-efficiency kernel-mode processing program. It configures DPDK to receive packets to the user buffer using zero-copy technology, performs preliminary classification in the driver layer using the XDP program, and passes metadata through BPF mapping to obtain high-speed captured and preprocessed traffic data. It uses DIRECT-IO and asynchronous IO to write the traffic data to disk, combines compression algorithms to reduce storage usage, and establishes a time index to achieve fast retrieval, resulting in a high-efficiency, queryable traffic database.

[0023] The DPDK achieves very high performance thanks to the following four technologies: (1) UIO and PMD technologies. In Linux systems, the UIO module is mainly used to manage device memory and handle hardware interrupts. For the former, the UIO module manages physical memory, virtual memory, and logical memory through mmap, mapping the network card hardware registers to user space, supporting the network card driver to run in user space. This avoids unnecessary switching between kernel mode and user mode, reducing system overhead. Then, DPDK uses PMD (Poll Mode Driver) technology to replace interrupt operations. In the Linux kernel, whether network devices receive packets through early hard interrupts or later NAPI (New API) technology polls devices by disabling hard interrupts and enabling soft interrupts, the overhead of interrupt context switching is not insignificant in high-traffic network conditions. Therefore, DPDK's PMD significantly reduces system interrupt overhead.

[0024] (2) Huge Page Memory. The default page size in Linux systems is 4KB. The smaller the page, the larger the system memory, and the greater the overhead of the page table, resulting in more memory occupied by the page table. Due to memory management needs, the CPU introduces the MMU (Memory Management Unit) to manage the mapping from virtual addresses to physical addresses. If an application needs to access data in memory, it first needs to look up the physical address corresponding to the virtual address in the page table, and then access the physical address to read the data. To mitigate the reduction in CPU processing power caused by the MMU, the TLB (Translation Lookaside Buffer) is introduced. The TLB is actually a cache for the page table, located inside the CPU, and stores frequently accessed page table entries. The page table in memory is only used when the page to be accessed is not in the TLB. Because the TLB is part of the CPU's internal cache, it is very expensive and generally can only store a limited number of page table entries. Therefore, if a process needs to use 1GB of memory, with a page size of 4KB, it requires 1GB / 4KB = 262,144 memory pages. One page corresponds to one page table entry, and one page table entry is 4 bytes in size. Therefore, the page table entry size for 1GB of memory is 262,144 * 4 bytes = 1MB. If the page size is changed to 2MB, it requires 1GB / 2MB = 512 memory pages, and the total page table entry size is 2KB. Because of the change in page size, the number of page table entries decreases from 2MB to 2KB. This allows the TLB to hold more page table entries. Therefore, DPDK uses 2MB or even 1GB huge pages, which can significantly improve the TLB hit rate and thus improve memory access performance.

[0025] (3) CPU affinity. In SMP (Symmetric Multi-Processing) Linux systems, without specific settings, the kernel will schedule tasks based on system load, regardless of which CPU core they run on. This can cause a task to frequently switch between multiple CPU cores during execution, incurring additional system overhead. Furthermore, after switching CPUs, the cached content fetched from memory becomes invalid, severely degrading CPU performance. Therefore, DPDK utilizes the CPU affinity feature to bind user processes to a specified CPU, avoiding the overhead of process CPU switching and cache invalidation.

[0026] (4) Lock-free queues. DPDK's lock-free queues are implemented by borrowing the kfifo lock-free queue mechanism from the Linux kernel. In essence, it is a first-in-first-out (FIFO) circular queue. It supports various producer-consumer models, and with its internal lock-free mechanism, it greatly saves system overhead.

[0027] High-speed traffic capture (DPDK) technology allows DPDK applications to run in user space and use their own data plane libraries to send and receive data packets, bypassing the Linux kernel protocol stack's handling of data packets. It is not a complete product that users can directly build applications from, and it does not include tools that require interaction with the control layer (including the kernel and protocol stack).

[0028] Compared to native Linux, adopting DPDK technology can significantly improve IPv4 forwarding performance, allowing users to gain better cost and performance advantages when migrating packet processing applications. It also allows for the deployment of different services, such as application processing, control processing, and packet processing services, on a unified platform.

[0029] like Figure 2 As shown, in this embodiment, the DPDK environment is set up on a general server by installation and configuration. Device memory management and hardware interrupt handling are achieved by loading the UIO module. The mmap technology is used to obtain the management capabilities of physical memory, virtual memory, and logical memory, and the network card hardware registers are mapped to user space. The PMD polling mode driver technology is used to replace the traditional interrupt operation mode, reducing interrupt overhead and obtaining the initialized DPDK runtime environment.

[0030] By copying packets to the application space for processing via UIO, unnecessary memory copies and system calls are avoided, facilitating rapid iterative optimization. Huge pages reduce cache misses (memory access overhead), and multi-channel interleaved memory access improves effective memory access bandwidth, thus increasing hit rate and CPU access speed. CPU affinity binds network cards and threads to fixed cores, reducing CPU task switching. Specific tasks can be assigned to work only on a single core, avoiding frequent thread switching between different cores and ensuring more cache hits. Lock-free queues reduce resource contention. Cache line alignment, data prefetching, and batch operations on diverse data are employed. Polling avoids the overhead of interrupting context switching during packet processing.

[0031] Driven by a polling model, this system improves the efficiency of sending / receiving data frames through a non-interruptible, zero-copy mechanism for data frames entering and leaving the application buffer memory. Stream classification provides optimized search algorithms for N-tuple matching and LPM (Longest Prefix Matching). A ring queue is established to provide a lock-free mechanism for the inbound and outbound queues of single or multiple packet producers and single packet consumers, effectively reducing system overhead. Memory is allocated to create buffers, and MBUF objects are created to encapsulate the actual data frames for application use.

[0032] Meanwhile, to achieve high-speed traffic capture, a large memory data cache and multi-queue data reading approach are employed. During operating system startup, a contiguous 2GB memory block is allocated and mapped directly to the network card's firmware via DMA. After the network card captures data packets, it directly writes them into a ring-shaped lock-free queue. The application layer reads data packets directly from the lock-free queue using a zero-copy method, reducing the time spent copying data areas and achieving an optimal read speed.

[0033] For gigabit data network traffic of 2Gb / s, the capture card's 2GB receive buffer can provide 8 seconds of data caching capability, greatly alleviating the data processing burden on upper-layer applications and ensuring packet capture without loss. To fully utilize the processing power of multi-core CPUs while matching disk write speed, the data capture module uses 8 queues to read captured data packets from the network card. When allocating data packets to the 8 queues, the network card uses a same-source, same-destination algorithm to ensure that data streams from the same session are always assigned to the same queue for processing. This is crucial for reading data from multiple queues.

[0034] Message capture, storage, and retrieval are the core foundation of the entire system, significantly impacting future functional expansion and support for various hardware platforms. Therefore, the message storage mechanism was carefully considered in the data storage module design. To improve message storage efficiency, a two-tiered storage mechanism of system memory and ordinary disk is adopted. At system startup, each message capture thread pre-allocates sufficient memory cache, and captured messages are initially stored in the memory cache. When the cache reaches a specified capacity, a combination of DirectIO and asynchronous I / O is used to directly store the captured messages on the hard disk, thereby improving I / O write throughput and achieving a match between network traffic and hard disk storage speed. Due to limited system storage capacity, a time-based ring overwrite mechanism is employed: when the storage capacity reaches a certain threshold, the system automatically overwrites the oldest stored message file.

[0035] While DPDK boasts numerous advantages, including high performance, a comprehensive development kit, and the ability to develop drivers in user space, its shortcomings are also apparent: DPDK bypasses the Linux kernel network protocol stack, meaning most network applications require a user-space protocol stack to function correctly. However, user-space protocol stack implementations are diverse, offering varying levels of support for network applications, and cannot achieve the comprehensiveness and compatibility of the kernel network protocol stack. Therefore, DPDK achieves high performance at the expense of versatility. Furthermore, DPDK simply pulls network data directly to user space; it lacks the functionality to analyze and filter network data itself.

[0036] To address these issues, researchers proposed a special eBPF program—XDP. This technology allows for packet processing before the protocol is complete, resulting in very high packet processing performance. The rise of XDP, or ExpressData Path, stems from the Linux kernel's need for high-performance packet processing capabilities. Many kernel-bypassing techniques (DPDK being the most prominent) aim to accelerate network operations by migrating packet processing to user space. This means eliminating the overhead of context switching, system call translation, or interrupt requests between the kernel and user space boundaries. The operating system relinquishes control of the network stack to user-space processes, which interact directly with the network card through their own drivers.

[0037] While this approach brings significant performance gains, it also introduces a series of drawbacks, including the need to reimplement the TCP / IP protocol stack and other network functions in user space, or the abandonment of the kernel's robust resource abstraction and security management.

[0038] like Figure 3 As shown in this embodiment, specific kernel parameters are added during Linux system startup. Huge pages are configured by setting default_hugepagesz=2MB hugepagesz=2MB hugepages=N, hugepagesz=1GB hugepages=M, where N is the specific number of pages and M is the number of 1GB pages. During the DPDK initialization phase, a contiguous physical memory region of huge pages is requested by calling the memory management API. The virtual address is directly mapped to the physical address, resulting in an optimized memory performance environment that reduces memory access latency and improves data processing.

[0039] like Figure 4As shown, in this embodiment, the CPU affinity API of DPDK is used to precisely allocate the data packet processing thread to the specified physical core by parsing the binding relationship parameters between the thread and the CPU core; the thread running context is closely associated with the CPU cache level, and the thread is fixed to execute on a specific core to avoid the cache content becoming invalid when the task is migrated; instructions and data are continuously resident in the core private cache, reducing cache line filling overhead and obtaining a stable task processing unit with low latency and high hit rate.

[0040] like Figure 5 As shown, in this embodiment, a packet filtering program is written in C language combined with eBPF, specifying the L2-L7 layer field matching rules and processing actions. The source code is converted into eBPF bytecode by the LLVM compiler, generating a binary file that conforms to the kernel verification specification. The bytecode is injected into the kernel using the bpf system call, and the verifier checks the legality of the instructions. The bytecode is then mounted to the XDP hook point of the network card driver layer. The filtering logic is deeply integrated with the hardware packet receiving path to obtain a kernel-mode high-efficiency packet processing program that does not require context switching.

[0041] In this embodiment, the zero-copy receive queue is configured using the rte_pktmbuf library of DPDK, and the network card DMA descriptor is directly mapped to the user space memory pool. The XDP program parses the five-tuple information in the packet header at the driver layer and performs flow classification according to preset rules. The classification result and packet pointer are encapsulated into a BPF mapping entry, which is then passed through the BPF_MAP_TYPE_PERF_EVENT_ARRAY mapping table shared between the kernel and user space to obtain traffic data that skips kernel protocol stack processing and carries pre-classification tags.

[0042] The goal of XDP is to achieve programmable packet processing within the kernel while still retaining the basic network protocol stack modules. In essence, XDP represents a natural extension of eBPF instructions. It uses maps, manageable helper functions, and a sandboxed bytecode runner to achieve programmability; these bytecode instructions are only loaded into the kernel for execution after being checked for safety.

[0043] The key to XDP's high-speed processing path lies in the fact that these programming bytecodes are loaded into the earliest possible processing point of the network protocol stack, immediately after the network packet receive queue. At this stage of the network protocol stack, no kernel attributes of the network protocol have yet been built, which is highly beneficial for improving network processing speed. The following section details XDP technology from the basics of the eBPF architecture.

[0044] eBPF revolutionized BPF by redefining an instruction set that brought the eBPF virtual machine closer to modern processors. Furthermore, eBPF supports Just-In-Time (JIT) compilation, enabling the conversion of eBPF bytecode into assembly code tailored to different architectures, thus improving execution efficiency. eBPF bytecode is written by the user and injected into the Linux kernel; different bytecode implementations can achieve different functionalities. This provides a method for changing system behavior in real-time during system runtime. Security is paramount for code running in the kernel. Before execution, eBPF programs undergo a comprehensive audit by the kernel's Verifier, thus imposing the following constraints: 1) The size of the bytecode cannot exceed 4096 lines, and the size of the program branches cannot exceed 1024 lines; 2) The program's stack size does not exceed 1024 bytes during runtime; 3) The total number of steps the program takes cannot exceed 131072 (128K) steps; 4) Loops, unused code, and jumps beyond the runtime scope are not allowed in the program; 5) Access to global variables other than BPF map data (key-value data type) is not allowed; 6) Calling functions other than the default functions is not allowed.

[0045] Based on the above conditions, the kernel treats the eBPF program as a function. This function has one input parameter, and the type of the input parameter varies depending on the application scenario. Among them, the BPF_ABS and BPF_IND instructions are built-in special instructions that can directly access the data field data in the kernel data sk_buff. BPF_XADD is an instruction for atomic operations in multiprocessor scenarios.

[0046] eBPF has many program types, each with a specific hook point in the system kernel. When an eBPF program is attached to a hook point and the hook point is triggered, the corresponding eBPF program is executed. XDP programs are a type of eBPF, and their hook points can be in the kernel protocol stack, network card driver (DevXDP), or even network card hardware (HWXDP). Their performance increases progressively from low to high. HWXDP mode programs can run eBPF programs directly on network cards equipped with FPGAs (Field-Programmable Gate Arrays), offering the most powerful performance, but are expensive, and dedicated XDP hardware is currently very limited. DevXDP mode adds hook points to the network card driver, allowing network packet processing to be completed at the driver level with minimal system overhead. GenericXDP mode hook points are in the kernel protocol stack, designed to allow network cards whose drivers do not support XDP to run XDP programs, but it also has the worst performance.

[0047] In Linux systems, network applications use sockets to communicate with the system and send and receive network packets. Applications use the following interface to create a socket: `int socket(int family, int type, int protocol)`. Here, `family` represents the protocol family, commonly including `AF_INET` (IPv4 communication), `AF_PACKET` (retrieving raw packets), etc. `type` represents the protocol type, commonly including `SOCK_STREAM` (stream socket), `SOCK_DGRAM` (packet socket), `SOCK_RAW` (raw socket), etc. `protocol` represents the packet protocol type. For `AF_INET`, its value is 0; for `AF_PACKET`, it is the specific protocol, for example, `proto=0x0800` indicates that only IPv4 packets are received.

[0048] The new network socket AF_XDP (also known as XSK) in the kernel is also created through the ordinary socket() system call: fd = socket(AF_XDP, SOCK_RAW, 0). Upon successful execution, this function call returns an integer value representing the AF_XDP handle. Applications bound to this handle can quickly receive network packets with the help of eBPF programs. A map is a database allocated in kernel space that can be used for data transfer between eBPF programs within the kernel, as well as for data exchange between user programs and eBPF. The database itself organizes data using key-value pairs.

[0049] XDP (Extreme High-Speed ​​Traffic Capture) technology employs the XDP architecture and utilizes the eBPF virtual mechanism to establish a three-dimensional capture framework at the driver, middleware, and application layers. This enables high-performance, full packet capture, achieving high-speed traffic capture. Leveraging XDPdriver hook technology, the XDP program executes within the network device driver without context switching; the program executes once for each packet received by the network device. The relevant code is implemented as a kernel library function, allowing the program to execute directly within the device driver without switching to user-space context. Execution at the earliest point in the software capable of processing packets achieves optimal performance.

[0050] The eBPF virtual machine is where XDP programs execute. eBPF is an extension of the earlier BSD packet filter (BPF), which has been widely used in various packet processing tools over the past few decades. The eBPF virtual machine supports dynamic loading and reloading of programs, and the kernel manages the lifecycle of all BPF programs.

[0051] eBPF is a RISC register machine with 11 64-bit registers, a program counter, and a fixed-size 512-byte stack. Nine registers are general-purpose read / write, one is a read-only stack pointer, and the program counter is implicit, meaning jumps can only be made to a specific offset of the counter. The VM registers are always 64 bits wide (even when running on a 32-bit ARM processor core) and support 32-bit sub-register addressing if the most significant 32 bits are zero. These registers are: r0: stores the return value of function calls and the current program exit code; r1-r5: used as arguments for function calls, with r1 containing a "context" parameter pointer at program start; r6-r9: these are reserved between kernel function calls; r10: a read-only pointer to the 512-byte stack for each eBPF program.

[0052] The eBPF program type provided at load time precisely determines which subset of kernel functions can be called, as well as the "context" parameters provided via r1 at program startup. The meaning of the program exit value stored in r0 is also determined by the program type. Each function call can have a maximum of five arguments in registers r1-r5; this applies to eBPF-to-eBPF and kernel function calls. Registers r1-r5 can only store numbers or pointers to the stack (passed as arguments to functions), never direct pointers to arbitrary memory. All memory accesses must first load data onto the eBPF stack before it can be used in an eBPF program. This restriction helps the eBPF verifier, which simplifies the memory model for easier correctness checking.

[0053] eBPF instructions are also fixed-size 64-bit codes, with approximately 100 instructions divided into 8 classes. The VM supports 1-8 byte load / store operations, forward / backward (NOT) conditional jumps, arithmetic / logical operations, and function calls from general memory (map, stack, "context" such as packet buffers, etc.).

[0054] BPF maps are a technology where eBPF programs execute when a kernel event is triggered (e.g., a packet reception event triggers the execution of an XDP program). Each time a program executes, its initial state is the same (i.e., the program is stateless), and it cannot directly access the persistent storage (BPF map) in the kernel. Therefore, the kernel provides helper functions for accessing the BPF map. The BPF map is a key / value store, defined upon loading an eBPF program. Its uses include: persistent storage (e.g., an eBPF program retrieves its previous state from it each time it executes); coordinating two or more eBPF programs (e.g., one writing data and another reading data); and communication between user-space programs and kernel eBPF programs.

[0055] Traffic acquisition in high-speed networks based on XDP technology mainly includes the following steps: Step 1: Create AF_XDP socket: Create an AF_XDP socket using the socket() system call and generate a file descriptor pointing to the packet storage area; Step 2: Map queues: Configure and create the RX_RING, TX_RING, FILL_RING, and COMPLETION_RING associated with the AF_XDP socket and UMEM using the _RINGsetsockopt() system call, and map them using mmap(); Step 3: Bind AF_XDP socket to interface: Before transmitting traffic, the bind() call must be used to bind the AF_XDP socket to a device and the queue specified by that device; Step 4: Forward packets to XDP socket: First, create a BPF mapping to associate the queue and the AF_XDP socket, then assemble the BPF program, and finally attach the BPF program to the target interface. The application uses the bpf() system call to put the AF_XDP socket into BPF for mapping, and the XDP program redirects packets to the socket according to the mapping index. During this process, it is verified whether the socket is indeed bound to the device and the specified queue. If the binding to the device is unsuccessful, the message will be discarded. Step 5: Receive data packets: Data packets are received by transferring the file descriptor pointing to the memory buffer address through FILL_RING and RX_RING. Step 6: Stop data packet transmission: When the XDP program needs to stop data packet transmission at the end of its execution, the close() system call is used to stop the data stream and release the XDP socket, and munmap() is used to cancel the mapping of the circular queue.

[0056] In this embodiment, the DIRECT-IO mode is used to bypass the kernel buffer and directly write the traffic data captured by DPDK to the disk block device. Asynchronous I / O is used to submit batch write requests to reduce context switching. The LZ4 compression algorithm is used to perform lossless compression on the original packets, reducing storage space usage by more than 60%. By extracting the packet timestamp as the index key, a time-series index structure based on a B+ tree is constructed. The compressed data and index metadata are written synchronously to obtain a traffic database that combines high compression ratio and millisecond-level retrieval response.

[0057] In this embodiment, the rte_pktmbuf library of DPDK is used for memory pool initialization, pre-allocating a contiguous physical memory region and constructing a user-space buffer; the receive queue is configured through the rte_eth_rx_queue_setup function, directly binding the network card DMA descriptor to the mbuf structure in the memory pool; memory alignment optimization technology is used to ensure that the starting address of each mbuf is aligned with the cache line boundary, reducing CPU prefetch overhead and obtaining a zero-copy data receive path without kernel intervention.

[0058] In this embodiment, the XDP program architecture is used to embed processing logic in the network card driver layer. The five-tuple information is obtained by parsing the first 128 bytes of the packet header. The source / destination IP, port and protocol type are hashed and matched with a preset rule table to quickly determine the category of the data flow. Differential processing actions are performed based on the classification results to obtain data packets carrying flow classification tags. The processing actions include allowing the flow to be forwarded directly to user space, discarding illegal flows, and marking suspicious flows.

[0059] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-speed traffic acquisition method based on DPDK and XDP optimization strategies, characterized in that, The high-speed traffic acquisition method based on DPDK and XDP optimization strategies includes the following steps: The environment deployment is completed using DPDK scripts and tools, the UIO module is loaded and the network card registers are mapped to user space, the PMD driver is configured to reduce interrupt overhead, and the initialized DPDK runtime environment is obtained. By configuring 2MB / 1GB of huge page memory through Linux system boot parameters, and allocating a contiguous physical memory region in DPDK, the number of page table entries is reduced and the TLB hit rate is improved, resulting in optimized memory access performance. By utilizing the CPU affinity API of DPDK, the packet processing thread can be bound to a specific CPU core to avoid cache invalidation caused by cross-core switching, thus obtaining a stable task processing unit. Write eBPF bytecode to implement packet filtering logic, generate an executable file using the BPF compiler, and load it into the kernel driver layer using the bpf system call to obtain a high-efficiency kernel-mode processing program; Configure DPDK to receive data packets to the user buffer using zero-copy technology. The XDP program performs preliminary classification at the driver layer and passes metadata through BPF mapping to obtain high-speed captured and preprocessed traffic data. By using DIRECT-IO and asynchronous IO to write traffic data to disk, combined with compression algorithms to reduce storage usage, and establishing a time index to achieve fast retrieval, a highly efficient and queryable traffic database is obtained.

2. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The process of deploying the environment using DPDK scripts and tools, loading the UIO module and mapping the network card registers to user space, configuring the PMD driver to reduce interrupt overhead, and obtaining the initialized DPDK runtime environment includes the following steps: The DPDK environment is set up on a general-purpose server by installation and configuration, and device memory management and hardware interrupt handling are achieved by loading the UIO module. The mmap technology is used to obtain the management capabilities of physical memory, virtual memory, and logical memory, and to map the network card hardware registers to user space; The PMD polling mode driver technology is used to replace the traditional interrupt operation mode, reducing interrupt overhead and obtaining an initialized DPDK runtime environment.

3. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The method of configuring 2MB / 1GB huge page memory through Linux system boot parameters, allocating contiguous physical memory regions in DPDK, reducing the number of page table entries and improving TLB hit rate to achieve optimized memory access performance includes the following steps: The method involves adding specific kernel parameters during Linux system startup. Huge pages are configured by setting default_hugepagesz=2MB hugepagesz=2MB hugepages=N, hugepagesz=1GB hugepages=M, where N is the specific number of pages and M is 1GB of pages. During the DPDK initialization phase, a contiguous huge page physical memory region is requested by calling the memory management API; By directly mapping virtual addresses to physical addresses, an optimized memory performance environment is obtained, which reduces memory access latency and improves data performance.

4. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The method of using the CPU affinity API of DPDK to bind the packet processing thread to a specified CPU core, avoiding cache invalidation caused by cross-core switching, and obtaining a stable task processing unit includes the following steps: By using the CPU affinity API of DPDK, the data packet processing thread can be precisely allocated to a specified physical core by parsing the binding relationship parameters between the thread and the CPU core. By closely linking the thread execution context with the CPU cache level, the thread is fixed to execute on a specific core, thus preventing cache contents from becoming invalid when tasks are migrated. This allows instructions and data to reside continuously in the core private cache, reducing cache line filling overhead and resulting in a stable task processing unit with low latency and high hit rate.

5. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The process of writing eBPF bytecode to implement packet filtering logic, generating an executable file using a BPF compiler, and loading it into the kernel driver layer using the BPF system call to obtain a high-efficiency kernel-mode processing program includes the following steps: A packet filtering program was written in C language in conjunction with eBPF, specifying the matching rules and processing actions for L2-L7 layer fields. The source code was converted into eBPF bytecode by the LLVM compiler to generate a binary file that conforms to the kernel verification specification. The bytecode is injected into the kernel using the BPF system call, the verifier checks the legality of the instructions, and the bytecode is then attached to the XDP hook point in the network card driver layer. By deeply integrating the filtering logic with the hardware packet receiving path, a high-efficiency kernel-mode packet processing program is obtained without context switching.

6. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The DPDK configuration uses zero-copy technology to receive data packets to the user buffer. The XDP program performs preliminary classification at the driver layer, and transmits metadata through BPF mapping to obtain high-speed captured and preprocessed traffic data, including the following steps: The zero-copy receive queue is configured using the rte_pktmbuf library of DPDK, and the network card DMA descriptor is directly mapped to the user space memory pool. The XDP program parses the five-tuple information in the packet header at the driver layer and performs flow classification actions according to preset rules. The classification results and message pointers are encapsulated into BPF mapping entries, which are then passed through the BPF_MAP_TYPE_PERF_EVENT_ARRAY mapping table shared between the kernel and user space, resulting in traffic data that skips kernel protocol stack processing and carries pre-classification tags.

7. The high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 1, characterized in that, The process of writing traffic data to disk using DIRECT-IO and asynchronous IO, combined with compression algorithms to reduce storage usage, and establishing a time index to achieve fast retrieval, resulting in an efficient and queryable traffic database, includes the following steps: By using the DIRECT-IO mode to bypass the kernel buffer, the traffic data captured by DPDK is directly written to the disk block device, and batch write requests are submitted through asynchronous I / O to reduce context switching; The LZ4 compression algorithm is used to perform lossless compression on the original message, reducing the storage space usage by more than 60%. By extracting the message timestamp as the index key, a time series index structure based on B+ tree is constructed. Compressed data and index metadata are written synchronously to obtain a traffic database that combines high compression ratio and millisecond-level retrieval response.

8. A high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 6, characterized in that, The method of configuring a zero-copy receive queue using the rte_pktmbuf library of DPDK to directly map the network card DMA descriptor to the user-space memory pool includes the following steps: The memory pool is initialized using the rte_pktmbuf library of DPDK, which pre-allocates a contiguous physical memory region and constructs a user-mode buffer. Configure the receive queue using the rte_eth_rx_queue_setup function to directly bind the network card DMA descriptor to the mbuf structure in the memory pool; By using memory alignment optimization techniques to ensure that the starting address of each mbuf is aligned with the cache line boundary, CPU prefetch overhead is reduced, resulting in a zero-copy data receiving path that does not require kernel intervention.

9. A high-speed traffic acquisition method based on DPDK and XDP optimization strategies according to claim 6, characterized in that, The step of parsing the five-tuple information in the packet header at the driver layer using the XDP program and performing flow classification according to preset rules includes the following steps: The XDP program architecture is used to embed processing logic in the network card driver layer, and the five-tuple information is obtained by parsing the first 128 bytes of the packet header. The source / destination IP, port, and protocol type are hashed and matched with a preset rule table to quickly determine the category to which the data stream belongs. Based on the classification results, differentiated processing actions are performed to obtain data packets carrying flow classification labels. The processing actions include allowing the flow to be directly forwarded to user space, discarding illegal flows, and marking suspicious flows.

Images