A high-speed network data real-time collection method

Abstract

The application discloses a high-speed network data real-time collection method, uses DPDK to acquire network data, and realizes IP collection, IP flow maintenance and IP flow feature calculation on a general server, and overcomes the problem that the prior art cannot simultaneously consider three application requirements.

Description

Technical Field

[0001] This invention belongs to the field of network data processing technology, and specifically relates to a method for real-time acquisition of high-speed network data. Background Technology

[0002] IP flow maintenance, flow feature calculation, and collection are common operations in network data processing. IP flow maintenance and flow feature calculation classify IP packets according to the five-tuples carried in the IP packets to obtain flow feature vectors for traffic control; IP collection directly filters and stores IP packets for offline analysis.

[0003] Existing technologies can be categorized into three types based on their underlying IP packet acquisition methods: libpcap-based processing, DPDK (Data Plane Development Kit)-based processing, and dedicated hardware-based processing. libpcap-based processing is the most common method, widely used across various operating system platforms. As an open-source library, libpcap can capture and parse network traffic, supports BPF filters, and is developer-friendly. However, it relies on network card interrupts to interact with the kernel, limiting performance under high load scenarios, with a typical throughput of approximately 1Gbps. DPDK-based processing is a general-purpose solution designed for high-performance requirements. DPDK directly acquires packets from the network interface through a polling mode, bypassing the kernel protocol stack and improving processing performance, with a typical throughput >10Gbps. Dedicated hardware-based processing uses FPGAs and ASICs to implement dedicated network interface cards, offloading most of the kernel's network data processing functions to hardware, with a typical throughput >100Gbps. However, this type of technology has high hardware costs and development complexity, and is mainly used in large-scale data centers.

[0004] Regardless of the acquisition method used, existing technologies typically focus on a single aspect in application. In IP flow maintenance and flow feature calculation applications, only a small amount of IP packet information is retained, without storing the original bytes of the packets; in IP acquisition applications, only packets that meet the criteria need to be stored, and flow maintenance is not required. Existing applications rarely combine IP acquisition with IP flow maintenance and IP flow feature calculation. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a high-speed network data real-time acquisition method. It uses DPDK to acquire network data and implements IP acquisition, IP flow maintenance, and IP flow feature calculation on a general-purpose server, thus overcoming the problem that existing technologies cannot simultaneously meet the three application requirements.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Data packet reception; Use the interface provided by the DPDK packet receiving module to receive IP packets from a general network card and write them into a circular buffer; Step 2: IP data parsing; Extract IP metadata from IP packets, including the 5-tuple and timestamp; IP metadata is stored directly in the circular buffer adjacent to IP packets and can be accessed through packet pointers; IP data parsing is completed independently and in parallel by multiple threads. Each thread takes multiple packets from the circular buffer in a polling manner to form a packet vector, and then parses out the metadata of each packet in the vector in turn. Step 3: IP Flow Maintenance; IP data is categorized into 5-tuples, and IP streams are output based on their end-of-stream status. IP stream maintenance is handled by multiple threads, each processing data independently. Specifically, each parsing thread distributes IP metadata evenly across L stream maintenance threads according to the 5-tuples. Each stream maintenance thread maintains a hash table where the key records the 5-tuple information of the data stream, and the value records a list of packet pointers belonging to that data stream. Each thread independently performs two tasks: processing IP metadata input and maintaining stream status and outputting the terminated stream. Each data stream maintains a separate read-write lock. Step 4: IP flow feature calculation and database entry; The completed IP stream contains a list of data packet pointers belonging to that stream. This part uses the data packet pointers to obtain IP data packet metadata and calculates the stream characteristics according to the backend application requirements. Step 5: IP data collection; After the IP stream is processed, the metadata in the IP stream is stored in different pcap files.

[0007] Preferably, the IP flow feature calculation and database entry, as well as the IP data collection, can all be accelerated using multi-threading as needed.

[0008] Preferably, a reference count is added to each data packet pointer, and the circular buffer is returned to release memory space after a data packet pointer completes feature calculation and data acquisition operations.

[0009] Preferably, the high-speed network data real-time acquisition method stores IP packets and metadata only once during the entire processing, reducing data copying overhead.

[0010] Preferably, the five-tuple includes the source IP address, destination IP address, source port, destination port, and protocol number.

[0011] Preferably, the stream features include a timestamp list and a packet length list.

[0012] An electronic device includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described high-speed network data real-time acquisition method.

[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described high-speed network data real-time acquisition method.

[0014] A chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to perform the above-described high-speed network data real-time acquisition method.

[0015] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described high-speed network data real-time acquisition method.

[0016] The beneficial effects of this invention are as follows: 1. This invention designs a processing framework based on DPDK that simultaneously meets the requirements of IP flow maintenance, feature calculation, and IP data collection. This framework can achieve a processing speed of 2 million packets / second on a conventional server. Based on an average packet length of 256 bytes, the data collection and disk writing speed is approximately 512MB / second.

[0017] 2. This invention is based on DPDK for network data collection and processing, and has good versatility.

[0018] 3. This invention adopts a multi-threaded parallel processing framework, which has a fast processing speed.

[0019] 4. This invention simultaneously supports IP flow maintenance, feature calculation, and IP data collection, and has high availability. Attached Figure Description

[0020] Figure 1 This is a framework diagram of the method of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] This invention provides a method for real-time acquisition of high-speed network data. This method uses DPDK to acquire network data and implements IP acquisition, IP flow maintenance, and IP flow feature calculation on a general-purpose server, overcoming the problem that existing technologies cannot simultaneously meet the three application requirements.

[0023] The overall processing flow of this invention is as follows: Figure 1As shown, it consists of five parts: data packet reception, IP data parsing, IP flow maintenance, IP flow feature calculation and database entry, and IP data collection.

[0024] Packet reception involves receiving IP packets from a general-purpose network interface card and writing them into a circular buffer. This part is done directly using the interface provided by DPDK.

[0025] IP data parsing extracts IP metadata from IP packets, including the 5-tuple (source IP address, destination IP address, source port, destination port, protocol number), timestamp, etc. This metadata is stored directly adjacent to the IP packet in a circular buffer and can be accessed through packet pointers. IP data parsing is performed independently and in parallel by multiple threads. Each thread retrieves multiple packets from the circular buffer in a polling manner to form a packet vector, and then parses the metadata of each packet in the vector sequentially. The design of multiple parsing threads effectively solves the problem that the packet parsing speed is slower than the packet receiving speed.

[0026] IP stream maintenance categorizes IP data into 5-tuples and outputs IP streams based on their end-of-stream status. IP stream maintenance can be performed by multiple threads, each processing data independently. Specifically, each parsing thread distributes IP metadata evenly across L stream maintenance threads according to the 5-tuples. Each thread maintains a hash table where the key records the 5-tuple information of the data stream, and the value records a list of data packet pointers belonging to that data stream. Each thread independently performs two tasks: processing IP metadata input and maintaining stream status and outputting terminated streams. IP metadata input requires processing the output of multiple parsing threads, while stream status maintenance and output require periodically scanning the hash table and deleting terminated streams. Both tasks require write operations to a data stream in the hash table; therefore, this invention mimics row-level locking in databases, maintaining a separate read-write lock for each data stream.

[0027] After the IP flow feature calculation and database processing are completed, the IP flow records a list of data packet pointers belonging to that flow. This part uses the data packet pointers to obtain IP data packet metadata and calculates flow features such as timestamp lists and packet length lists according to the requirements of the backend application.

[0028] After the IP data acquisition process is completed, the IP streams are stored in different pcap files according to the metadata in the IP streams, such as time, tag, IP address, etc.

[0029] IP stream feature calculation and database entry, as well as IP data acquisition, can all be accelerated using multi-threading as needed. Furthermore, since both parts require processing the list of packet pointers contained in the IP stream, this invention adds a reference count to each packet pointer. Once a packet pointer has completed feature calculation and data acquisition operations, it is returned to the circular buffer to release memory space.

[0030] This invention stores IP packets and metadata only once during the entire processing, reducing data copying overhead.

[0031] Example: Step 1: Use a single thread to receive data on multiple general-purpose network cards in a loop and write it into a circular buffer. When the buffer is full, discard the data packets. Step 2: Use J parsing threads to independently parse the data packets in the circular buffer. Here, only the 5-tuple and timestamp metadata of the data packets are obtained. Step 3: Based on the different hash values ​​of the quintuple data of the data packets, write them evenly to L stream maintenance threads. Each stream maintenance thread records the quintuple -> (data packet pointer list, start time, end time, data packet length list), etc. At the same time, there are L timeout polling threads to implement timeout stream output. Step 4: For the output timeout stream, calculate the characteristics of the data stream, such as the variance of the stream length, the stream duration, etc. The characteristic calculation should avoid parsing from the original data packets again, and should be obtained directly from the stream maintenance information. The calculated stream characteristics are written to the ClickHouse database in batches. Step 5: For the output timeout stream, write the data packets it contains to disk storage and release the space in the circular buffer.

[0032] This invention was implemented in C++ on a rack-mount server configured with: an Intel(R) Xeon(R) Gold 5120 CPU, two Ethernet Connection X722 10GbE network cards, 256GB DDR4 memory, a 30TB HDD, Ubuntu 20.04 system, and g++ 10.5.0. The ring buffer size was set to 80GB, the number of parsing threads to 7, and the number of stream maintenance threads to 1. A traffic generator was used to generate a specified number of data streams with a specified packet size. The measured peak processing capacity was 2 million packets / second, and the IP packet write speed was 256MB / second.

Claims

1. A method for real-time acquisition of high-speed network data, characterized in that, Includes the following steps: Step 1: Data packet reception; Use the interface provided by the DPDK packet receiving module to receive IP packets from a general network card and write them into a circular buffer; Step 2: IP data parsing; Extract IP metadata from IP packets, including the 5-tuple and timestamp; IP metadata is stored directly in the circular buffer adjacent to IP packets and can be accessed through packet pointers; IP data parsing is completed independently and in parallel by multiple threads. Each thread takes multiple packets from the circular buffer in a polling manner to form a packet vector, and then parses out the metadata of each packet in the vector in turn. Step 3: IP Flow Maintenance; The IP data is categorized into 5-tuples, and the IP stream is output based on the flow end status. IP stream maintenance is handled by multiple threads, and the data processed by each thread is independent of the others. Specifically, each parsing thread distributes IP metadata evenly to L stream maintenance threads according to the five-tuple; each stream maintenance thread holds a hash table, where the key records the five-tuple information of the data stream, and the value records a list of data packet pointers belonging to that data stream; each thread independently completes two tasks: processing IP metadata input and implementing stream state maintenance and outputting the terminated stream; each data stream maintains a separate read-write lock; Step 4: IP flow feature calculation and database entry; The completed IP stream contains a list of data packet pointers belonging to that stream. This part uses the data packet pointers to obtain IP data packet metadata and calculates the stream characteristics according to the backend application requirements. Step 5: IP data collection; After the IP stream is processed, the metadata in the IP stream is stored in different pcap files.

2. The high-speed network data real-time acquisition method according to claim 1, characterized in that, The IP flow feature calculation and database entry, as well as IP data collection, can all be accelerated using multi-threading as needed.

3. The high-speed network data real-time acquisition method according to claim 2, characterized in that, Add a reference count to each data packet pointer, and return the circular buffer to release memory space after a data packet pointer has completed feature calculation and data acquisition operations.

4. The high-speed network data real-time acquisition method according to claim 3, characterized in that, The high-speed network data real-time acquisition method only stores IP packets and metadata once during the entire processing, reducing data copying overhead.

5. The high-speed network data real-time acquisition method according to claim 4, characterized in that, The five-tuple contains the source IP address, destination IP address, source port, destination port, and protocol number.

6. The method for real-time acquisition of high-speed network data according to claim 5, characterized in that, The stream features include a timestamp list and a packet length list.

7. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

9. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 6.

Images