A network data collection and processing system and method based on dpdk

Abstract

The invention discloses a DPDK-based network data collection and processing system and a method thereof, and relates to the field of massive data processing. This system is: the process management module interacts with the data collection and distribution module, the timer module and the state monitoring module respectively to realize management control; the state monitoring module interacts with the data collection and distribution module and the memory management module respectively to realize information collection and recording; The memory management module interacts with the data collection and distribution module and the TCP / IP protocol processing module to control the memory usage during work; the data collection and distribution module, the TCP / IP protocol processing module and the timer management module interact in turn to realize the original network data packet The collection, forwarding and initial restoration according to the TCP / IP protocol. The invention optimizes the factors affecting the performance of data processing in a targeted manner, improves the processing efficiency of the program, and can quickly reuse different business requirements, reduces the difficulty of development and shortens the development period.

Description

technical field [0001] The invention relates to the field of massive data processing, in particular to a DPDK-based network data collection and processing system and method thereof. Background technique [0002] Massive data processing platforms usually have high requirements on system processing performance, and need to process network data quickly and efficiently. With the development of technology, the processing performance of hardware is getting higher and higher, and resources such as multi-core platforms and large-capacity memory have greatly improved the efficiency of massive data processing, while the bottleneck of system processing is gradually inclined to processing software. [0003] From the perspective of software processing, the processing performance of the system often decreases rapidly with the increase of processing traffic and processing flow. There are many reasons for performance degradation. From the perspective of system calls and thread switching ov...

AI Technical Summary

Problems solved by technology

[0003] From the perspective of software processing, the processing performance of the system often decreases rapidly with the increase of processing traffic and the increase of processing processes.

There are many reasons for performance degradation. From the perspective of system calls and thread switching overhead, the reasons include the following points: ①Multi-thread processing performance is not linearly related to the number of threads, and the overhead caused by high-frequency switching between threads will cause The reduction of processing performance; ②The system interrupt 0x80 and the switching between user mode and kernel mode have certain overhead, and the impact brought by frequent system calls is particularly obvious; Reduced performance; ④ not fully utilizing the CPU resources of the multi-core platform

Images