DPDK-based received packet two-level caching method and system

Abstract

The invention discloses a DPDK-based received packet two-level caching method. The method comprises the following steps that a mbulk structure and a two-level caching structure are customized, mbulksstore mbuf pointers and have the same size, received packet first-level queues store mbuf pointers, and a received packet second-level queue stores mbulk pointers; a DPDK is initialized; received packet caching is conducted, wherein a first received packet processing thread is used for caching portmbuf pointers into the received packet first-level queues, a second received packet processing threadis used for taking out the mbulks from a mbulk_pool memory pool, taking out the mbuf pointers from the corresponding received packet first-level queues and putting the mbuf pointers into the mbulks,and after any one mbulk is full, the second received packet processing thread puts the corresponding mbulk pointer into the received packet second-level queue and takes out the next mbulk from the mbulk_pool for next operation; and a public worker thread takes out the mbulk pointers from the received packet second-level queue and processes the mbulk pointers. According to the method, stable dequeuing operation needing the appropriate data amount is provided for a back end to enable back end processing to be more stable, therefore, real-time and stable packet receiving and processing are achieved, and the packet loss probability is decreased.

Description

technical field [0001] The present invention relates to the technical field of big data collection, and specifically relates to a packet receiving and caching method based on DPDK. This method makes data processing more stable and reduces the packet loss rate by adding a second-level pipeline. Background technique [0002] In big data applications such as PET, it is necessary to collect and analyze the flow data packets of multiple network outlets. As the accuracy required by the application becomes higher and higher, the corresponding data flow will also increase, and the rate of general applications will increase. To achieve 10Gbps, how to receive and process packets stably in real time and reduce the probability of packet loss is a challenge for data packet collection. [0003] At present, for data collection based on Linux, if traditional Socket packet capture is used, the program needs to switch frequently between user mode and kernel mode, resulting in weakened perform...

AI Technical Summary

Problems solved by technology

However, when it is necessary to process data with continuous large traffic (such as >10Gbps), this method is used for data caching. Due to the large amount of data, there is still the problem of packet loss.

Images