System and method for high-speed processing and interaction of mass data

Abstract

The invention discloses a system and a method for high-speed processing and interaction of mass data. The system comprises a data transceiver module, a central processing module, and a data processing module. The data transceiver module is used for receiving and transmitting data. The central processing module opens n ring buffer areas to buffer received data according to the number n of GPUs (Graphics Processing Unit), copies data to the system memory after a certain amount of data is accumulated or within a certain period of time, directly maps the system memory to each GPU in advance, and sends the processing result through the data transceiver module after processing by the GPUs. The data processing module acquires corresponding data in the memory through DMA (Direct Memory Access) mapping, makes the corresponding GPUs carry out multi-thread high-concurrency processing, and returns the processing result to the central processing module. High-speed interaction of mass data between NIC (Network Interface Card) and GPU is realized. The mass network data processing ability and interaction ability of the system are improved. A more reliable technical support is provided for real-time network analysis.

Description

technical field [0001] The invention relates to massive data processing and interaction technology, in particular to a system and method for high-speed processing and interaction of massive data. Background technique [0002] At present, the data packet processing of the network card adopts a multi-core multi-queue architecture, threads are bound to cores, and each thread processes data packets in a queue. A large ring buffer is first allocated in the kernel, and the buffer is cut into several memory nodes. When the network card receives the data packet, it obtains the memory node from the ring buffer, and DMAs the data to the memory node. And maintain a receiving queue ring, the memory node address of the data packet is saved in the ring. When the application opens a receiving queue to capture packets, the receiving queue ring and the large ring buffer allocated by the kernel will be mapped to the application process space, and the application layer only needs to obtain t...

AI Technical Summary

Problems solved by technology

This method is referred to as zero copy, which can greatly improve the performance of network data capture. The processing of data packets after capture is done on the CPU. However, in high-performance networks (>40GE), real-time network analysis faces great challenges. Even with multi-CPU and multi-threading, due to hardware limitations, the number of CPU cores cannot reach enough to achieve high concurrency, and packet loss is prone to occur.

Images