A High Throughput Data Stream Processing Method

Abstract

The invention discloses a high-throughput data stream processing method, which receives and processes data by asynchronous data reception, uses a small number of receiving threads to send requests from as many clients as possible, makes full use of the CPU time of the receiving threads, and improves the system The overall throughput rate; the received data is sent to the business processing thread for processing as quickly as possible through the lock-free task scheduling method, which reduces the lock overhead and ensures the real-time performance of data processing; in addition, because data reception is an IO-intensive Data processing is a CPU-intensive operation. The two layers are separated and run on different ranges of CPU cores to ensure that the data receiving and processing processes do not interfere with each other, and there will be no data processing threads occupying the CPU for a long time. , the data receiving thread cannot be scheduled to run by the system, and the ratio of the number of receiving threads to the number of processing threads can be configured and balanced according to business requirements, making full use of the system's cpu and memory resources.

Description

technical field [0001] The invention relates to the field of high-performance network flow processing, in particular to a high-throughput data flow processing method. Background technique [0002] In current network projects, multi-level architecture design is often carried out, such as front-end data acquisition layer, data aggregation and processing layer, data storage layer, etc. The server on the data aggregation and processing layer usually needs to process hundreds of The front-end connection has instantaneous traffic of up to gigabits per second, and there is also a "surge" phenomenon, that is, there is a huge difference between the peak and valley values ​​of the traffic. If the peak traffic is too large, the server will not receive and process data in time, or even go down. Higher performance requirements are placed on the server. In order to ensure real-time and reliable reception and processing of data streams from front-end servers, existing solutions mainly exp...

AI Technical Summary

Problems solved by technology

[0008] This method uses a thread pool when receiving data, and the client no longer corresponds to the only receiving thread, and quickly reads data from the system buffer in an asynchronous manner, avoiding IO blocking waiting; a thread pool is used for processing, The ratio of the receiving thread to the processing thread can be configured. The disadvantage is that the receiving thread pool and the processing thread pool are in the mode of multi-reading and multi-writing to the queue, and mutual exclusion locks are required, resulting in a large system overhead.

Images