A Zero-Copy-Based Multi-Threaded High-Throughput Data Stream Channel Separation Method and System

Abstract

This invention belongs to the field of data transmission and processing technology, and discloses a method and system for separating high-throughput data stream channels based on zero-copy multithreading. Through a data receiving module, the mixed data stream is directly written to a front-end buffer configured as a ring structure using direct memory access. Subsequently, a multithreaded processing module dynamically allocates multiple processing threads from a thread pool to separate the data of each channel in parallel. Each thread uses a pointer offset-based zero-copy algorithm to locate the data of each channel in memory and create pointer references, associating them with the corresponding back-end buffer, achieving logical separation without physical copying. Finally, a data storage module efficiently writes the separated data to persistent storage using asynchronous I / O. This invention combines zero-copy, multithreaded parallelism, and a two-level dynamic buffer strategy, significantly improving data separation efficiency, greatly reducing CPU usage and memory bandwidth, and ensuring the real-time performance and stability of high-throughput data processing.

Description

Technical Field

[0001] This invention belongs to the field of data transmission and processing technology, specifically relating to an efficient data stream channel separation method and system based on zero-copy and multi-threaded architecture, which is suitable for real-time separation, processing and storage of multi-channel mixed data in high-speed network environments. Background Technology

[0002] With the rapid development of industrial automation and test and measurement technologies, multi-channel data acquisition and processing systems have been widely used in industrial control, scientific research, signal analysis, and other fields. However, when processing high-throughput, multi-channel mixed data streams, traditional data processing systems generally face the following key technical challenges:

[0003] 1) The data transmission volume is large and the real-time requirements are high. Traditional processing methods are prone to data backlog and loss.

[0004] 2) Multi-channel data is transmitted in the form of mixed streams, and the memory copying operation during the separation of data in each channel leads to excessive consumption of system resources.

[0005] 3) The data caching management mechanism lacks flexibility. The fixed-size buffer is difficult to adapt to changing data traffic, and there is a risk of memory overflow.

[0006] 4) In single-threaded processing mode, channel data separation and storage operations block each other, making it impossible to fully utilize the parallel processing capabilities of modern multi-core processors.

[0007] Existing technologies typically employ simple data queues or fixed-size buffers, separating data across channels through repeated memory copying. This approach is inefficient when processing high-speed, high-volume data streams, exhibiting high CPU utilization, significant processing latency, and difficulty in guaranteeing real-time performance. Especially as the number of data channels increases, system performance degrades sharply, failing to meet the demands of application scenarios.

[0008] Therefore, the industry urgently needs a high-efficiency solution that can minimize memory operations and achieve high-speed separation and parallel processing of multi-channel data in order to break through the key technical bottlenecks in high-throughput data processing. Summary of the Invention

[0009] The purpose of this invention is to propose a high-throughput data stream channel separation method and system based on zero-copy and multi-threaded architecture to solve the problems mentioned in the background art. Through an efficient data separation algorithm, mixed multi-channel data streams are separated in real time. A multi-threaded parallel processing mechanism then transmits the separated data to each independent channel, ultimately achieving high-speed real-time data separation and storage.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a method and system for separating multi-threaded high-throughput data stream channels based on zero-copy, wherein the implementation architecture of the system includes: a data receiving module, a multi-threaded processing module, and a data storage module.

[0011] Furthermore, the data receiving module consists of a network receiving unit, a data buffering unit, and a data preprocessing unit.

[0012] Furthermore, the multi-threaded processing module includes a thread management unit, a data separation unit, and a buffer management unit. The data separation unit implements a zero-copy data separation algorithm based on pointer offset.

[0013] Furthermore, the data storage module includes a storage management unit, a file system interface unit, and a performance monitoring unit.

[0014] Furthermore, the data receiving module connects to the data source via a high-speed network interface and uses Direct Memory Access (DMA) to provide a stable data stream input.

[0015] Furthermore, the multi-threaded processing module connects to the data receiving module and the data storage module through a zero-copy mechanism that creates pointer references, providing multiple independent channel data streams after data separation.

[0016] Furthermore, the data storage module can dynamically adjust the storage strategy according to different storage needs, and adopt asynchronous I / O to achieve efficient data storage and management.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. This invention adopts a pointer offset-based zero-copy data separation algorithm, which avoids redundant data copying operations through direct memory references. Compared with traditional methods, it significantly improves processing speed and greatly reduces CPU usage and memory bandwidth consumption.

[0019] 2. This invention implements a two-level buffer architecture based on front-end-back-end separation. Combined with an adaptive capacity adjustment algorithm and an asynchronous I / O write strategy, it effectively prevents buffer overflow and processing blocking problems under high-speed data flow, and ensures low latency in data processing.

[0020] 3. This invention supports dynamic adjustment of the number of processing threads and buffer capacity, and can adaptively optimize system resource allocation according to real-time load, flexibly adapting to data acquisition needs of different scales. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0022] Figure 1This is a schematic diagram of the overall system architecture and data processing flow provided in the embodiments of the present invention;

[0023] Figure 2 This is a schematic diagram of the core logic of the zero-copy separation algorithm provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of multi-threaded management and task allocation provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a two-level buffering strategy provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] In the accompanying diagrams, solid arrows typically indicate the direction of data flow, while dashed arrows typically indicate control signals or logical connections. The boxes represent functional modules or processing units, and the connecting lines between them represent the data or control interactions between them.

[0028] Please see Figure 1 This invention provides a zero-copy, multi-threaded, high-throughput data stream channel separation system. Its overall architecture comprises three main modules: a data receiving module, a multi-threaded processing module, and a data storage module. These modules are connected via efficient data transmission interfaces, forming a complete data processing chain.

[0029] The data receiving module includes a network receiving unit, a data buffering unit, and a data preprocessing unit. The network receiving unit is responsible for receiving the raw, mixed data stream from the data source via a high-speed network interface, such as Gigabit Ethernet.

[0030] In a preferred embodiment, the unit employs an efficient transfer mechanism based on Direct Memory Access (DMA) technology, whereby the network device directly writes data packets into the system's physical memory. This avoids the multiple copies that occur in traditional network protocol stacks due to data transfer between kernel space and user space.

[0031] The core of the data caching unit is a front-end buffer. This buffer is constructed as a zero-copy-based ring buffer structure and managed using a single producer-single consumer SPSC model. In this model, the network card driver acts as the sole writer (producer), and the multi-threaded processing module acts as the sole reader (consumer), thereby achieving a high-performance data transmission path with lock-free and non-redundant replication.

[0032] The data preprocessing unit is responsible for parsing and preprocessing the received data packets. Its tasks include checking data packet integrity, parsing protocol headers such as the frame header and trailer, and the channel identification information (Channel ID). For example, a data packet can be defined as containing a 4-byte frame header, where the high 2 bytes are the synchronization code and the low 2 bytes contain the channel ID and the length information of the subsequent data payload.

[0033] The multi-threaded processing module is the core part of this system, as shown in the attached diagram. Figure 1 and Figure 3 As shown, it mainly includes a thread management unit, a data separation unit, and a buffer management unit.

[0034] The thread management unit is responsible for the creation, allocation, and coordination of threads. It employs a hierarchical architecture to initialize and maintain a thread pool. This unit achieves dynamic load balancing by monitoring the workload of each processing thread and the system resource utilization in real time. Its preset strategies include, for example, adding new processing threads to increase processing capacity when the overall CPU utilization is consistently above 70%; conversely, reclaiming idle threads to optimize resource utilization when the system load is low.

[0035] The data separation unit is the core of the multi-threaded processing module. It consists of multiple processing threads from the thread pool and achieves efficient data separation by executing a pointer offset-based zero-copy data separation algorithm.

[0036] Please see Figure 2 The core idea of ​​this algorithm is to directly manipulate the original data using memory pointers, thereby completely avoiding multiple memory copies in traditional data processing. Its specific implementation steps are as follows:

[0037] The first step is data block location. The processing thread obtains a mixed data stream from the front-end buffer, calculates the pointer offset based on the channel ID and data length parsed by the preprocessing unit, and thus directly locates the exact memory location of the data belonging to a specific channel in the front-end buffer.

[0038] The second step is to create pointer references. At this stage, physical memory copying is not performed. Instead, a pointer reference pointing to the original data's memory address is created, logically associating a specific portion of the original data with the corresponding channel's backend buffer. This pointer reference can be a descriptor structure containing a starting address and data length. To ensure the safety of pointer references, the system also implements a pointer lifecycle management mechanism. For example, using reference counting, it ensures that the space occupied in the frontend buffer can only be overwritten and reused after all references to a data block have been released and the data block has been successfully written to persistent storage by the downstream data storage module. This effectively avoids the problem of dangling pointers and data inconsistency.

[0039] The third step is concurrency control. In order to achieve safe and efficient data separation in a multi-threaded environment, the system adopts concurrency control strategies. For example, when updating shared read and write pointers, atomic operations of type atomic_fetch_add are used to ensure the indivisibility of the operation. When processing complex data blocks, a locking mechanism can be used to protect the data blocks to prevent multiple threads from modifying them at the same time and causing data competition.

[0040] Furthermore, to achieve efficient multi-threaded collaboration, data is transferred between processing thread groups through lock-free queues. For example, after creating pointer references, the thread group responsible for channel resolution places these pointer references into a lock-free queue. The thread group responsible for data writing then retrieves these references and performs subsequent storage operations, thereby decoupling the processing stages and achieving efficient parallelism.

[0041] The buffer management unit is responsible for implementing a high-performance, low-latency two-level buffering strategy. Please refer to [link / reference]. Figure 4 This strategy is based on a two-tier architecture, consisting of a high-speed front-end buffer and multiple parallel back-end buffers. The front-end buffer adopts a ring structure, is configured with a large initial capacity, and supports dynamic capacity adjustment.

[0042] Specifically, automatic expansion is triggered when the utilization rate exceeds 80%; when the utilization rate remains below 20%, a shrinking operation is performed to optimize memory resource allocation. The backend buffer adopts a channel isolation architecture, allocating independent buffer space for each data channel to achieve physical isolation and parallel processing of data streams.

[0043] The data storage module includes a storage management unit, a file system interface unit, and a performance monitoring unit. The storage management unit is responsible for implementing and managing various storage strategies, such as cyclic overwrite or time-sharing. The file system interface unit is responsible for file creation, writing, and closing operations.

[0044] The system employs asynchronous I / O technology. When the data accumulation in any backend buffer reaches a preset write threshold, such as 50% of its capacity, an asynchronous write operation is immediately triggered to write the data to persistent storage. This process does not block subsequent data reception and processing. The performance monitoring unit collects system operating status in real time, including metrics such as CPU utilization, memory usage, data throughput, and buffer utilization, and provides an intuitive monitoring interface and performance alerts.

[0045] In summary, the system of the present invention can stably process data streams from multiple channels, ensure data integrity and avoid system crashes, meet the requirements of a high-reliability data acquisition system, and has good applicability and scalability.

[0046] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for separating multi-threaded high-throughput data stream channels based on zero-copy, characterized in that: The following steps are involved: Step S1, Data Reception: The data receiving module receives a mixed data stream containing data from multiple channels from the data source, and directly writes the mixed data stream into a front-end buffer using Direct Memory Access (DMA). Step S2, Data Processing: The mixed data stream in the front-end buffer is processed by a multi-threaded processing module. The multi-threaded processing module includes a thread pool and dynamically allocates multiple processing threads from the thread pool to separate the data of each channel in the mixed data stream in parallel. Each processing thread performs the following operations: A zero-copy data separation algorithm is adopted to directly locate the channel data belonging to a specific data channel in the front-end buffer by calculating the pointer offset; Create a pointer reference to the located channel data and pass the pointer reference to the back-end buffer corresponding to the specific data channel for association, thereby realizing the logical separation of channel data in the mixed data stream; Step S3, Data Storage: Through the data storage module, channel data referenced by pointers in one or more back-end buffers is written to the persistent storage medium in an asynchronous I / O manner.

2. The method for separating multi-threaded high-throughput data stream channels based on zero-copy according to claim 1, characterized in that, The front-end buffer is a circular buffer and is managed using a single producer-single consumer (SPSC) model.

3. The method for separating multi-threaded high-throughput data stream channels based on zero-copy according to claim 1, characterized in that, The multi-threaded processing module also performs dynamic load balancing by dynamically monitoring the workload and system resource utilization of each processing thread and dynamically adjusting the number of active threads in the thread pool according to a preset strategy.

4. The method for separating multi-threaded high-throughput data stream channels based on zero-copy according to claim 1, characterized in that, The method also includes a two-level buffer management strategy, which includes: dynamically adjusting the capacity of the front-end buffer; and / or allocating an independent, physically isolated back-end buffer for each data channel.

5. The method for separating multi-threaded high-throughput data stream channels based on zero-copy according to claim 4, characterized in that, When the amount of data in any of the back-end buffers reaches a preset write threshold, the asynchronous I / O operation is automatically triggered.

6. The method for separating multi-threaded high-throughput data stream channels based on zero-copy according to claim 1, characterized in that, The data processing steps also include concurrency control strategies, which include using locking mechanisms and / or atomic operations to ensure data consistency in a multi-threaded environment.

7. A multi-threaded high-throughput data stream channel separation system based on zero-copy, characterized in that, include: The data receiving module is configured to receive a mixed data stream containing data from multiple channels and to write the mixed data stream directly into a front-end buffer using a direct memory access (DMA) method. A multi-threaded processing module, connected to the data receiving module, is configured to dynamically allocate multiple processing threads to separate data from each channel of the mixed data stream in the front-end buffer in parallel, wherein each processing thread performs the following operations: A zero-copy data separation algorithm is adopted to directly locate the channel data belonging to a specific data channel in the front-end buffer by calculating the pointer offset; Create pointer references to the located channel data and associate the pointer references with the back-end buffer corresponding to the specific data channel, thereby achieving logical separation of channel data in the mixed data stream; The data storage module, connected to the multi-threaded processing module, is configured to write channel data pointed to by pointers in the back-end buffer to the persistent storage medium in an asynchronous I / O manner.

8. The multi-threaded high-throughput data stream channel separation system based on zero-copy according to claim 7, characterized in that, The multi-threaded processing module includes: The thread management unit is used to create and manage a thread pool. The data separation unit, consisting of multiple processing threads allocated from the thread pool, is used to perform the creation of the pointer reference; The buffer management unit is used to manage the front-end buffer and multiple back-end buffers corresponding to each data channel.

9. The multi-threaded high-throughput data stream channel separation system based on zero-copy according to claim 8, characterized in that, The thread management unit is also equipped with a dynamic load balancing function, which can dynamically adjust the number of processing threads according to the real-time system load.

10. The multi-threaded high-throughput data stream channel separation system based on zero-copy according to claim 8, characterized in that, The buffer management unit is also equipped with a dynamic capacity adjustment function, which can automatically adjust the capacity of the front-end buffer according to its usage rate.

Images