Multi-thread high-throughput data flow channel separation method and system based on zero copy

Abstract

The invention belongs to the technical field of data transmission and processing, and discloses a zero-copy-based multi-thread high-throughput data stream channel separation method and system, and the method comprises the steps: directly writing a mixed data stream into a front-end buffer region configured as an annular structure through a data receiving module by adopting direct memory access; then, a multi-thread processing module dynamically allocates a plurality of processing threads from a thread pool to separate channel data in parallel, each thread adopts a zero copy algorithm based on pointer offset, positions the channel data in a memory, creates pointer reference and associates the channel data to a corresponding rear-end buffer area, and logic separation is achieved without physical copy; and finally, the data storage module efficiently writes the separated data into persistent storage in an asynchronous I / O mode. According to the method, zero-copy, multi-thread parallel and two-stage dynamic buffering strategies are combined, the data separation efficiency is remarkably improved, CPU occupation and memory bandwidth are greatly reduced, and the real-time performance and stability of high-throughput data processing are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of data transmission and processing technology, and specifically relates to an efficient data stream channel separation method and system based on zero-copy and multi-threaded architecture, which is suitable for real-time separation, processing and storage of multi-channel mixed data in a high-speed network environment. Background Art

[0002] With the rapid development of industrial automation and test and measurement technologies, multi-channel data acquisition and processing systems have been widely used in industrial control, scientific research, signal analysis and other fields. However, when processing high-throughput, multi-channel mixed data streams, traditional data processing systems generally face the following key technical challenges: 1) The data transmission volume is large and the real-time requirement is high. Traditional processing methods are prone to data backlog and loss.

[0003] 2) Multi-channel data is transmitted in the form of mixed streams, and the memory copy operation during the separation of channel data leads to excessive consumption of system resources.

[0004] 3) The data cache management mechanism lacks flexibility. The fixed-size buffer is difficult to adapt to changing data traffic and there is a risk of memory overflow.

[0005] 4) In single-threaded processing mode, channel data separation and storage operations block each other, making it impossible to fully utilize the parallel processing capabilities of modern multi-core processors.

[0006] Existing technologies typically use simple data queues or fixed-size buffers, and repeatedly copy data between channels to separate data. This approach is inefficient when processing high-speed, high-volume data streams, resulting in high CPU utilization, significant processing latency, and difficulty ensuring real-time performance. In particular, as the number of data channels increases, system performance degrades dramatically, failing to meet the demands of application scenarios.

[0007] Therefore, the industry urgently needs an efficient solution that can minimize memory operations and achieve high-speed separation and parallel processing of multi-channel data to break through the key technical bottlenecks in high-throughput data processing. Summary of the Invention

[0008] The purpose of this invention is to propose a high-throughput data stream channel separation method and system based on a zero-copy and multi-threaded architecture to address the problems mentioned in the background art. Through an efficient data separation algorithm, mixed multi-channel data streams are separated in real time. The separated data is then transferred to each independent channel through a multi-threaded parallel processing mechanism, ultimately achieving real-time separation and storage of high-speed data.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a zero-copy-based multi-threaded high-throughput data stream channel separation method and system, the implementation architecture of which includes: a data receiving module, a multi-threaded processing module and a data storage module.

[0010] Furthermore, the data receiving module is composed of a network receiving unit, a data buffer unit and a data pre-processing unit.

[0011] Furthermore, the multi-thread processing module includes a thread management unit, a data separation unit and a buffer management unit, and the data separation unit implements a zero-copy data separation algorithm based on pointer offset.

[0012] Furthermore, the data storage module includes a storage management unit, a file system interface unit and a performance monitoring unit.

[0013] Furthermore, the data receiving module is connected to the data source via a high-speed network interface and adopts a direct memory access (DMA) method to provide a stable data stream input.

[0014] Furthermore, the multi-thread processing module is connected with the data receiving module and the data storage module by creating a zero-copy mechanism of pointer reference, thereby providing multiple independent channel data streams after data separation.

[0015] Furthermore, the data storage module can dynamically adjust storage strategies according to different storage requirements and use asynchronous I / O methods to achieve efficient data storage and management.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention adopts a zero-copy data separation algorithm based on pointer offset, which avoids redundant data copy operations through direct memory reference. Compared with traditional methods, the processing speed is significantly improved and the CPU usage and memory bandwidth consumption are greatly reduced.

[0017] 2. The present invention implements a two-level buffer architecture based on front-end-back-end separation, combined with an adaptive capacity adjustment algorithm and an asynchronous I / O write strategy, which effectively prevents buffer overflow and processing blocking problems under high-speed data flow, ensuring low latency in data processing.

[0018] 3. The present invention supports dynamic adjustment of the number of processing threads and buffer capacity, can adaptively optimize system resource allocation according to real-time load, and flexibly adapt to data collection needs of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0020] Figure 1A schematic diagram of the overall system architecture and data processing flow provided by an embodiment of the present invention; Figure 2 A schematic diagram of the core logic of the zero-copy separation algorithm provided by an embodiment of the present invention; Figure 3 A schematic diagram of multi-thread management and task allocation provided by an embodiment of the present invention; Figure 4 A schematic diagram of a secondary buffering strategy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] In the accompanying drawings, solid arrows generally indicate the direction of data flow, and dashed arrows generally indicate control signals or logical associations. Each box represents a functional module or processing unit, and the connecting lines between them represent the data or control interaction relationship between them.

[0023] See also Figure 1 The present invention provides a multi-threaded, high-throughput data stream channel separation system based on zero-copy technology. Its overall architecture comprises three main modules: a data receiving module, a multi-threaded processing module, and a data storage module. These modules are connected via efficient data transmission interfaces, forming a complete data processing chain.

[0024] The data receiving module includes a network receiving unit, a data buffer unit and a data pre-processing unit. The network receiving unit is responsible for receiving the original mixed data stream from the data source through a high-speed network interface, such as Gigabit Ethernet.

[0025] In a preferred embodiment, the unit adopts an efficient transmission mechanism based on direct memory access (DMA) technology, where the network device directly writes data packets into the system physical memory. This avoids the multiple copies generated by the traditional network protocol stack when data is transferred between kernel space and user space.

[0026] The core of the data cache unit is a front-end buffer. This buffer is constructed as a zero-copy ring buffer structure and managed in conjunction with the single-producer-single-consumer SPSC model. The network card driver is the sole writer, or producer, and the multi-threaded processing module is the sole reader, or consumer. This enables a high-performance data transmission path without locks or redundant replication.

[0027] The data preprocessing unit is responsible for parsing and preprocessing received data packets. This includes checking packet integrity, parsing protocol headers such as the frame header and trailer, and parsing the channel ID. For example, a data packet can be defined as consisting of a 4-byte frame header, where the upper 2 bytes are the synchronization code, and the lower 2 bytes contain the channel ID and the length of the subsequent data payload.

[0028] The multi-thread processing module is the core part of this system. Figure 1 and Figure 3 As shown, it mainly includes a thread management unit, a data separation unit and a buffer management unit.

[0029] The thread management unit is responsible for thread creation, allocation, and coordination. It employs a hierarchical architecture to initialize and maintain a thread pool. This unit implements dynamic load balancing by monitoring the workload of each processing thread and system resource utilization in real time. For example, if overall CPU utilization remains above 70%, new processing threads are added to increase processing capacity. Conversely, when system load is low, idle threads are recycled to optimize resource utilization.

[0030] The data separation unit is the core of the multi-threaded processing module. It consists of multiple processing threads in the thread pool and achieves efficient data separation by executing a zero-copy data separation algorithm based on pointer offset.

[0031] See also Figure 2 The core idea of ​​this algorithm is to use memory pointers to directly operate on original data, thus completely avoiding the multiple memory copies in traditional data processing. The specific implementation steps are as follows: The first step is data block positioning. The processing thread obtains a mixed data stream from the front-end buffer, calculates the pointer offset based on the channel ID and data length information parsed by the preprocessing unit, and thus directly locates the exact memory location of the data belonging to a specific channel in the front-end buffer.

[0032] The second step is to create a pointer reference. In this case, no physical memory copy is performed. Instead, a pointer reference pointing to the memory address of the original data is created to logically associate a specific portion of the original data with the backend buffer of the corresponding channel. The pointer reference can be a descriptor structure containing the starting address and data length. To ensure the security of pointer references, the system also implements a pointer lifecycle management mechanism. For example, through reference counting, it ensures that only when all references to a data block are released and the data block has been successfully written to persistent storage by the downstream data storage module, the space occupied by it in the frontend buffer can be overwritten and reused, effectively avoiding the problems of dangling pointers and data inconsistencies.

[0033] The third step is concurrency control. In order to achieve safe and efficient data separation in a multi-threaded environment, the system adopts a concurrency control strategy. For example, when updating shared read-write pointers, atomic operations such as atomic_fetch_add are used to ensure the indivisibility of the operation; when processing complex data blocks, a locking mechanism can be used to protect the data blocks to prevent multiple threads from modifying them at the same time and causing data competition.

[0034] Furthermore, to achieve efficient multi-threaded collaboration, data is transferred between processing thread groups via lock-free queues. For example, after creating pointer references, the thread group responsible for channel parsing places these pointer references into a lock-free queue. The thread group responsible for data writing retrieves these references and performs subsequent storage operations, thus achieving decoupling and efficient parallelization of each processing link.

[0035] The buffer management unit is responsible for implementing a high-performance, low-latency secondary buffering strategy. Figure 4 ,This strategy is based on a two-tier architecture, including a high-speed front-end buffer and multiple ,parallel back-end buffers. The front-end buffer adopts a ring structure, is configured with a large initial capacity, and supports ,dynamic capacity adjustment.

[0036] Specifically, when utilization exceeds an 80% threshold, automatic capacity expansion is triggered; when utilization remains below 20%, capacity contraction is performed to optimize memory resource allocation. The backend buffer utilizes a channel isolation architecture, allocating independent buffer space to each data channel, thereby achieving physical isolation and parallel processing of data flows.

[0037] The data storage module includes a storage management unit, a file system interface unit, and a performance monitoring unit. The storage management unit is responsible for implementing and managing various storage strategies, such as round-robin overwriting and time slicing. The file system interface unit is responsible for operations such as creating, writing, and closing files.

[0038] The system utilizes asynchronous I / O technology. When data accumulation in any backend buffer reaches a preset write threshold, such as 50% of its capacity, an asynchronous write operation is immediately triggered, writing the data to persistent storage without blocking subsequent data reception and processing. The performance monitoring unit collects real-time system operating status, including metrics such as CPU utilization, memory usage, data throughput, and buffer utilization, and provides an intuitive monitoring interface and performance alerts.

[0039] In summary, the system of the present invention can stably process data streams of multiple channels, ensure data integrity and avoid system crashes, meet the requirements of a high-reliability data acquisition system, and has good applicability and scalability.

[0040] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-threaded high-throughput data stream channel separation method based on zero copy, characterized in that: The following steps are involved: Step S1, data reception: receiving a mixed data stream containing multiple channel data from a data source through a data receiving module, and directly writing the mixed data stream into a front-end buffer using a direct memory access (DMA) method; Step S2, data processing: The mixed data stream in the front-end buffer is processed by a multi-threaded processing module. The multi-threaded processing module includes a thread pool and dynamically allocates multiple processing threads from the thread pool to separate the data of each channel in the mixed data stream in parallel. Each processing thread performs the following operations: Adopting a zero-copy data separation algorithm, by calculating a pointer offset, the channel data belonging to a specific data channel is directly located in the front-end buffer; Creating a pointer reference to the located channel data, and passing the pointer reference to a backend buffer corresponding to the specific data channel for association, thereby achieving logical separation of each channel data in the mixed data stream; Step S3, data storage: writing the channel data referenced by the pointer in one or more backend buffers to a persistent storage medium in an asynchronous I / O manner through a data storage module.

2. The zero-copy-based multi-threaded high-throughput data stream channel separation method according to claim 1, characterized in that: The front-end buffer is a ring buffer and is managed using a single-producer-single-consumer SPSC model.

3. The zero-copy-based multi-threaded high-throughput data stream channel separation method according to claim 1, characterized in that: The multi-thread processing module also performs dynamic load balancing by dynamically monitoring the workload and system resource utilization of each processing thread and dynamically adjusting the number of active threads in the thread pool according to a preset strategy.

4. The zero-copy-based multi-threaded high-throughput data stream channel separation method according to claim 1, characterized in that: The method further includes a secondary buffer management strategy, which includes: dynamically adjusting the capacity of the front-end buffer; and / or allocating an independent, physically isolated back-end buffer to each data channel.

5. The zero-copy-based multi-threaded high-throughput data stream channel separation method according to claim 4, characterized in that: When the amount of data in any of the backend buffers reaches a preset write threshold, the asynchronous I / O operation is automatically triggered.

6. The zero-copy-based multi-threaded high-throughput data stream channel separation method according to claim 1, characterized in that: The data processing step also includes a concurrency control strategy, which includes using a lock mechanism and / or using atomic operations to ensure data consistency in a multi-threaded environment.

7. A multi-threaded high-throughput data stream channel separation system based on zero-copy technology, characterized in that: include: a data receiving module configured to receive a mixed data stream containing data from multiple channels and directly write the mixed data stream into a front-end buffer using a direct memory access (DMA) method; a multi-thread processing module, connected to the data receiving module, and configured to dynamically allocate multiple processing threads to separate data of each channel in the mixed data stream in the front-end buffer in parallel by creating a pointer reference to the corresponding channel data in the front-end buffer and associating it with the corresponding back-end buffer; The data storage module is connected to the multi-thread processing module and is configured to write the channel data pointed to by the pointer reference in the backend buffer into a persistent storage medium in an asynchronous I / O manner.

8. The zero-copy-based multi-threaded high-throughput data stream channel separation system according to claim 7, characterized in that: The multi-thread processing module includes: A thread management unit, used to create and manage a thread pool; a data separation unit, composed of a plurality of processing threads allocated from the thread pool, for executing the creation of the pointer reference; The buffer management unit is used to manage the front-end buffer and a plurality of back-end buffers corresponding to each data channel.

9. The zero-copy-based multi-threaded high-throughput data stream channel separation system according to claim 8, characterized in that: The thread management unit is also configured with a dynamic load balancing function, which can dynamically adjust the number of processing threads according to the real-time system load.

10. The zero-copy-based multi-threaded high-throughput data stream channel separation system according to claim 8, characterized in that: The buffer management unit is also configured with a dynamic capacity adjustment function, which can automatically adjust the capacity of the front-end buffer according to the usage rate of the front-end buffer.

Images