PXIe high-speed data zero-copy processing and real-time streaming disk method for domestic multi-core processor
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHENGDU CHANGBO INSTR CO LTD
- Filing Date
- 2026-05-07
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies in PXIe bus data processing suffer from multiple data copies, context switching overhead, lock contention and resource conflicts, and file system overhead, resulting in insufficient system bandwidth, processing latency, and storage speed, which cannot meet the requirements of high-speed data acquisition and real-time storage.
By employing VFIO technology to directly map the DMA buffer to user space, combined with asynchronous I/O technology and lock-free circular queues, the file system is bypassed. Through the AMP mode and thread pool technology of domestic multi-core processors, zero-copy data and real-time streaming disk are achieved, eliminating the performance bottlenecks in traditional solutions.
It achieves zero-copy data transfer throughout the process, reduces CPU load and transmission latency, enhances multi-core parallel processing capabilities, improves I/O throughput and real-time performance, and meets the requirements of high real-time performance and high reliability for independent control.
Smart Images
Figure CN122450874A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of PXIe bus data processing and storage technology, and in particular to a PXIe high-speed zero-copy data processing and real-time streaming disk method for domestic multi-core processors. Background Technology
[0002] PXIe (PCIe eXtensions for Instrumentation) bus, as a high-performance, modular instrument platform, is widely used in automated testing systems in aerospace, military electronics, and other fields. As the core of a PXIe system, the bus controller (usually an embedded computer) needs to have powerful data processing and transmission capabilities to meet the ever-increasing demands for high-speed data acquisition and real-time storage.
[0003] As the complexity of the objects under test increases, the amount of data that the PXIe bus needs to process is growing explosively, placing stringent demands on system bandwidth, processing latency, and storage speed. For example, in high-speed signal acquisition, radar, or communication testing, it is necessary to store up to 16GB / s of data from the PXIe backplane into local storage media in real time without loss.
[0004] Traditional data processing methods typically involve multiple data copies and frequent switching between kernel and user modes. Data flows through hardware DMA to the kernel buffer, is then copied by the CPU to the user-space application, and finally written to disk via the file system API. This process suffers from the following technical bottlenecks: Data is copied multiple times: Data is copied at least twice between kernel space and user space, consuming a lot of CPU resources and memory bandwidth, and increasing data transfer latency.
[0005] Context switching overhead: System calls, interrupt handling, etc. cause the CPU to switch frequently between user mode and kernel mode, consuming a lot of processing time and reducing the system's real-time responsiveness.
[0006] Lock contention and resource conflicts: In a multi-threaded environment, lock contention for shared data structures and CPU cache thrashing severely limit the parallel processing capabilities of multi-core processors.
[0007] File system overhead: In traditional file systems, operations such as metadata management and logging can become significant performance bottlenecks during high-throughput writes.
[0008] To address the aforementioned issues, a high-speed PXIe zero-copy data processing and real-time streaming disk method for domestically produced multi-core processors is proposed to solve the problems existing in the prior art, which is an urgent problem to be solved by those skilled in the art. Summary of the Invention
[0009] In view of this, the present invention provides a method for zero-copy processing and real-time streaming disk of high-speed PXIe data for domestic multi-core processors, which eliminates the performance bottlenecks in data copying, multi-core scheduling, I / O processing, etc., and fully leverages the high bandwidth of the PXIe bus (16GB / s) and the parallel processing advantages of domestic multi-core processors to achieve zero-copy processing of high-speed PXIe data and real-time efficient streaming disk of massive data, while meeting the application requirements of independent controllability, high real-time performance, and high reliability.
[0010] To achieve the above objectives, the present invention adopts the following technical solution: A method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe processors with domestic multi-core processors. The PXIe high-speed data zero-copy processing flow includes the following steps: S1. VFIO technology is used to directly map the DMA buffer to user space, realizing zero-copy data transfer between kernel space and user space; S2. Configure 2MB large page memory as a DMA cache to reduce the number of page table entries and improve the TLB address translation cache hit rate; S3. The core functions of the domestic multi-core processor are divided into AMP mode. Core 0 runs the operating system and system management tasks, cores 1-3 process control commands and configuration requests, and cores 4 to the last core form the data plane and use thread pool technology to process PXIe high-speed data. S4, inter-core communication uses a lock-free circular queue to avoid the performance bottleneck caused by mutex locks; S5. Bind the data processing thread to the corresponding PXIe hardware interrupt to the specified CPU core to achieve affinity processing between the core and the interrupt. S6. Set the highest priority of SCHED_FIFO for the critical data processing threads of the operating system, and thread the bottom half of the interrupt handling and give it a high priority. The process of real-time streaming disk for massive data includes the following steps: T1. It uses raw I / O technology to bypass the file system and directly writes block-level data to raw disk partitions / logical volumes. It opens the disk device file with the O_DIRECT flag and completes data reading and writing with 2MB memory alignment. T2. Employ io_uring asynchronous I / O, communicate with the kernel through the commit queue and completion queue of the memory-mapped circular buffer, and enable polling mode to eliminate interrupt overhead; T3. A fixed buffer pool of 2MB in size and memory alignment is pre-allocated at system startup to achieve efficient management of the data buffer; T4. Construct a lock-free producer-consumer model. The PXIe module acts as a producer, filling the DMA cache with data and triggering an interrupt to notify the data processing thread. The data processing thread acts as a consumer, retrieving an empty buffer from the buffer pool, processing the data, and writing it to the disk. The producer and consumer exchange data buffers through a lock-free circular queue.
[0011] Optionally, the above method is implemented based on the domestic Phytium multi-core processor and the domestically produced Kylin operating system with the Preempt-RT real-time patch added, and is adapted to the PXIe bus controller hardware platform composed of Phytium X100 bridge chip.
[0012] Optionally, in the above method, the data plane in S3 is core 4-31, and the DMA buffer configuration matches the 16GB / s transmission bandwidth requirement of the PXIe bus Gen3×16.
[0013] Optionally, in the above method, the lock-free circular queues in S4 and T4 are both SPSC single-producer single-consumer circular queues to ensure high throughput and low latency of data transmission.
[0014] Optionally, the thread pool technology in S3 can dynamically schedule the processing threads of the data plane cores based on the PXIe data processing workload, thereby avoiding multi-core resource contention and uneven load distribution.
[0015] Optionally, in the above method, the pre-allocated buffer pool in T3 does not perform dynamic memory allocation during system operation, eliminating the extra overhead of runtime memory allocation, and the buffer is returned to the pool for reuse after use.
[0016] The above method can optionally include the DMA buffer in S1, the 2MB large page memory in S2, and the pre-allocated buffer pool in T3, all of which are implemented based on 64GB of DDR4 memory in the memory unit.
[0017] Optionally, the PXIe bus controller hardware platform uses a 4-channel PCIe Gen3.0 communication interface as described in the technical documentation. The processor and the bridge chip are interconnected via PCIe 3.0 × 16 to ensure signal integrity for high-speed data transmission.
[0018] As can be seen from the above technical solution, compared with the prior art, the present invention provides a PXIe high-speed zero-copy data processing and real-time streaming disk method for domestic multi-core processors, which has the following beneficial effects: Achieve zero-copy data transfer throughout the process: By using VFIO technology to directly expose the DMA buffer to the user space and combining it with asynchronous I / O to write directly to the raw device, the multiple copies of data between the kernel and user space in traditional solutions are completely eliminated, significantly reducing CPU load and data transfer latency; Fully leverage multi-core parallel capabilities: Through collaborative scheduling of multi-core resources, the control plane and data plane are physically isolated to avoid mutual interference. The data plane core implements a high-efficiency, contention-free parallel data processing pipeline through lock-free queues and thread / interrupt affinity technology; Significantly improve I / O throughput and real-time performance: Adopting advanced asynchronous I / O frameworks such as io_uring and raw I / O technology reduces system call overhead and file system interference. Combined with real-time kernel patches, it ensures real-time and stable storage of massive amounts of data under high throughput. Support domestically developed and controllable technologies: Based on domestically produced processors and operating systems, meet the requirements of key areas for independent control of software and hardware. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0020] Figure 1 This is a flowchart of a PXIe high-speed zero-copy data processing and real-time streaming disk method for domestic multi-core processors disclosed in this invention; Figure 2 This is a diagram of the overall hardware architecture of the PXIe bus controller disclosed in this invention; Figure 3 This is a design drawing of the heat dissipation component disclosed in this invention; Among them, 1-steel plate, 2-PCB board, 3-copper plate, 4-power device, 5-thermal grease. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0023] See Figure 1 As shown, this invention discloses a PXIe high-speed zero-copy data processing and real-time streaming disk method for domestic multi-core processors. The PXIe high-speed data zero-copy processing flow includes the following steps: S1. VFIO technology is used to directly map the DMA buffer to user space, realizing zero-copy data transfer between kernel space and user space; S2. Configure 2MB large page memory as a DMA cache to reduce the number of page table entries and improve the TLB address translation cache hit rate; S3. The core functions of the domestic multi-core processor are divided into AMP mode. Core 0 runs the operating system and system management tasks, cores 1-3 process control commands and configuration requests, and cores 4 to the last core form the data plane and use thread pool technology to process PXIe high-speed data. S4, inter-core communication uses a lock-free circular queue to avoid the performance bottleneck caused by mutex locks; S5. Bind the data processing thread to the corresponding PXIe hardware interrupt to the specified CPU core to achieve affinity processing between the core and the interrupt. S6. Set the highest priority of SCHED_FIFO for the critical data processing threads of the operating system, and thread the bottom half of the interrupt handling and give it a high priority. The process of real-time streaming disk for massive data includes the following steps: T1. It uses raw I / O technology to bypass the file system and directly writes block-level data to raw disk partitions / logical volumes. It opens the disk device file with the O_DIRECT flag and completes data reading and writing with 2MB memory alignment. T2. Employ io_uring asynchronous I / O, communicate with the kernel through the commit queue and completion queue of the memory-mapped circular buffer, and enable polling mode to eliminate interrupt overhead; T3. A fixed buffer pool of 2MB in size and memory alignment is pre-allocated at system startup to achieve efficient management of the data buffer; T4. Construct a lock-free producer-consumer model. The PXIe module acts as a producer, filling the DMA cache with data and triggering an interrupt to notify the data processing thread. The data processing thread acts as a consumer, retrieving an empty buffer from the buffer pool, processing the data, and writing it to the disk. The producer and consumer exchange data buffers through a lock-free circular queue.
[0024] Furthermore, this invention is based on a domestically produced multi-core processor and a domestically produced operating system with the Preempt-RT real-time patch added, and is adapted to the domestically produced PXIe bus controller hardware platform.
[0025] Furthermore, in S3, the data plane is core 4-31, and the DMA buffer configuration matches the 16GB / s transmission bandwidth requirement of the PXIe bus Gen3×16.
[0026] Furthermore, the lock-free circular queues in S4 and T4 are both SPSC single-producer single-consumer circular queues, ensuring high throughput and low latency of data transmission.
[0027] Furthermore, the thread pool technology in S3 dynamically schedules the processing threads of the data plane cores based on the amount of PXIe data processing tasks, avoiding multi-core resource contention and uneven load distribution.
[0028] Furthermore, the pre-allocated buffer pool in T3 does not perform dynamic memory allocation during system operation, eliminating the additional overhead of runtime memory allocation. After the buffer is used, it is returned to the pool for reuse.
[0029] Furthermore, the DMA buffer in S1, the 2MB large page memory in S2, and the pre-allocated buffer pool in T3 are all implemented based on the 64GB DDR4 memory of the memory unit in the technical data.
[0030] Furthermore, the PXIe bus controller hardware platform uses a 4-channel PCIe Gen3.0 communication interface as described in the technical documentation. The processor and the bridge chip are interconnected via PCIe 3.0 ×16 to ensure signal integrity for high-speed data transmission.
[0031] In one specific embodiment, the method of the present invention is implemented based on a fully domestically produced PXIe bus controller hardware platform and a domestically produced Kylin operating system. All software and hardware are domestically produced, meeting the requirements of independent controllability.
[0032] I. Implementation Basis
[0033] Hardware platform: Based on the Phytium S5000C 32-core processor and Phytium X100 bridge chip, with 64GB of DDR4 memory, a 4-channel PCIe Gen3.0 PXIe bus communication interface, and the processor and bridge chip interconnected via PCIe 3.0 x16. Hardware architecture reference. Figure 2 ; Cooling system reference Figure 3 The design ensures temperature control during high-load data processing of the processor.
[0034] For details, see Figure 3 The heat dissipation structure consists of the following core components: PCB board 2, power device 4 (soldered on PCB board 2), thermal grease 5 (filled between power device 4 and copper plate 3), copper plate 3 (covering the power device 4), and steel plate 1 (covering the copper plate 3).
[0035] Software platform: Kylin OS, with Preempt-RT real-time patch added to enable kernel preemption; VFIO driver configured, large page memory support provided, and C / C++ programs for Phytium ARM64 architecture compiled to adapt to all peripheral drivers of the hardware platform.
[0036] II. Specific Implementation Steps
[0037] 1. System initialization configuration: (1) Enable large page memory support: Configure 2MB large page memory in the system startup parameters, allocate 128GB 2MB large page memory as a DMA cache pool, bind it to the DMA controller corresponding to the PXIe bus, and implement it based on 64GB DDR4 domestic memory; (2) Configure VFIO technology: Enable the operating system VFIO driver, directly map the DMA buffer to user space, disable the kernel's default cache management of the buffer, and achieve zero-copy mapping; (3) Pre-allocated buffer pool: When the system starts, 1024 memory-aligned buffers of 2MB each are pre-allocated from the 2MB large page memory to build a fixed buffer pool for data processing and buffer exchange with the streaming disk; (4) Operating system real-time configuration: Add the Preempt-RT real-time patch, enable the SCHED_FIFO scheduling policy, and reserve the highest priority for PXIe data processing critical tasks.
[0038] 2. AMP mode multi-core core binding
[0039] Perform AMP mode partitioning and core bonding on the Phytium Tengyun S5000C 32-core processor: (1) Kernel 0 is bound to the operating system kernel and system management process, and only handles basic system-level tasks (process scheduling, device management, status monitoring). (2) Core 1-3 is bound to the control plane process to handle control requests such as configuration commands, parameter queries, and status feedback for PXIe devices; (3) Core 4-31 is bound to the data plane thread pool, and 128 data processing threads are initialized. The threads are dynamically scheduled according to the PXIe data traffic and are dedicated to the reception, parsing and processing of high-speed data. (4) Bind the PXIe bus hardware interrupt and disk I / O interrupt to the designated core of core 4-31 respectively to achieve one-to-one affinity between interrupt and data processing thread.
[0040] 3. PXIe high-speed zero-copy data reception and processing
[0041] (1) The PXIe bus transmits high-speed data to the controller through a 4-channel PCIe Gen3.0 interface. The data is directly written to the DMA cache (2MB large page memory) mapped to the user space through VFIO, realizing zero-copy transmission between the kernel and the user space, with a transmission bandwidth of 16GB / s. (2) When the DMA cache is full, a hardware interrupt is triggered. After the bound core receives the interrupt, it schedules the corresponding data processing thread to parse and preprocess the data in the DMA cache. (3) Configuration parameters and status information between the control plane and the data plane are transmitted through the lockless circular queue of SPSC. The lockless mechanism eliminates the performance loss of lock contention. (4) After the data processing thread completes the data processing, it marks the data buffer as "to be written to disk" and notifies the disk thread through a lock-free circular queue.
[0042] 4. Real-time zero-copy streaming disk for massive data.
[0043] (1) The streaming disk thread acts as a consumer, obtains the buffer of "disk to be written" from the lockless circular queue of SPSC, opens the raw disk partition ( / dev / nvme0n1p1) using raw I / O technology, adds the O_DIRECT flag and configures the read and write parameters aligned to 2MB memory; (2) Construct a commit queue (SQ) and a completion queue (CQ) through the io_uring asynchronous I / O framework. Submit disk write requests to the SQ. The kernel obtains the requests through a memory-mapped circular buffer and performs the disk write operation. After the operation is completed, write the result to the CQ. (3) The disk thread uses a polling mode to read the CQ result, eliminating the overhead of interrupt handling and achieving low latency in disk writing operations; (4) After writing to the disk, the streaming thread puts the buffer back into the pre-allocated buffer pool and marks it as "free" for the PXIe data receiving stage to use in a loop. There is no dynamic memory allocation throughout the process.
[0044] 5. Real-time system monitoring and dynamic scheduling
[0045] The system management process of core 0 monitors the load of each core, DMA cache usage status, and disk I / O throughput in real time. When a sudden change in PXIe data traffic is detected, the number of data processing threads in cores 4-31 is dynamically adjusted through thread pool technology to avoid uneven load. When the number of free buffer pools is insufficient, an early warning mechanism is triggered to ensure the continuity of data transmission. At the same time, the processor temperature is monitored (based on CPLD health management in the technical documentation), and the cooling fan speed is adjusted accordingly to ensure stable system operation.
[0046] This invention presents a method for high-speed, zero-copy data processing and real-time streaming disk operation for PXIe using domestically produced multi-core processors. This method can be directly applied to fully domestically produced PXIe bus controllers built on Phytium processors, Phytium bridge chips, and the Kylin operating system. It requires no significant modifications to existing hardware platforms and can be implemented solely through software configuration and development, resulting in low development costs and strong adaptability. This method can be widely applied in fields with high requirements for high-speed PXIe data processing, such as military testing instruments, industrial control, aerospace, and rail transportation. It is particularly suitable for scenarios with stringent requirements for independent controllability, high real-time performance, and high reliability, demonstrating extremely high industrial practicality and promotional value.
[0047] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for high-speed zero-copy data processing and real-time streaming disk operation using PXIe for domestically produced multi-core processors, characterized in that: The PXIe high-speed data zero-copy processing flow includes the following steps: S1. VFIO technology is used to directly map the DMA buffer to user space, realizing zero-copy data transfer between kernel space and user space; S2. Configure 2MB large page memory as a DMA cache to reduce the number of page table entries and improve the TLB address translation cache hit rate; S3. The core functions of the domestic multi-core processor are divided into AMP mode. Core 0 runs the operating system and system management tasks, cores 1-3 process control commands and configuration requests, and cores 4 to the last core form the data plane and use thread pool technology to process PXIe high-speed data. S4, inter-core communication uses a lock-free circular queue to avoid the performance bottleneck caused by mutex locks; S5. Bind the data processing thread to the corresponding PXIe hardware interrupt to the specified CPU core to achieve affinity processing between the core and the interrupt. S6. Set the highest priority of SCHED_FIFO for the critical data processing threads of the operating system, and thread the bottom half of the interrupt handling and give it a high priority. The process of real-time streaming disk for massive data includes the following steps: T1. It uses raw I / O technology to bypass the file system and directly writes block-level data to raw disk partitions / logical volumes. It opens the disk device file with the O_DIRECT flag and completes data reading and writing with 2MB memory alignment. T2. Employ io_uring asynchronous I / O, communicate with the kernel through the commit queue and completion queue of the memory-mapped circular buffer, and enable polling mode to eliminate interrupt overhead; T3. A fixed buffer pool of 2MB in size and memory alignment is pre-allocated at system startup to achieve efficient management of the data buffer; T4. Construct a lock-free producer-consumer model. The PXIe module acts as a producer, filling the DMA cache with data and triggering an interrupt to notify the data processing thread. The data processing thread acts as a consumer, retrieving an empty buffer from the buffer pool, processing the data, and writing it to the disk. The producer and consumer exchange data buffers through a lock-free circular queue.
2. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 1, characterized in that, This invention is based on the domestic Phytium multi-core processor and the domestically produced Kylin operating system with the Preempt-RT real-time patch added, and is adapted to the PXIe bus controller hardware platform composed of Phytium X100 bridge chip.
3. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 2, characterized in that, In S3, the data plane is core 4-31, and the DMA buffer configuration matches the 16GB / s transmission bandwidth requirement of the PXIe bus Gen3×16.
4. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 3, characterized in that, Both S4 and T4 use lock-free circular queues, which are SPSC single-producer single-consumer circular queues to ensure high throughput and low latency in data transmission.
5. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 4, characterized in that, In S3, the thread pool technology dynamically schedules the processing threads of the data plane cores based on the amount of data processing tasks in PXIe, avoiding multi-core resource contention and uneven load distribution.
6. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 5, characterized in that, In T3, the pre-allocated buffer pool does not perform dynamic memory allocation during system operation, eliminating the extra overhead of runtime memory allocation. After the buffer is used, it is returned to the pool for reuse.
7. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 6, characterized in that, The DMA buffer in S1, the 2MB large page memory in S2, and the pre-allocated buffer pool in T3 are all implemented based on 64GB DDR4 memory in the memory unit.
8. The method for high-speed zero-copy data processing and real-time streaming disk operation for PXIe of domestically produced multi-core processors according to claim 7, characterized in that, The PXIe bus controller hardware platform uses a 4-channel PCIe Gen3.0 communication interface as described in the technical documentation. The processor and the bridge chip are interconnected via PCIe 3.0 × 16 to ensure signal integrity for high-speed data transmission.