A zero-copy data transmission method, device, medium and product

Abstract

The application discloses a zero-copy data transmission method, device, medium and product, and belongs to the technical field of data transmission, which comprises the following steps: an application program requests a memory buffer to a kernel module; the kernel module allocates a physical memory buffer according to the request and returns a specially designed descriptor (including a handle corresponding to an address pointer); the application program submits the descriptor to the kernel module and requests the address pointer of the memory buffer; after verifying the descriptor, the kernel module maps the physical memory buffer to the virtual address space of the application program and returns the corresponding address pointer; the application program writes data to be sent according to the address pointer, and after the writing is completed, the application program submits the descriptor to the kernel module and requests to send the written data; after verifying the descriptor, the kernel module configures and starts a DMA controller, directly reads the written data from the physical memory buffer and pushes the data to a network card for sending. The application can guarantee transmission performance, improve safety and certainty.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a zero-copy data transmission method, device, medium, and product. Background Technology

[0002] In safety-critical fields such as avionics and autonomous driving, airborne systems need to handle ever-increasing high-throughput data streams (such as sensor data and high-definition video) while simultaneously meeting stringent real-time and security requirements. Existing data transmission technologies face three main problems in addressing these challenges:

[0003] 1. Performance and Efficiency Bottlenecks: Traditional network protocol stacks (such as TCP / IP) require multiple memory copies of data packets between kernel space and user space during data transmission and reception. Each copy consumes a significant amount of CPU cycles and memory bandwidth. In high-concurrency and high-throughput scenarios, this becomes a serious bottleneck limiting the overall system performance and may affect the execution of other critical tasks.

[0004] 2. Lack of Security and Isolation: To address performance issues, existing technologies have exposed some (zero-copy) techniques, such as sharing kernel and user memory through memory mapping (mmap). However, these solutions sacrifice security, breaking the most fundamental memory isolation principle of the operating system. This allows any defect (bug) or malicious behavior in user-space applications to directly contaminate kernel or other process data, triggering systemic crashes. For avionics systems requiring high reliability and functional safety, this security risk is unacceptable, preventing such solutions from passing airworthiness certification (e.g., DO-178C).

[0005] 3. Insufficient real-time determinism: Safety-critical systems require all operations to be completed within a predictable timeframe. In traditional solutions, dynamic memory allocation (such as malloc) may introduce unpredictable latency; while when using DMA for data transfer, the disordered contention for the memory bus by multiple hardware devices can also lead to uncertain transmission delays for critical data, failing to meet the requirements of hard real-time tasks. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a zero-copy data transmission method, device, medium and product, which solves the technical problem that current data transmission technology is difficult to meet current usage requirements in terms of transmission performance, transmission security and transmission determinism.

[0007] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0008] In a first aspect, the present invention provides a zero-copy data transmission method, comprising:

[0009] In response to the application's need to send data, the application calls a function in the user-space library to request a memory buffer from the kernel module for sending data.

[0010] The kernel module allocates a physical memory buffer based on the request and returns the descriptor of the physical memory buffer;

[0011] The application calls a function in the user-space library to submit a descriptor to the kernel module and request the address pointer of the memory buffer to write data;

[0012] The kernel module verifies the descriptor. After successful verification, it maps the physical memory buffer to the application's virtual address space and returns the address pointer of the virtual address space.

[0013] The application writes the data to be sent according to the address pointer. After writing is complete, it calls a function in the user space library to submit the descriptor to the kernel module and request the data to be sent.

[0014] The kernel module verifies the descriptor, and after successful verification, configures and starts the DMA controller;

[0015] The DMA controller bypasses the CPU and directly reads the written data from the physical memory buffer and pushes it to the network card for transmission, achieving data transmission with zero CPU involvement and zero memory copy.

[0016] Optionally, in response to the user-space library receiving a function call request, it converts the request into a low-level request to the kernel module and sends it to the kernel module via the ioctl interface.

[0017] Optionally, the kernel module allocates a physical memory buffer upon request and returns a descriptor for the physical memory buffer, including:

[0018] The kernel module matches a physical memory buffer of appropriate size from the memory pool based on the buddy algorithm according to the request, and looks up a free entry in the built-in descriptor table to generate a corresponding descriptor; the entry contains the necessary information of the physical memory buffer, and the descriptor is returned to the application via the user space library.

[0019] Optionally, the descriptor includes a handle ID, a usage count, and a checksum;

[0020] The handle ID is the entry index of the descriptor;

[0021] The usage count is the number of times the descriptor is released by the application. When the descriptor is released by the application, the usage count of the corresponding descriptor recorded in the descriptor table is incremented by one. When the application uses the descriptor to access the kernel module again, the usage count in the descriptor is matched with the usage count in the corresponding descriptor in the descriptor table. If the match fails, the kernel module refuses access.

[0022] The checksum is used to prevent the descriptor from being maliciously tampered with.

[0023] Optionally, in response to the completion of the network card transmission, the DMA controller generates a DMA interrupt notification to the kernel module to indicate that the transmission is complete. The kernel module sets the entry corresponding to the descriptor in the descriptor table to idle and generates a status notification indicating successful transmission, which is then returned to the application via the user space library.

[0024] In a second aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0025] The storage medium is used to store instructions;

[0026] The processor is configured to operate according to the instructions to perform the steps according to the method described above.

[0027] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0028] Fourthly, the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0029] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0030] This invention provides a zero-copy data transmission method, device, medium, and product that establishes a secure firewall between the application and the zero-copy buffer in the kernel space through kernel modules and descriptor mechanisms. It achieves efficient zero-copy data transfer from memory to the network card via DMA while completely avoiding security isolation issues caused by direct memory mapping. This makes high-speed performance secure and usable.

[0031] By eliminating memory copies between kernel space and user space and allowing the DMA controller to handle core data transfer, the CPU is freed from these inefficient and repetitive tasks, significantly reducing CPU utilization. The data path is greatly simplified, effectively shortening end-to-end latency and significantly improving system data processing throughput.

[0032] A statically pre-allocated memory pool is used for buffer management, and its allocation and reclamation operations are completed within a predictable time, avoiding uncertain delays.

[0033] In summary, the present invention can improve security and determinism while ensuring transmission performance. Attached Figure Description

[0034] Figure 1 This is a flowchart of the zero-copy data transmission method provided in an embodiment of the present invention;

[0035] Figure 2 This is a timing diagram of the zero-copy data transmission method provided in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0037] Example 1:

[0038] like Figure 1 As shown, the present invention provides a zero-copy data transmission method, comprising the following steps:

[0039] Step S1: In response to the application needing to send data, the application calls a function in the user-space library to request a memory buffer from the kernel module for sending data.

[0040] like Figure 2 As mentioned above, the application (App) needs to send data, so it calls the zc_alloc_buffer() function provided by the user space library (Lib) in order to obtain a memory buffer of a specified size.

[0041] Step S2: The kernel module allocates a physical memory buffer according to the request and returns the descriptor of the physical memory buffer.

[0042] When the user-space library receives the call, it translates it into a low-level request to the kernel module. The purpose of this low-level request is to instruct the kernel to allocate a buffer for zero-copy.

[0043] The kernel module matches a suitably sized physical memory buffer from the buddy system-based memory pool upon request, and then searches for a free entry in the built-in descriptor table to generate a corresponding descriptor. This entry contains the necessary information about the physical memory buffer, and the descriptor is returned to the application via the user-space library. The descriptor serves as the unique "key" to the memory buffer obtained by the application.

[0044] Specifically, in this embodiment, the descriptor includes a handle ID, a usage count, and a checksum.

[0045] The handle ID is the entry index of the descriptor; the handle ID is the core of the descriptor, serving as a direct index to access a large descriptor table in the kernel. Each entry in the descriptor table contains all the necessary information, including a pointer to the actual physical memory buffer, the buffer size, status (free / in use), and the owner process ID. Using a direct index avoids any lookups or hash calculations. After obtaining the ID, the kernel only needs one array access (table[ID]) to locate all the information, achieving an O(1) lookup efficiency.

[0046] The usage count represents the number of times a descriptor has been freed by an application. Usage counting is crucial for security, preventing dangling pointer or use-after-free vulnerabilities. Mechanism: In the kernel's descriptor table, each entry, in addition to a buffer pointer, also has a corresponding kernel usage count. When a descriptor is created and returned to the user, it carries the current usage count. When the application frees the descriptor (the timing of descriptor freeing occurs when the data to be sent has been completed and the application is no longer in the memory buffer to be allocated, at which point the application frees the address pointer), the kernel's corresponding entry's kernel usage count is incremented by 1. Verification: When an application uses the descriptor again (which may be expired or freed) to access the kernel, the kernel compares the usage count carried in the descriptor with the current usage count in the kernel table. If they do not match, access is immediately denied. This ensures that expired descriptors become invalid immediately.

[0047] The checksum is used to prevent the descriptor from being maliciously tampered with. The checksum can be the result of a simple XOR operation between the handle ID and the usage count, or other fast checksum algorithms. The kernel recalculates and compares the checksum during each verification.

[0048] Depending on actual needs, the descriptor can be set as an array of handle ID (32-bit), usage count (16-bit), and checksum (16-bit).

[0049] Step S3: The application calls a function in the user-space library to submit a descriptor to the kernel module and request the address pointer of the memory buffer to be used for writing data.

[0050] Although the application has a memory descriptor, it cannot directly access memory. It needs to call the library function `zc_get_pointer(desc_A)` to obtain a pointer to a usable memory address. The user-space library then makes a request to the kernel again through a low-level request and submits the memory descriptor.

[0051] Step S4: The kernel module verifies the descriptor. After successful verification, it maps the physical memory buffer to the application's virtual address space and returns the address pointer of the virtual address space.

[0052] Upon receiving the request, the kernel module performs rigorous security checks: verifying the existence of the descriptor's handle ID, checking if the usage count matches the kernel's record, and confirming the checksum is correct. If verification passes, the kernel performs just-in-time memory mapping. It temporarily and on demand maps this physical memory buffer into the application's virtual address space, then returns the address pointer to the user-space library, which in turn returns it to the application.

[0053] Step S5: The application writes the data to be sent according to the address pointer. After writing is completed, it calls the function in the user space library to submit the descriptor to the kernel module and request to send the written data.

[0054] After obtaining the address pointer, the application can copy the data to be sent directly into this buffer as if it were ordinary memory. This process occurs entirely in user space, making it highly efficient and fast. Once the data is written, the application calls the `zc_send(desc_A)` function to notify the kernel that sending can begin.

[0055] Step S6: Verify the kernel module descriptor. After successful verification, configure and start the DMA controller.

[0056] The kernel module performs a final, equally rigorous security check. Once the check passes, the kernel module configures and starts the DMA controller. It tells the DMA engine: "Transfer the data from the source address (the physical address of the buffer) to the destination address (the network card's transmit queue)."

[0057] Step S7: The DMA controller bypasses the CPU and directly reads the written data from the physical memory buffer and pushes it to the network card for transmission, realizing data transmission with zero CPU involvement and zero memory copy.

[0058] In response to the network card completing the transmission, the DMA controller generates a DMA interrupt to notify the kernel module that the transmission is complete. The kernel module sets the entry corresponding to the descriptor in the descriptor table to idle and generates a successful transmission status notification, which is then returned to the application via the user-space library. This completes a single zero-copy transmission process.

[0059] The kernel module mentioned in this embodiment is the core logic running in kernel space. It is responsible for: 1. Memory pool management: Implementing a static pre-allocated memory pool based on the buddy algorithm. 2. Descriptor table maintenance: Managing a large array of descriptors, handling descriptor allocation, release, and usage count updates. 3. Security executor: Implementing the "high-speed kernel proxy interface" described below, verifying every request from user space (checksum, usage count, process ID, etc.). 4. Hardware interaction: After successful verification, configuring the DMA controller, interacting with the network card, and initiating actual data transmission.

[0060] The user space library mentioned in this embodiment is designed to be easy for upper-layer application developers to use. We encapsulate all the complexity in one library and provide multiple APIs.

[0061] For example: handle = zero_copy_init(): Initializes the library and establishes a connection with the kernel module.

[0062] `desc = zc_alloc_buffer(size)`: Allocates a buffer for sending data and returns a descriptor.

[0063] void* ptr = zc_get_pointer(desc): (Key step) Get a usable memory pointer based on the descriptor.

[0064] int ret = zc_commit_write(desc, length): Notifies the kernel that the data has been written.

[0065] int ret = zc_send(desc): Sends data.

[0066] `desc = zc_receive(timeout)`: Receives data and returns a descriptor containing the new data.

[0067] void zc_free_buffer(desc): Frees a descriptor and its corresponding buffer.

[0068] In this embodiment, in response to the user-space library receiving a function call request, it is converted into a low-level request to the kernel module and sent to the kernel module via the ioctl interface. Standard system calls (syscalls) have significant overhead. For performance reasons, a dedicated, lightweight ioctl interface is designed, or on supported systems (such as Linux), more modern asynchronous interfaces such as io_uring are utilized. The ioctl interface enables: 1. Batch processing: The interface should support submitting multiple commands at once (e.g., requesting 3 buffers and sending 5 buffers), reducing the number of switches between user mode and kernel mode. 2. Minimizing context: The interface design should ensure that the kernel only accesses a minimal set of data during processing, avoiding unnecessary memory accesses and computations, thus streamlining the entire processing flow.

[0069] The advantages of this patented technical solution are as follows:

[0070] 1. Combining performance and efficiency

[0071] Compared with traditional TCP / IP data paths, the present invention has significant performance advantages.

[0072] • Significantly reduced CPU load: By eliminating data copying between kernel space and user space and utilizing DMA for autonomous hardware transfers, the CPU is freed from heavy data handling tasks. In avionics systems, this means valuable CPU resources can be used for more critical tasks such as flight control, navigation calculations, or sensor data fusion.

[0073] • Significantly reduced data latency: The path of data from the application's buffer to the network card is drastically shortened and simplified. Information no longer needs to "travel long distances" between multiple memory regions, thus significantly reducing end-to-end data transmission latency, which is crucial for real-time applications requiring rapid response, such as alarm message delivery and real-time video surveillance.

[0074] • Significantly increased data throughput: Due to reduced latency and the elimination of CPU bottlenecks, the entire data path becomes smoother, and the amount of data that can be transmitted per unit time (i.e., throughput) is significantly increased. This is crucial for applications processing high-bandwidth sensor data streams such as radar and photoelectric sensors.

[0075] 2. Provides strong security and isolation with "authentication capability".

[0076] This is the core advantage of this patent. It solves the fatal flaw of standard zero-copy technologies (such as direct mmap), enabling its safe application in the field of avionics.

[0077] • Innovative proxy isolation mechanism: This invention does not directly expose kernel memory to applications. Instead, it proxies it through a kernel module that acts as a mandatory access control point. Any application access to the shared buffer must go through the descriptor and undergo strict review by the kernel module. This fundamentally eliminates the risk of applications accessing memory out of bounds or tampering with other applications or kernel data, providing robust memory isolation.

[0078] • Inherent security vulnerability protection: Our "use counting" mechanism is a security feature built into the architecture. It can accurately identify and deny applications from using released, expired descriptors, effectively immunizing against high-risk memory safety vulnerabilities such as "use-and-reuse".

[0079] • Explicit support for software airworthiness certification: Since the safety and isolation logic is primarily implemented in software (kernel agent modules), its rules and behaviors are explicit, traceable, testable, and verifiable. Compared to relying on complex hardware configurations (such as IOMMUs) that may exhibit "black box" behavior, this software-defined safety approach is easier to formally verify and review, thus making it easier to meet the requirements of aviation software airworthiness standards such as DO-178C.

[0080] 3. Ensure a high degree of real-time performance and determinism.

[0081] For avionics systems, especially flight-critical systems, "predictability" is more important than "speed." This invention ensures system determinism through design.

[0082] • Predictable memory management: Employing a static pre-allocated memory pool based on the "buddy system" ensures that memory buffer allocation and deallocation are completed within bounded time. This avoids the unpredictable latency that can occur with standard dynamic memory allocation (such as malloc), guaranteeing the responsiveness of real-time tasks.

[0083] • Controllable data transmission: By setting priority or quality of service (QoS) policies for DMA transmission of different data streams, it can be ensured that the highest priority data streams such as navigation and flight control can always obtain bus resources first, ensuring the determinism of their data transmission and preventing them from being blocked by non-critical tasks (such as logging).

[0084] Example 2:

[0085] Based on the zero-copy data transmission method provided in Embodiment 1, this embodiment of the invention provides an electronic device, including a processor and a storage medium;

[0086] The storage medium is used to store instructions;

[0087] The processor is configured to operate according to the instructions to perform the steps according to the method described above.

[0088] Example 3:

[0089] Based on the zero-copy data transmission method provided in Embodiment 1, this embodiment of the invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above method.

[0090] Example 4:

[0091] Based on the zero-copy data transmission method provided in Embodiment 1, this embodiment of the invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described method.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A zero-copy data transmission method, characterized in that, include: In response to the application's need to send data, the application calls a function in the user-space library to request a memory buffer from the kernel module for sending data. The kernel module allocates a physical memory buffer based on the request and returns a descriptor for the physical memory buffer; the descriptor includes a handle ID, a usage count, and a checksum. The application calls a function in the user-space library to submit a descriptor to the kernel module and request the address pointer of the memory buffer to write data; The kernel module verifies the descriptor. After successful verification, it maps the physical memory buffer to the application's virtual address space and returns the address pointer of the virtual address space. The application writes the data to be sent according to the address pointer. After writing is complete, it calls a function in the user space library to submit the descriptor to the kernel module and request the data to be sent. The kernel module verifies the descriptor, and after successful verification, configures and starts the DMA controller; The DMA controller bypasses the CPU and directly reads the written data from the physical memory buffer and pushes it to the network card for transmission, achieving data transmission with zero CPU involvement and zero memory copy.

2. The zero-copy data transmission method according to claim 1, characterized in that, In response to a user-space library receiving a function call request, it converts the request into a low-level request to the kernel module and sends it to the kernel module via the ioctl interface.

3. The zero-copy data transmission method according to claim 1, characterized in that, The kernel module allocates a physical memory buffer upon request and returns the descriptor of the physical memory buffer, including: The kernel module matches a physical memory buffer of appropriate size from the memory pool based on the buddy algorithm according to the request, and looks up a free entry in the built-in descriptor table to generate a corresponding descriptor; the entry contains the necessary information of the physical memory buffer, and the descriptor is returned to the application via the user space library.

4. The zero-copy data transmission method according to claim 1, characterized in that, The handle ID is the entry index of the descriptor; The usage count is the number of times the descriptor is released by the application. When the descriptor is released by the application, the usage count of the corresponding descriptor recorded in the descriptor table is incremented by one. When the application uses the descriptor to access the kernel module again, the usage count in the descriptor is matched with the usage count in the corresponding descriptor in the descriptor table. If the match fails, the kernel module refuses access. The checksum is used to prevent the descriptor from being maliciously tampered with.

5. The zero-copy data transmission method according to claim 1, characterized in that, In response to the completion of the network card transmission, the DMA controller generates a DMA interrupt notification to the kernel module to indicate that the transmission is complete. The kernel module sets the entry corresponding to the descriptor in the descriptor table to idle and generates a status notification indicating successful transmission, which is then returned to the application via the user space library.

6. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-5.

Images