Zero-copy data transmission method and device, medium and product

Abstract

The invention discloses a zero-copy data transmission method and device, a medium and a product, and belongs to the technical field of data transmission, and the method comprises the steps that an application program requests a memory buffer area from a kernel module; the kernel module allocates a physical memory buffer area according to the request, and returns a specially designed descriptor (including a handle corresponding to an address pointer); the application program submits a descriptor to the kernel module and requests an address pointer of the memory buffer area; after the kernel module verifies the descriptor, the physical memory buffer area is mapped to a virtual address space of the application program, and a corresponding address pointer is returned; the application program writes data needing to be sent according to the address pointer, and after writing is completed, the descriptor is submitted to the kernel module and the written data is requested to be sent; after the kernel module verifies the descriptor, a DMA controller is configured and started, and the written data is directly read from the physical memory buffer area and pushed to a network card to be sent. According to the invention, the security and certainty can be improved while the transmission performance is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a zero-copy data transmission method, device, medium and product. BACKGROUND

[0002] In safety-critical fields such as avionics and unmanned driving, airborne systems need to process increasingly growing high-traffic data streams (such as sensor data, high-definition video, etc.), while meeting stringent real-time and safety requirements. Existing data transmission technologies mainly have the following three problems when dealing with these challenges: 1. Performance and efficiency bottleneck: In the data transmission process of traditional network protocol stack (such as TCP / IP), data packets need to be copied multiple times between kernel space and user space. Each copy consumes a large number of CPU cycles and memory bandwidth, which becomes a serious bottleneck limiting the overall performance of the system in high-concurrency and high-throughput scenarios, and may affect the execution of other critical tasks.

[0003] 2. Lack of security and isolation: To solve the performance problem, existing technologies disclose some (Zero-Copy) technologies, such as sharing kernel and user memory through memory mapping (mmap) and the like. However, such solutions sacrifice security at the cost of breaking the basic memory isolation principle of the operating system, so that any defect (Bug) or malicious behavior of the user-mode application program can directly pollute the data of the kernel or other processes, causing system collapse. For avionics systems requiring high reliability and functional safety, this security risk is unacceptable, which also prevents such solutions from passing the airworthiness certification (such as DO-178C).

[0004] 3. Lack of real-time determinism: Safety-critical systems require all operations to be completed within a predictable time. In traditional technical solutions, dynamic memory allocation (such as malloc) may introduce unpredictable latency; and when using DMA for data transmission, the unordered contention of multiple hardware devices for the memory bus also causes the transmission delay of critical data to become uncertain, which cannot meet the needs of hard real-time tasks. SUMMARY

[0005] The present application aims to overcome the deficiencies in the prior art and provide a zero-copy data transmission method, device, medium and product, which solves the technical problem that current data transmission technologies are difficult to meet current use requirements in terms of transmission performance, transmission security and transmission determinism.

[0006] To achieve the above-mentioned purpose, the present application is implemented by using the following technical solutions: In a first aspect, the present application provides a zero-copy data transmission method, comprising: In response to the application needing to send data, the application calls a function in the user space library to request a memory buffer for sending data from the kernel module; The kernel module allocates a physical memory buffer according to the request and returns a descriptor of the physical memory buffer; The application calls a function in the user space library to submit the descriptor to the kernel module and request an address pointer of the memory buffer for writing data; The kernel module verifies the descriptor, and after verification, maps the physical memory buffer to the virtual address space of the application and returns an address pointer of the virtual address space; The application writes the data to be sent according to the address pointer, and after writing is completed, calls a function in the user space library to submit the descriptor to the kernel module and request sending the written data; The kernel module verifies the descriptor, and after verification, configures and starts the DMA controller; The DMA controller directly reads the written data from the physical memory buffer and pushes it to the network card for sending, realizing CPU zero participation and memory zero copy data transmission.

[0007] Optionally, in response to the user space library receiving a call function request, the user space library converts the call function request into a bottom request to the kernel module and sends the bottom request to the kernel module through an ioctl interface.

[0008] Optionally, the kernel module allocates a physical memory buffer according to the request and returns a descriptor of the physical memory buffer, including: The kernel module matches a physical memory buffer with a proper size from a memory pool based on a buddy algorithm according to the request, and finds a free entry in a built-in descriptor table to generate a corresponding descriptor; the entry contains necessary information of the physical memory buffer, and the descriptor is returned to the application through the user space library.

[0009] Optionally, the descriptor includes a handle ID, a use count, and a check code; The handle ID is an entry index of the descriptor; The use count is a number of times the descriptor is released by the application; when the descriptor is released by the application, the use count of the corresponding descriptor recorded in the descriptor table is increased by one; when the application uses the descriptor to access the kernel module again, the use count in the descriptor is matched with the use count in the corresponding descriptor in the descriptor table; if the matching fails, the kernel module refuses to access; The check code is used to prevent the descriptor from being maliciously tampered with.

[0010] Optionally, in response to the network card sending being completed, the DMA controller generates a DMA interrupt to notify the kernel module of the sending being completed, the kernel module sets the entry corresponding to the corresponding descriptor in the descriptor table to be idle, and generates a sending success state notification, which is returned to the application program through the user space library.

[0011] In a second aspect, the present application provides an electronic device, comprising a processor and a storage medium; The storage medium is configured to store instructions. The processor is configured to operate according to the instructions to perform the steps of the above method.

[0012] In a third aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the above method.

[0013] In a fourth aspect, the present application provides a computer program product, which comprises a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the above method.

[0014] Compared with the prior art, the present application has the following beneficial effects: The zero-copy data transmission method, device, medium and product provided by the present application establish a security firewall between the application program and the zero-copy buffer in the kernel space through the kernel module and the descriptor mechanism. The zero-copy efficient transmission of data from the memory to the network card through the DMA is realized, and the security isolation problem caused by the direct memory mapping is completely avoided. It makes the extremely fast performance safe and available.

[0015] By eliminating the memory copy between the kernel space and the user space, and letting the DMA controller be responsible for the core data transfer work, the CPU is freed from these inefficient and repetitive tasks, and the CPU occupancy is significantly reduced. The data path is greatly simplified, the end-to-end delay is effectively shortened, and the data processing throughput of the system is greatly improved.

[0016] The static pre-allocated memory pool is used for buffer management, and the allocation and recovery operations are completed within a predictable time, avoiding uncertain delay.

[0017] In summary, the present application can improve the security and certainty while ensuring the transmission performance. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the zero-copy data transmission method provided by the embodiment of the present application; Figure 2 is a timing diagram of the zero-copy data transmission method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0019] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0020] Example 1

[0021] As shown in the figure, the application provides a zero-copy data transmission method, comprising the following steps: Figure 1 Step S1, in response to the application needing to send data, the application calls a function in a user space library to request a memory buffer for sending data from a kernel module.

[0022] As shown in the figure, the application (App) needs to send data, and then calls the zc_alloc_buffer() function provided by the user space library (Lib) to obtain a memory buffer with a specified size. Figure 2

[0023] Step S2, the kernel module allocates a physical memory buffer according to the request and returns a descriptor of the physical memory buffer.

[0024] After receiving the call, the user space library converts it into a bottom request to the kernel module (Kernel). The purpose of the bottom request is to let the kernel allocate a buffer for zero-copy.

[0025] The kernel module matches a physical memory buffer with a proper size from a memory pool based on the buddy algorithm according to the request, and finds a free entry in the built-in descriptor table to generate a corresponding descriptor; the entry contains necessary information of the physical memory buffer, and the descriptor is returned to the application by the user space library. The descriptor is the only "key" for the application to operate the obtained memory buffer.

[0026] Specifically in this embodiment, the descriptor includes a handle ID, a usage count and a check code.

[0027] The handle ID is the entry index of the descriptor; the handle ID is the core of the descriptor, which is used as a direct index to access a large descriptor table in the kernel. Each entry of the descriptor table contains a pointer to the real physical memory buffer, the buffer size, the state (free / in use), the owner process ID and all necessary information. Using the direct index can avoid any lookup or hash calculation; after obtaining the ID, the kernel only needs to perform one array access (table[ID]) to locate all information, achieving O(1) lookup efficiency.

[0028] ​​The use count is the number of times the descriptor is released by the application; the use count is critical to security, to prevent "dangling pointers" or "use-after-free" vulnerabilities. Working mechanism: in the kernel's descriptor table, each entry has a corresponding "kernel use count value" in addition to the buffer pointer. When a descriptor is created and returned to the user, it carries the current use count value. When the application releases the descriptor (the timing of the application releasing the descriptor is when the data to be sent has been sent, and the application no longer needs the allocated memory buffer, at which time the application releases the address pointer), the "kernel use count value" of the corresponding entry in the kernel is incremented by 1. Verification: when the application accesses the kernel again using the descriptor (which may be expired and released), the kernel compares the use count carried in the descriptor with the current use count in the kernel table. If they do not match, access is immediately denied. This ensures that expired descriptors are invalidated immediately.

[0029] The check code is used to prevent the descriptor from being maliciously tampered with. The check code can be the result of a simple XOR of the handle ID and the use count or other fast check algorithms. The kernel recalculates and compares the check code at each verification.

[0030] According to actual needs, the descriptor can be set as an array of handle IDs (32 bits), use counts (16 bits), and check codes (16 bits).

[0031] Step S3, the application calls a function in the user space library, submits the descriptor to the kernel module, and requests an address pointer of a memory buffer for writing data.

[0032] Although the application has the descriptor, it cannot directly access the memory. It needs to call the library function zc_get_pointer(desc_A) to obtain an available memory address pointer. The user space library again requests the kernel through the underlying request and submits the descriptor.

[0033] Step S4, the kernel module verifies the descriptor, and after verification, maps the physical memory buffer to the virtual address space of the application and returns the address pointer of the virtual address space.

[0034] After receiving the request, the kernel module performs strict security checks: verifies whether the handle ID of the descriptor exists, whether the use count matches the kernel record, and whether the check code is correct. After verification, the kernel performs Just-in-Time memory mapping. It temporarily and on-demand maps this physical memory buffer to the virtual address space of the application, and then returns the address pointer to the user space library, which is returned to the application through the user space library.

[0035] Step S5, the application writes the data to be sent according to the address pointer, and after the writing is completed, calls a function in the user space library to submit the descriptor to the kernel module and request sending the written data.

[0036] After obtaining the address pointer, the application can directly copy the data to be sent to the buffer just like operating ordinary memory. This process is completely in the user space and is efficient and fast. After the data writing is completed, the application calls the zc_send(desc_A) function to inform the kernel that the sending can be started.

[0037] Step S6, the kernel module verifies the descriptor, and after the verification is passed, configures and starts the DMA controller.

[0038] The kernel module performs the last and also strict security verification. After the verification is passed, the kernel module configures and starts the DMA controller. It tells the DMA engine: "Please transfer the data of the source address (the physical address of the buffer) to the target address (the sending queue of the network card)".

[0039] Step S7, the DMA controller directly reads the written data from the physical memory buffer and pushes it to the network card for sending to realize the CPU zero-participation and memory zero-copy data transmission.

[0040] In response to the completion of the sending of the network card, the DMA controller generates a DMA interrupt to notify the kernel module of the completion of the sending, the kernel module sets the corresponding entry of the corresponding descriptor in the descriptor table to be idle, and generates a status notification of the success of the sending, which is returned to the application through the user space library. Thus, the complete zero-copy sending process is ended.

[0041] The kernel module mentioned in the embodiment is the core logic running in the kernel space. It is responsible for: 1. Memory pool management: realizing the static pre-allocated memory pool based on the buddy algorithm. 2. Descriptor table maintenance: managing a huge descriptor array and processing the allocation, release and usage count updating of the descriptor. 3. Security executor: realizing the "extremely fast kernel proxy interface" described below and verifying each request from the user space (checking code, usage count, process ID, etc.). 4. Hardware interaction: after the verification is passed, being responsible for configuring the DMA controller, interacting with the network card and initiating the real data transmission.

[0042] The user space library mentioned in the embodiment is to make the upper application developers easy to use. We encapsulate all the complexities in a library and provide multiple APIs, For example: handle = zero_copy_init(): initializing the library and establishing a connection with the kernel module.

[0043] desc = zc_alloc_buffer(size): Apply for a buffer for sending data, return a descriptor.

[0044] void* ptr = zc_get_pointer(desc): (Key step) Get a usable memory pointer according to the descriptor.

[0045] int ret = zc_commit_write(desc, length): Notify the kernel that the data has been written.

[0046] int ret = zc_send(desc): Send data.

[0047] desc = zc_receive(timeout): Receive data, return a descriptor containing new data.

[0048] void zc_free_buffer(desc): Free a descriptor and its corresponding buffer.

[0049] In this embodiment, in response to the user space library receiving a call function request, it is converted into a bottom request for the kernel module and sent to the kernel module through the ioctl interface. The standard system call (syscall) has a large overhead. In order to improve performance, a dedicated, lightweight ioctl interface is designed, or a more modern asynchronous interface such as io_uring is used on supported systems (such as Linux). The ioctl interface can achieve: 1. Batch processing: the interface should support submitting multiple commands at a time (for example, applying 3 buffers and sending 5 buffers), reducing the number of user and kernel state switches. 2. Minimize context: the design of the interface should ensure that the kernel only needs to access the minimum set of data when processing, avoiding unnecessary memory access and calculation, and simplifying the entire processing flow.

[0050] The advantages of the patent technical solution are as follows: 1. Performance and efficiency Compared with the traditional TCP / IP data path, the performance advantage of the present application is obvious.

[0051] • Significantly reduce CPU load: by eliminating data copying between kernel space and user space, and using DMA for hardware autonomous transmission, the CPU is freed from the heavy data transfer work. In avionics systems, this means that valuable CPU resources can be used for more critical flight control, navigation calculation or sensor data fusion tasks.

[0052] • Extremely short data latency: The path of data from the application's buffer to the network card is extremely shortened and simplified. Information does not need to "travel long distances" between multiple memory regions, significantly reducing the end-to-end data transmission delay, which is crucial for real-time applications that require fast response (such as alarm information transmission, real-time video monitoring).

[0053] • Substantial increase in data throughput: Due to the reduction of latency and the elimination of CPU bottlenecks, the entire data path becomes more smooth, and the amount of data that can be transmitted per unit of time (i.e., throughput) is greatly improved. This is of decisive significance for application scenarios that handle radar, photoelectric, and other high-bandwidth sensor data streams.

[0054] 2. Provide "authenticatable" strong security and isolation This is the core advantage of this patent. It solves the fatal flaw of standard zero-copy technology (such as direct mmap), making it safe to apply in the avionics field.

[0055] • Innovative proxy isolation mechanism: Instead of directly exposing kernel memory to applications, the invention uses a kernel module as a mandatory access control point. Any application access to the shared buffer must go through the descriptor and be strictly reviewed by the kernel module. This fundamentally eliminates the risk of application boundary access, tampering with other application or kernel data, and provides strong memory isolation.

[0056] • Internal security vulnerability protection: The "usage count" mechanism we designed is a security feature built into the architecture. It can accurately identify and reject applications that use expired descriptors, effectively immunizing high-risk memory security vulnerabilities such as "reuse after use."

[0057] • Explicitly support software airworthiness certification: Since the security and isolation logic is mainly implemented by software (kernel proxy module), its rules and behaviors are explicit, traceable, testable, and verifiable. Compared to relying on complex hardware (such as IOMMU) configurations that may have "black box" behaviors, this software-defined security solution is easier to formalize and code review, making it easier to meet the requirements of DO-178C and other aviation software airworthiness standards.

[0058] 3. Ensure high real-time and determinism For avionics systems, especially flight-critical systems, "predictable" is more important than "fast." The invention guarantees the determinism of the system through design.

[0059] • Predictable memory management: A static pre-allocated memory pool based on "buddy algorithm" is adopted to ensure that the memory buffer application and release are completed within a bounded time. This avoids unpredictable delays that may be caused by standard dynamic memory allocation (such as malloc), ensuring the responsiveness of real-time tasks.

[0060] • Controllable data transmission: By setting priority or quality of service (QoS) strategy for DMA transmission of different data streams, it can be ensured that the highest priority data streams such as navigation and flight control can always have priority to obtain bus resources, ensuring the determinacy of their data transmission and not being blocked by non-critical tasks (such as log recording).

[0061] Embodiment two:

[0062] Based on the zero-copy data transmission method provided in embodiment one, the embodiment of the present application provides an electronic device, comprising a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the above method.

[0063] Embodiment three:

[0064] Based on the zero-copy data transmission method provided in embodiment one, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the above method.

[0065] Embodiment four:

[0066] Based on the zero-copy data transmission method provided in embodiment one, the embodiment of the present application provides a computer program product, comprising a computer program / instruction, which is executed by a processor to realize the steps of the above method.

[0067] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0068] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0069] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0070] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0071] The above only is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the technical field, without departing from the technical principles of the present application, can also make a number of improvements and variations, these improvements and variations should also be considered as the protection scope of the present application.

Claims

1. A zero-copy data transmission method, characterized in that: include: In response to the application needing to send data, the application calls a function in the user space library to request a memory buffer for sending data from the kernel module; The kernel module allocates a physical memory buffer according to the request and returns a descriptor of the physical memory buffer; the descriptor includes a handle ID, a usage count and a check code; The application calls a function in the user space library, submits a descriptor to the kernel module and requests the address pointer of the memory buffer for writing data; The kernel module verifies the descriptor, and after passing the verification, maps the physical memory buffer to the virtual address space of the application and returns the address pointer of the virtual address space; The application writes the data to be sent according to the address pointer. After writing is completed, it calls the function in the user space library, submits the descriptor to the kernel module and requests to send the written data; The kernel module verifies the descriptor and, if verified, configures and starts the DMA controller. The DMA controller bypasses the CPU to directly read the written data from the physical memory buffer and pushes it to the network card for transmission, thereby realizing data transmission with zero CPU involvement and zero memory copy.

2. The zero-copy data transmission method according to claim 1, wherein: In response to the user space library receiving the function call request, it converts it into a low-level request to the kernel module and sends it to the kernel module through the ioctl interface.

3. The zero-copy data transmission method according to claim 1, wherein: The kernel module allocates a physical memory buffer according to the request, and returns a descriptor of the physical memory buffer including: The kernel module matches a physical memory buffer of appropriate size from a memory pool based on a buddy algorithm according to the request, and searches for a free entry in a built-in descriptor table to generate a corresponding descriptor; the entry contains the necessary information of the physical memory buffer, and the descriptor is returned to the application via the user space library.

4. The zero-copy data transmission method according to claim 1, wherein: The handle ID is the entry index of the descriptor; The usage count is the number of times the descriptor is released by the application; when the descriptor is released by the application, the usage count of the corresponding descriptor recorded in the descriptor table is increased by one, and when the application uses the descriptor to access the kernel module again, the usage count in the descriptor is matched with the usage count in the corresponding descriptor in the descriptor table. If the match fails, the kernel module denies access; The check code is used to prevent the descriptor from being maliciously tampered with.

5. The zero-copy data transmission method according to claim 1, wherein: In response to the completion of the network card transmission, the DMA controller generates a DMA interrupt to notify the kernel module of the completion of the transmission. The kernel module sets the entry corresponding to the corresponding descriptor in the descriptor table to idle, and generates a status notification of successful transmission, which is returned to the application through the user space library.

6. An electronic device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images