Embedded network protocol stack construction method and device based on Rust

Abstract

The invention belongs to the technical field of network protocol stacks, and particularly relates to a method and a device for constructing an embedded network protocol stack based on Rust, common memory errors can be avoided by using the network protocol stack compiled by the Rust, a memory management mechanism of the Rust comprises an automatic recovery mechanism and an ownership system, the memory which is not used any more can be automatically released, and the reliability of the network protocol stack is improved. The common memory security problems of memory leakage, buffer overflow, repeated release and the like of the existing network protocol stack in an embedded system are solved, and the stability and the performance of the system are improved. The performance of the network protocol stack is optimized while the security of the memory is ensured, and the protocol stack is suitable for various embedded devices, especially in environments requiring high-reliability and efficient memory management. According to the network protocol stack, the memory security problem existing in a common network protocol stack is solved, and meanwhile, the effect which cannot be achieved by rewriting the network protocol stack only by simply using the Rust is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network protocol stacks, and in particular relates to a method and device for constructing an embedded network protocol stack based on Rust. Background Art

[0002] With the development of the Internet of Things (IoT) and embedded systems, network communication is becoming increasingly important in embedded devices. These devices are widely used in smart homes, industrial automation, medical equipment and other fields, and require efficient, secure and reliable network protocol stacks to achieve communication between devices. However, traditional network protocol stacks are mainly based on C language, which has problems such as complex memory management and prone to security vulnerabilities such as buffer overflow. Due to limited resources in embedded devices, these problems can lead to system crashes, data leaks and even malicious control of devices.

[0003] At present, network protocol stacks based on C language, such as LWIP and uIP, have been widely used in embedded systems. Although these protocol stacks have been optimized and maintained for a long time, their performance and functions are relatively mature, but due to the memory management characteristics of C language, it is still difficult to avoid problems such as memory leaks, pointer errors and buffer overflows. In addition, C language programming is more complex, and the development and maintenance costs are high, which is not conducive to rapid iteration and innovation. Summary of the invention

[0004] Based on the above problems, this application uses Rust's unique memory safety features and efficient concurrency support to overcome the inherent memory management and concurrency processing difficulties of the C language network protocol stack. Its technical solution is:

[0005] A method for building an embedded network protocol stack based on Rust, including a batch processing module, an object pool module, a data transmission module, and a bottom-level interface module;

[0006] Object pool module: pre-allocates N memory blocks, uses Rust's lock-free queue to manage the object pool, and uses ownership and lifecycle mechanisms to automatically allocate and release memory blocks;

[0007] Bottom-layer interface module: when receiving data, directly write the data received by the network card driver into the pre-allocated empty memory block to form a data packet; when sending data, write the data into the network card driver, and return the memory block to the object pool module after sending.

[0008] Batch processing module: merges multiple data packets into one batch for processing. The allocation of memory blocks of these data packets is managed by the object pool module;

[0009] Data transmission module: enables data packets to be transferred between the layers of the protocol stack in a zero-copy manner, and the use of data packets is allocated by the object pool module.

[0010] Preferably, the batch processing steps of the batch processing module are as follows:

[0011] Step 1: The receiving thread of the network protocol stack is responsible for monitoring and receiving data packets. The received data packets will be temporarily stored in a ring buffer. When the ring buffer reaches the batch processing condition, the receiving thread transfers the batch of data packets to the batch buffer.

[0012] Step 2: The batch buffer is used as a temporary storage structure to collect data packets that have reached the batch processing conditions and wait for them to be delivered to the network protocol stack thread;

[0013] Step 3: When the data packet is passed from the receive buffer to the protocol stack thread, Rust's zero-copy technology is used, that is, the reference or pointer of the data is directly passed without actually copying the data.

[0014] Preferably, the batch size adjustment step is:

[0015] Step 1: Obtain the CPU occupancy Q and measure the total time T from receiving to processing the data packet, and calculate the deviation between the actual monitoring value (Q, T) and the preset target value;

[0016] Step 2: Estimate the cumulative deviation of Q and T, and adjust the batch size appropriately to offset the accumulated deviation;

[0017] Under high load conditions, when the monitored high CPU utilization or high network traffic exceeds the set threshold, the controller will increase the batch size to reduce the frequency of context switching and IPC calls; under low load conditions, when the monitored latency indicators exceed the preset requirements, the controller will reduce the batch size to reduce the waiting time of data packets in the system, thereby reducing latency.

[0018] Preferably, the object pool module is implemented as follows:

[0019] When initializing the object pool, pre-allocate N memory blocks and put these memory blocks into the lock-free queue;

[0020] When a memory block needs to be used, the thread tries to get an allocated empty memory block from the lock-free queue; if there is an available memory block in the lock-free queue, the memory block is directly returned for use; if the lock-free queue is empty, it is necessary to request a new memory block from the operating system and mark the memory block;

[0021] When the life cycle of a data packet ends, the thread puts the data packet back into the lock-free queue instead of releasing the memory directly. The marked memory block requested from the operating system will be released to the operating system.

[0022] Preferably, the steps for implementing dynamic adjustment of the object pool memory block are as follows:

[0023] By using the load monitoring function mentioned in the batch processing module, when the system is detected to be under high load, the strategy of incremental allocation of memory blocks is triggered, and additional memory blocks are added to the object pool by requesting additional memory blocks from the operating system; when the system load is lower than the set low load threshold, the memory block release strategy is triggered, and the system will gradually reduce the number of memory blocks in the object pool;

[0024] A usage status mark is added to each memory block. When the system is under low load, the status information is used to determine which memory blocks can be released. When a memory block needs to be added, the new memory block is added to the lock-free queue. When a memory block is released, the memory block is removed from the object pool and returned to the operating system. Under low load conditions, an inert recycling strategy can be adopted, that is, gradually releasing memory blocks that are no longer used, rather than releasing a large number of memory blocks at one time, and setting minimum and maximum thresholds to ensure that the object pool can be used normally.

[0025] The memory blocks in the object pool are automatically recycled through life cycle management:

[0026] When the life cycle of a memory block ends, the system will automatically put the memory block back to the object pool module or return it to the operating system based on the source of the memory block to prevent memory leaks. The object pool module is used to manage resources and use the life cycle and ownership mechanism to ensure memory safety.

[0027] Preferably, the data transmission process is as follows:

[0028] Each layer of the network protocol stack divides the data packet into two parts, the header and the data, according to the protocol header specified by each protocol. Both parts exist in the form of slice references.

[0029] Each layer verifies the data packet according to the header information in the data, and after the processing is completed, the data part of the data packet is passed to the subsequent protocol layer of the network protocol stack, and the subsequent protocol layer repeats the above process; in this process, since what is passed is the reference of the data packet, only a shallow copy of the pointer is made, and no deep copy of the data is involved;

[0030] Rust's borrow checker automatically checks the references of data packets during compilation. Each data packet has its own boundaries to ensure that there will be no data contention or data out-of-bounds problems when using the data packet, thus avoiding buffer overflows. Each data packet has its own owner, who is responsible for managing the life cycle of the data packet. When the network protocol stack has processed the data and the data packet life cycle ends, the memory of the data packet is automatically released and the data packet is returned to the object pool or released back to the operating system.

[0031] For data that needs to be shared, it is managed with smart pointers of shared ownership. The data will be released only when there is no place to use the data anymore. When a data packet is requested, an owner is assigned to it, and each pointer clearly points to an allocated resource.

[0032] Preferably, the bottom layer interface module implements the steps as follows:

[0033] When receiving data, the bottom-level interface module will obtain a free memory block from the object pool, and then temporarily store the memory block in a cache, obtain a raw pointer pointing to the memory block, and write the data received by the network card driver directly into the memory block pointed to by the raw pointer. After writing is completed, the bottom-level interface module will pass the data to the upper layer of the network protocol stack through a callback function for data processing;

[0034] When sending data, the bottom interface module receives the data packet from the upper layer of the network protocol stack, writes the data into the network card driver, and returns the data packet to the object pool after sending;

[0035] The network protocol stack is responsible for managing the life cycle of raw pointers to ensure that the network card driver does not exceed the MTU limit when writing data.

[0036] Preferably, a user interface module is included:

[0037] Design basic interface: provide methods for connecting, sending, receiving and closing connections; establish connection through connect method, process data exchange through send and receive methods, and close connection through close method;

[0038] Through Rust's trait mechanism, a general plug-in interface is defined. Users can customize processing logic and data processing methods by implementing this trait. Each plug-in can independently implement specific processing logic and is decoupled from the core logic of the system.

[0039] Design a plug-in manager to manage the registration, execution and life cycle of plug-ins; the plug-in manager has a storage function that can store multiple plug-ins and call the corresponding methods of these plug-ins at the appropriate time;

[0040] Through generic design of network programming interface, it can support different types of data structures and processing logic;

[0041] Ensuring data is always secure during processing in the network programming interface through ownership and lifecycle management of the plugin manager;

[0042] The plugin manager is responsible for the registration, execution, and lifecycle management of plugins. Users can customize processing logic and data processing methods according to specific needs, and develop their own plugins by implementing specific traits. Each plugin interacts through a unified interface, and users can freely combine and replace different plugins to meet different business needs.

[0043] Preferably, the data transmission process is as follows:

[0044] After receiving data from the network card driver, the receiving thread puts the data into the batch buffer through the underlying interface module. The batch module then manages the data. The memory space allocation for these data comes from the object pool module, and a data packet is directly obtained from the object pool module.

[0045] The bottom interface module decides when to pass data to the protocol stack processing thread based on preset conditions, and communicates between threads through message queues;

[0046] When receiving data from the underlying interface module, the protocol stack processing thread first passes through the data link layer to determine the legitimacy of the data packet and MAC address. After a successful determination, it decides which layer of protocol to hand it over to based on the protocol type, and then passes it to the transport layer protocol. The transport layer will also perform a header check on the data it receives, and perform related operations based on the characteristics of the protocol. Finally, it parses the application layer data and hands the data over to the application thread for user use.

[0047] When the user needs to send data, the user interface module obtains a data packet from the object pool, puts the user data into the data packet, and then passes the data packet to the application layer. The application layer adds the protocol header and passes it downward, passing through the transport layer, network layer and data link layer. Finally, the underlying interface module hands the data to the network card driver to send it out. The data sent may not necessarily pass through the application layer, but may also be sent through protocols such as ARP or ICMP. These data sending processes will also obtain a data packet from the object pool, put the data into the data packet and send it out through the underlying interface module. After the data is sent out, the data packet is returned to the object pool.

[0048] During the data transmission process, the data transmission module is used to enable data packets to be transmitted between the layers of the protocol stack in a zero-copy manner.

[0049] A device, comprising porting a network protocol stack implemented in rust language to a seL4 microkernel and a hardware development board to realize data interaction with an operating system;

[0050] Configure the virtual memory and device management of the seL4 microkernel to ensure that the network protocol stack and network card driver can correctly access the required memory areas and device registers;

[0051] Configure network card related parameters in the seL4 microkernel configuration file to ensure that the microkernel can identify and correctly configure the network card device when it starts, and map the network card address to the user space for the protocol stack to read and write data.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention avoids unnecessary deep copy operations by deeply understanding and applying the ownership system and shallow copy mechanism, realizes efficient transmission of data packets between various layers of the protocol stack, significantly reduces CPU and memory overhead, and improves the efficiency of data transmission and the overall performance of the system. The batch processing and refined state management technology are used to reduce redundant operations and state switching during the protocol stack processing, further improving the response speed and throughput of the system. The object pool mechanism of the lock-free queue is introduced, which reduces the cost of object creation and destruction by pre-allocating and reusing memory resources, effectively reduces lock contention during concurrent processing, and improves the throughput of the system in high-concurrency scenarios.

[0054] Through RAII (resource acquisition is initialization) and lifecycle management mechanism, automatic resource management is achieved, memory safety is ensured, and common memory leaks, buffer overflows and multiple releases in traditional C language implementations are avoided. Rust's powerful type system and ownership model significantly improve the memory safety of the code. Common problems such as memory leaks and null pointer references are avoided, effectively reducing potential security vulnerabilities and errors. The use of dynamic data structures to automatically adjust the buffer size avoids memory waste, overflow and data truncation problems caused by fixed-size buffers, ensuring that the system can efficiently and safely process data of dynamic size. In the implementation of the network protocol stack, most common programming errors can be captured and fixed at the compilation stage, improving the stability and reliability of the system. For the protection of network security and data integrity, it can effectively prevent network attacks and data tampering, and ensure the security of communication data. The use of concurrency models and thread safety features makes it easier to implement data transmission and processing between layers of the network protocol stack in a multi-threaded environment, improving the concurrent processing capability of the system. In response to the multi-threaded processing requirements of the network protocol stack, an efficient concurrency control mechanism is designed to avoid the performance overhead caused by simple lock abuse. It also avoids the competition conditions and deadlock problems in the traditional C language implementation, and improves the stability and security of the system in a high-concurrency environment.

[0055] Through Rust's traits and generics, a plug-in extension interface is designed, allowing users to flexibly expand the protocol stack functions according to specific needs without modifying the core code. This design improves the modularity of the system, reduces code coupling, and enhances the maintainability and reusability of the system. By modularizing the various functions of the protocol stack, users can combine and replace different modules and plug-ins according to actual needs, thereby achieving high configurability. This not only simplifies development and maintenance work, but also can quickly adapt to the needs of different application scenarios.

[0056] The cross-platform portability of the network protocol stack can be deployed and run on different operating systems and hardware platforms. This makes the technology not only suitable for traditional server and PC environments, but also can be effectively applied to embedded devices and microkernel operating systems.

[0057] It is not just a simple Rust rewrite of the traditional C language protocol stack, but through deep customization and optimization, it surpasses the performance and security of traditional implementations, solves the difficult memory management problems in C language, and realizes a more efficient, safer and more flexible network protocol stack. This invention has achieved significant improvements in performance optimization, memory safety, flexibility and security by cleverly combining the advanced features of Rust language. Compared with the traditional C language implementation, this system not only solves the common memory management and concurrency control problems in C language, but also provides a highly modular and extensible interface design, making the network protocol stack more adaptable and competitive. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a data flow diagram of the network protocol stack of the present invention.

[0059] Figure 2 It is a structural diagram of a specific example of the present invention.

[0060] Figure 3 This is a schematic diagram of the object pool. DETAILED DESCRIPTION

[0061] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0062] The network protocol stack from bottom to top is the data link layer, network layer, transport layer, and application layer.

[0063] A method for building an embedded network protocol stack based on Rust, including a batch processing module, an object pool module, a data transmission module, a bottom-level interface module, and a user interface module;

[0064] The underlying interface module and batch processing module are located in the data link layer; the batch processing module is only used when the underlying interface module receives data;

[0065] Batch processing module: merges multiple data packets into one batch for processing. The memory area of ​​these data packets is managed by the object pool module. The memory space is the memory block.

[0066] Bottom-layer interface module: when receiving data, directly write the data received by the network card driver into the pre-allocated memory block to form a data packet; when sending data, write the data into the network card driver, and return the memory block to the object pool module after sending.

[0067] The object pool module is involved in the data reception of the underlying interface module and the data transmission of each layer of the network protocol stack. The specific contents are as follows:

[0068] Object pool module: pre-allocates N memory blocks, uses Rust's lock-free queue to manage the object pool, and uses ownership and lifecycle mechanisms to automatically allocate and release memory blocks;

[0069] Data transmission module: enables data packets to be transferred between the layers of the protocol stack in a zero-copy manner, and the use of data packets is allocated by the object pool module.

[0070] The application layer transfers data with the user interface module.

[0071] Batch processing module, specifically implemented as follows:

[0072] The network protocol stack implements a batch processing mechanism. In order to improve the performance of the network protocol stack, multiple data packets are merged into one batch for processing, reducing context switching and IPC call overhead, thereby improving system performance.

[0073] The receiving thread of the network protocol stack is responsible for listening and receiving data packets. The received data packets are temporarily stored in a receiving buffer, which uses a circular buffer, which allows efficient enqueue and dequeue operations and avoids the dynamic memory allocation overhead of traditional queues. The receiving thread continuously reads data packets from the network interface and stores them in a temporary batch buffer.

[0074] The receiving thread listens to the network interface and continuously receives data. The received data is stored in a circular buffer. The receiving thread continuously puts the received data packets into the circular buffer until the circular buffer reaches the preset batch size or meets the time interval condition. When the circular buffer reaches the batch processing condition, the receiving thread transfers the batch of data packets to the batch buffer. As a temporary storage structure, the batch buffer is responsible for collecting data that reaches the batch processing condition and waits to be passed to the network protocol stack thread.

[0075] The ring buffer is used to store received data, and a fixed-size memory area is used to avoid the overhead of dynamic memory allocation. The design of the ring buffer makes memory utilization more efficient and can maintain stable performance in high-concurrency scenarios. When passing data from the receive buffer to the protocol stack thread, Rust's zero-copy technology is used, that is, directly passing a reference or pointer to the data without actually copying the data. This can reduce the overhead of memory copying and ensure the efficiency of the data transmission path. The memory space where the data is stored is obtained from the object pool and returned to the object pool when the data needs to be released. This process uses Rust's Drop trait to automatically return the data packet to the object pool after the life cycle ends. The object pool we designed is combined with the life cycle mechanism of the Rust language to ensure the memory safety of the data packet.

[0076] The system monitors key performance indicators such as network traffic, CPU utilization, and latency in real time. Use these monitoring data to evaluate the current system load status. Monitor the number of packets and total data volume per unit time through the network interface to evaluate network pressure. Regularly sample CPU occupancy and pay attention to the high and low fluctuations in CPU occupancy to reflect the pressure on the system processing capacity. Measure the total time from the receipt of a packet to its completion of processing, including network latency, queue waiting time, and processing time. Calculate the deviation between the actual monitoring value (such as CPU utilization, latency) and the preset target value. Adjust the batch size directly according to the size of the deviation. The larger the deviation, the greater the adjustment. Accumulate the deviation value to prevent the long-term accumulation of deviations from causing continuous deviations in the system state, and adjust the batch size appropriately to offset the accumulated deviations. Under high load conditions, when the monitored high CPU utilization or high network traffic exceeds the set threshold, the controller will increase the batch size to reduce the frequency of context switching and IPC calls, thereby improving throughput. Under low load conditions, when the monitored latency indicators exceed the preset requirements, the controller will reduce the batch size to reduce the waiting time of the packet in the system, thereby reducing latency. The system periodically evaluates batch size and adjusts accordingly to ensure that the system continues to optimize performance under varying network loads.

[0077] Our solution has better security and efficiency compared with the traditional network protocol stack implemented in C language or the network protocol stack simply implemented in Rust.

[0078] In our implementation, the receiving buffer uses a circular buffer combined with Rust's zero-copy technology to avoid the dynamic memory allocation overhead of traditional queues while ensuring the efficiency and security of data processing. Rust's ownership mechanism eliminates the need for additional copy operations when transferring data, directly optimizing the data transmission path in the batch processing stage and improving processing performance.

[0079] Compared to traditional C or Rust implementations, which usually use fixed batch sizes based on preset thresholds or simple time intervals. Although this approach works in specific scenarios, it may not achieve optimal results under dynamic load environments. Our network protocol stack implements an intelligent adjustment mechanism that combines load monitoring and adaptive algorithms to dynamically adjust the batch size based on factors such as current network traffic, CPU utilization, and latency requirements. This can automatically increase the batch size to increase throughput under high load conditions and reduce the batch size to reduce latency under low load conditions, thereby achieving optimal performance in all situations.

[0080] The object pool module is specifically implemented as follows:

[0081] This system implements an object pool based on Rust lock-free queues to improve the performance of the network protocol stack and reduce the overhead of object creation and destruction. Lock-free queues can provide more efficient concurrent access in a multi-threaded environment, thereby improving the system's throughput and response speed and ensuring multi-threaded safety.

[0082] When initializing the object pool, a group of memory blocks for data packets are pre-allocated, and the pointers to these memory blocks are placed in a lock-free queue. This can avoid the overhead caused by frequent allocation and release of memory at runtime. When a data packet is needed, the thread tries to get a pointer to an already allocated memory block from the lock-free queue. If there is an available memory block in the queue, the pointer to the memory block is directly returned for use. If the queue is empty, it is necessary to request a new memory block from the operating system and mark the memory block. When the life cycle of the data packet ends, the thread puts the pointer to the memory block back into the lock-free queue instead of directly releasing the memory. The marked memory requested from the operating system will be released to the operating system. In this way, the memory block can be reused, reducing frequent memory allocation and release operations, and improving the memory utilization efficiency of the system.

[0083] Implement dynamic adjustment of object pool memory blocks. Use the load monitoring function mentioned in the batch processing module. When the system is detected to be under high load, trigger the strategy of incremental allocation of memory blocks. Add these new memory blocks to the object pool by requesting additional memory blocks from the operating system. When the system load is lower than the set low load threshold, trigger the memory block release strategy. The system will gradually reduce the number of memory blocks in the object pool. Add a usage status mark to each memory block. When the system is under low load, use this status information to determine which memory blocks can be released. When additional memory blocks are needed, add new memory blocks to the lock-free queue; when releasing memory blocks, remove the memory blocks from the object pool and return them to the operating system. Under low load conditions, a lazy recycling strategy can be adopted, that is, gradually releasing memory blocks that are no longer used, rather than releasing a large number of memory blocks at one time, to avoid causing unnecessary system fluctuations, and set minimum and maximum thresholds to ensure that the object pool can be used normally.

[0084] The memory blocks in the object pool are automatically recycled through life cycle management. When the life cycle of a memory block ends, the system will automatically put the memory block back into the object pool or return it to the operating system based on the source of the memory block to prevent memory leaks and ensure that the system will not crash due to memory exhaustion. Manage resources through the object pool and use the life cycle and ownership mechanism to ensure memory safety.

[0085] The object pool implementation of C language in a multi-threaded environment usually relies on a lock mechanism to ensure thread safety. This method is prone to lock contention and frequent context switching under high concurrency, which ultimately affects the throughput and response speed of the system. This situation is particularly prominent when processing a large number of short-lifetime data packets. The system may not be able to fully utilize the advantages of multi-core CPUs, and performance is limited. In addition, C language requires manual management of memory allocation and release, which increases the complexity of programming and is prone to problems such as memory leaks or dangling pointers. Especially when implementing complex lifecycle management, the risk of programming errors increases significantly, affecting the stability and security of the system.

[0086] Although Rust's ownership system and borrowing checks can ensure memory safety, simply using the concurrent data structures of the standard library does not automatically improve performance. In particular, in a high-concurrency environment, if the pooling mechanism is not optimized, performance bottlenecks and resource waste may still occur. Although Rust supports lock-free programming, to implement an efficient lock-free queue object pool, issues such as data contention and deadlock need to be considered in depth in the design. A simple Rust rewrite may ignore these complexities, resulting in unsatisfactory actual performance. In addition, the implementation of a static object pool is not flexible enough. If the preset memory of the object pool is too large, it will cause too much memory of the operating system to be occupied, resulting in memory waste; if the preset memory is too small, memory will be frequently requested from the operating system, and the effect of the object pool will not be achieved. If the strategy of requesting memory from the operating system to join the object pool when the object pool is insufficient is adopted, without a dynamic increase and decrease strategy, the object pool will become larger and larger, and eventually run out of memory.

[0087] In our system, we use Rust's lock-free queue to implement the object pool, optimize the performance overhead caused by the use of locks, and use the ownership and life cycle mechanism to automatically allocate and release memory to ensure the memory safety of the system. On this basis, we design a dynamic adjustment strategy for the object pool memory block to dynamically adjust the size of the object pool according to the system load, thereby improving the flexibility and efficiency of the object pool.

[0088] The underlying interface modules are as follows:

[0089] When receiving data, the underlying interface module will obtain a free memory block (data packet) from the object pool, and then temporarily store the memory block in a cache, obtain a raw pointer to the memory block, and write the data received by the network card driver directly into the memory pointed to by the raw pointer. In this way, the data is received, and no data copying is involved in this process, thereby reducing memory overhead. In this process, the ownership of this memory always belongs to the network protocol stack, so the network protocol stack manages the allocation and release of memory. The network card driver will not be involved in the allocation and release of the memory pointed to by the raw pointer, ensuring memory security. After writing is completed, the underlying interface module will pass the data to the upper layer of the network protocol stack through a callback function for data processing. Implement boundary checking to ensure that the data does not exceed the memory boundary during the data writing process of the network card driver to avoid buffer overflow errors.

[0090] The process of sending data is the opposite of the above process. After the bottom interface module receives the data packet from the upper layer of the network protocol stack, it writes the data into the network card driver and returns the data packet to the object pool after sending.

[0091] In C language, the interaction between the network card driver and the network protocol stack often relies on the transmission of raw pointers. Although this method can directly operate the memory area and reduce the overhead of data copying, it also brings significant risks. Since C language does not have a memory safety mechanism, developers must be very careful to manage the life cycle of pointers to prevent problems such as dangling pointers, memory leaks, and memory out-of-bounds. This manual management is not only complicated, but also prone to serious system crashes or security vulnerabilities due to details.

[0092] Simply using Rust to implement the network driver and the network protocol stack inevitably requires copy operations, resulting in performance loss. In addition, if the memory allocation strategy and raw pointer management are not fully optimized, the rewritten Rust implementation may not perform as expected under high load and cannot achieve truly efficient data transmission.

[0093] The underlying interface module can directly write the data received by the network card driver into the pre-allocated memory area, avoiding unnecessary memory copying during data transmission, significantly reducing system overhead and latency. The network protocol stack is responsible for managing the life cycle of raw pointers, avoiding common memory leaks and dangling pointer problems in C language. Our implementation uses a dynamic safety check mechanism to ensure that the network card driver does not exceed the MTU limit when writing data.

[0094] The data transmission modules are as follows:

[0095] Design a universal data packet structure so that data can be efficiently transferred between the layers of the protocol stack without copying the data, including a pointer to the actual data buffer. Each layer of the network protocol stack divides the data packet into two parts, the header and the data, according to the protocol header specified by each protocol. Both parts exist in the form of slice references, so there is no data copying involved. Each layer verifies the data packet according to the header information in the data, and after processing, passes the data part of the data packet to the subsequent protocol layer of the network protocol stack (the network protocol stack passes the received data to the upper layer, and passes the sent data to the next layer), and the subsequent protocol layer repeats the above process. In this process, since the data packet reference is transmitted, there is only a shallow copy of the pointer, and no deep copy of the data is involved, avoiding the performance overhead caused by data copying.

[0096] Rust's borrow checker automatically checks the references of data packets at compile time. Each data packet has its own boundaries to ensure that there will be no data contention or data out-of-bounds problems when using the data packet, avoiding buffer overflows. Each data packet has its own owner, who is responsible for managing the life cycle of the data packet. When the network protocol stack has processed the data and the data packet life cycle ends, the memory of the data packet is automatically released and the data packet is returned to the object pool or released back to the operating system to avoid memory leaks.

[0097] For data that needs to be shared, smart pointers with shared ownership are used to manage it. The data is released only when there is no place to use it, avoiding the problem of repeated release caused by multiple releases. When a data packet is requested, it is assigned an owner, and each pointer clearly points to an allocated resource to avoid the generation of dangling pointers.

[0098] The user interface modules are as follows:

[0099] Design the basic interface: provide methods for connecting, sending, receiving, and closing the connection. Use the connect method to establish a connection, use the send and receive methods to handle data exchange, and use the close method to close the connection.

[0100] Traditional C language protocol stacks usually use fixed-size buffers to manage data transmission and reception. Although this method is simple, it is prone to buffer overflow problems and requires developers to manually manage the size of the buffer. In addition, fixed-size buffers are inefficient when processing data of dynamic size, which can easily cause memory waste or data truncation. In traditional C language protocol stack designs, network programming interfaces are often fixed and lack flexibility. To add new functions or custom processing logic to an existing protocol stack, the source code needs to be modified directly, resulting in high code coupling and poor scalability. In addition, this method is not conducive to code reuse and maintenance.

[0101] Simply rewriting the C language protocol stack with Rust can improve the memory security of the code, but if the interface is simply designed in the C language, the network programming interface still lacks flexibility. Users may be restricted by the interface design when using it, and cannot flexibly expand and adjust it according to specific business needs. If a simple Rust rewrite does not fully utilize Rust's advanced features such as traits and generics, it may lead to a lack of modularity and abstraction in the interface design, making it difficult to achieve flexible plug-in expansion.

[0102] The network programming interface we designed uses a dynamic data structure to dynamically manage the data buffer. Unlike the traditional fixed-size buffer, it can automatically adjust the size according to the actual needs of the data, avoiding buffer overflow and memory waste. This design not only improves memory utilization, but also effectively handles data of dynamic size, improving system stability and performance.

[0103] Through Rust's trait mechanism, a general plugin interface is defined. Users can customize processing logic and data processing methods by implementing this trait. Each plugin can independently implement specific processing logic and is decoupled from the core logic of the system. Design a plugin manager to manage the registration, execution, and life cycle of plugins. The plugin manager has a storage function that can store multiple plugins and call the corresponding methods of these plugins at the right time.

[0104] By designing the network programming interface generically, it can support different types of data structures and processing logic. This makes the interface more flexible, and users can pass different types of data according to actual needs. The functions of the network programming interface are modularized so that each module can be developed and tested independently. This design can improve the maintainability and scalability of the code.

[0105] Through the ownership and lifecycle management of the plugin manager, it is ensured that in the network programming interface, data is always safe during processing, avoiding problems such as memory leaks, dangling pointers and buffer overflows.

[0106] Users can customize processing logic and data processing methods according to specific needs, and develop their own plug-ins by implementing specific traits. The plug-in manager is responsible for the registration, execution, and lifecycle management of plug-ins. This design makes the system highly flexible, and users can easily expand the functionality of the network protocol stack without modifying the core code. By using Rust traits and generics, our network programming interface achieves a highly modular design. Each plug-in interacts through a unified interface to avoid code coupling. Users can freely combine and replace different plug-ins to meet different business needs without affecting the overall stability and performance of the system.

[0107] Our design not only focuses on performance improvement, but also on memory safety. Through dynamic buffer management and plug-in expansion interface, we ensure efficient data transmission and processing while improving system flexibility.

[0108] Compared with the simple Rust rewrite solution, our design avoids the performance bottleneck in the traditional protocol stack and achieves a balance between high performance and high security.

[0109] Avoid copy overhead during network protocol stack processing:

[0110] Although Rust's ownership system can ensure memory safety, if the C language protocol stack logic is simply rewritten in Rust, the advantages of the ownership system cannot be fully utilized. Especially in the multi-layer structure of the network protocol stack, if it is not properly designed, the transfer of ownership may become complicated and difficult to manage, which may lead to unnecessary overhead.

[0111] A simple rewrite of the Rust implementation may ignore the reasonable use of shallow copies and ownership, resulting in unnecessary deep copy operations during packet delivery, and failing to achieve performance optimization. In addition, Rust's ownership mechanism requires strict lifecycle management. If these mechanisms are not fully considered in a complex network protocol stack, performance degradation and programming errors may occur.

[0112] The various layers of the network protocol stack need to share ownership of resources. In order to avoid deep copies, the ownership of resources that need to share ownership should be managed through smart pointers. Use Rust's Arc (atomic reference counting) or Rc (non-atomic reference counting) to share the ownership of data packets, so that the same data packet can be shared in multiple processing stages without copying the data.

[0113] When passing data packets between protocol layers, only ownership or references are passed, not copies of the data. This allows all layers to share the same data without having to copy the data multiple times. Using a pre-allocated object pool to manage data buffers, combined with Rust's zero-copy technology, data does not need to be actually copied when it is passed between protocol stacks, only a shallow copy is made. To ensure that data packets are properly managed during the transmission process, Rust's RAII (resource acquisition is initialization) mechanism can be used to automatically manage the life cycle of resources. When the data packet is processed, ownership will be automatically reclaimed, and the memory will be released or returned to the object pool.

[0114] During the protocol stack processing, memory mapping technology (such as mmap) is used to directly access data in hardware devices or file systems without loading the data into user space memory. This can greatly reduce the overhead of data copying.

[0115] Our system design ensures that after the network protocol stack is processed, the memory of the data packet can be automatically returned to the object pool or released back to the operating system according to the allocation method of the data packet. By combining Rust's RAII mechanism and lifecycle management, our implementation can accurately control the use and recycling of resources between various layers of protocols, avoiding the common memory leak problem in C language and surpassing the complexity of lifecycle management that may be brought about by simple Rust rewrite.

[0116] We deeply customized Rust's memory management so that data packets can be efficiently transferred between layers of the protocol stack in a zero-copy manner. We use Rust's advanced features, such as reference counting, smart pointers, and memory mapping, to ensure the security and efficiency of data during transmission, avoiding performance bottlenecks and memory waste that may occur in simple Rust implementations.

[0117] Through refined state management and context transfer optimization, our design reduces redundant checks when packets are passed between layers of the protocol stack. Compared with traditional C language implementation, we can share state information and processing results more efficiently, further reducing processing overhead and improving the overall response speed and throughput of the network protocol stack.

[0118] This implementation example describes how to deploy an embedded network protocol stack based on the Rust language to the seL4 microkernel and the hardware YT507H development board to form an embedded system that can perform network communication. The specific steps include the implementation of hardware drivers, the configuration of hardware interfaces, the porting and system integration of the network protocol stack, and the design and implementation of user interfaces and operating system interfaces.

[0119] 1. Hardware driver implementation

[0120] Analyze the hardware manual and network card specifications Obtain and study the hardware manual of the YT507H development board, especially the specifications and register layout of the network card. Determine the basic functions that the network card driver needs to implement, such as initialization, sending and receiving data packets.

[0121] Write a driver program to initialize the relevant registers of the network card to ensure that the network card is in working state. Implement the function of sending data packets to ensure that the data can be correctly written into the send buffer of the network card and trigger the transmission. Implement the function of receiving data packets to ensure that the data can be correctly read from the receive buffer.

[0122] 2. Hardware interface configuration

[0123] Configure the network card's hardware address, memory size and other parameters in the seL4 microkernel configuration file to ensure that the microkernel can identify and correctly configure the network card device when it starts, and map the network card address to the user space for the protocol stack to read and write data.

[0124] Configure the hardware timer, initialize the hardware timer, configure the interrupt function, map the hardware address to the user space, and implement clock control.

[0125] 3. Network protocol stack transplantation

[0126] The network protocol stack implemented in rust language is ported to the seL4 microkernel to implement the interface with the operating system so as to use the services provided by the operating system, including thread creation, message queues, semaphores, mutex locks, clocks, etc.

[0127] Integrate the network protocol stack with the hardware driver to ensure that each layer can correctly call and transfer data. Implement the interface between the physical layer and the network card driver so that data packets can be sent and received through the hardware interface.

[0128] 4. System Integration

[0129] Configure the virtual memory and device management of the seL4 microkernel to ensure that the network protocol stack and network card driver can correctly access the required memory areas and device registers.

[0130] Create a message queue for data transfer between threads. Use a mutex to ensure the concurrency safety of the message queue. Use a semaphore mechanism to block and wake up threads and control the thread's access to and from the message queue. If a thread wants to get data from the message queue and the message queue is empty, the thread is blocked. When other threads put data into the message queue, a notification is sent to wake up the blocked thread to get the data.

[0131] Create a receiving thread, which is responsible for receiving data from the network card and passing the data to the network protocol stack thread through the message queue.

[0132] Create a protocol stack processing thread. After receiving data from the message queue, analyze and process the network protocol headers of each layer of the data, and perform corresponding processing according to different data packets.

[0133] Create a sending thread, which is responsible for sending network protocol stack data.

[0134] Create an application thread to process application layer data.

[0135] 5. Network protocol stack process

[0136] After the stack receiving thread receives the data from the network card driver, it puts the data into the batch buffer through the underlying interface module. The batch processing module then manages the data and decides when to pass the data to the protocol stack processing thread based on preset conditions. Here, the message queue is used for inter-thread communication. The space allocation of these data comes from the object pool module. A data packet is directly obtained from the object pool module to avoid the overhead caused by memory allocation.

[0137] The protocol stack processing thread receives data and first passes through the data link layer to determine the legitimacy of the data packet and MAC address. After the judgment is successful, it determines which layer of protocol to hand over to based on the protocol type, ARP protocol or IP protocol. If it is the ARP protocol, the ARP request is returned based on the ARP response, or the ARP response from other hosts is processed. If it is the IP protocol, the data packet is transmitted to the IP layer, and the IP layer will perform header verification, including data length, protocol type, IP address, checksum, etc. After the IP layer verifies, it will pass it to the transport layer protocol based on the protocol type of the next layer, usually TCP or UDP protocol. The transport layer will also perform header verification when receiving data, and perform a series of operations based on the characteristics of the protocol, such as TCP needs to slide the window, modify the sequence number, the reply number, and congestion control, flow control and other operations. Finally, the application layer data is parsed and handed over to the application thread for user use. The above is the process of receiving data, and the process of sending data is the opposite of the above. In the process of data transmission, the data transmission module is used to ensure the efficiency and security of data transmission between protocols at each layer.

[0138] Users can use user interface modules to perform network programming, write application code, customize processing logic and data processing methods, and develop their own plug-ins by implementing specific traits according to their needs.

[0139] 6. System testing and verification

[0140] Deploy the completed network protocol stack and driver to the YT507H development board, and perform functional and performance tests.

[0141] Verify the correctness of the functions of the network protocol stack, design and run functional test cases, and verify the basic functions of the network protocol stack on the YT507H development board, including the sending and receiving of data packets, the processing of each layer of protocols, etc. Ensure that the network protocol stack can correctly handle various types of data packets and network events.

[0142] Performance testing, by calling the application layer interface to implement iperf, test the sending and receiving performance of the network protocol stack. Based on the test results, optimize the network protocol stack to improve the overall performance.

[0143] Through the above detailed steps, the embedded network protocol stack based on the memory-safe language was successfully ported to the seL4 microkernel and the YT507H development board, realizing an embedded system capable of network communication, and verifying the correctness and feasibility of this system. This system not only utilizes the memory safety and concurrency safety features of the Rust language, but also combines the modularity and high reliability of the microkernel operating system, meeting the needs for efficient and stable network communication in embedded systems.

[0144] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an embedded network protocol stack based on Rust, characterized in that: It includes batch processing module, object pool module, data transmission module and bottom layer interface module; Object pool module: pre-allocates N memory blocks, uses Rust's lock-free queue to manage the object pool, and uses ownership and lifecycle mechanisms to automatically allocate and release memory blocks; Bottom-layer interface module: when receiving data, directly write the data received by the network card driver into the pre-allocated empty memory block to form a data packet; when sending data, write the data into the network card driver, and return the memory block to the object pool module after sending. Batch processing module: merges multiple data packets into one batch for processing. The allocation of memory blocks of these data packets is managed by the object pool module; Data transmission module: enables data packets to be transferred between the layers of the protocol stack in a zero-copy manner, and the use of data packets is allocated by the object pool module.

2. The Rust-based embedded network protocol stack construction method according to claim 1 is characterized in that: Batch processing module The batch processing steps are as follows: Step 1: The receiving thread of the network protocol stack is responsible for monitoring and receiving data packets. The received data packets will be temporarily stored in a ring buffer. When the ring buffer reaches the batch processing condition, the receiving thread transfers the batch of data packets to the batch buffer. Step 2: The batch buffer is used as a temporary storage structure to collect data packets that have reached the batch processing conditions and wait for them to be delivered to the network protocol stack thread; Step 3: When the data packet is passed from the receive buffer to the protocol stack thread, Rust's zero-copy technology is used, that is, the reference or pointer of the data is directly passed without actually copying the data.

3. The Rust-based embedded network protocol stack construction method according to claim 2 is characterized in that: The steps for batch size adjustment are: Step 1: Obtain the CPU occupancy Q and measure the total time T from receiving to processing the data packet, and calculate the deviation between the actual monitoring value (Q, T) and the preset target value; Step 2: Estimate the cumulative deviation of Q and T, and adjust the batch size appropriately to offset the accumulated deviation; Under high load conditions, when the monitored high CPU utilization or high network traffic exceeds the set threshold, the controller will increase the batch size to reduce the frequency of context switching and IPC calls; under low load conditions, when the monitored latency indicators exceed the preset requirements, the controller will reduce the batch size to reduce the waiting time of data packets in the system, thereby reducing latency.

4. The Rust-based embedded network protocol stack construction method according to claim 1 is characterized in that: The object pool module is implemented as follows: When initializing the object pool, pre-allocate N memory blocks and put these memory blocks into the lock-free queue; When a memory block needs to be used, the thread tries to get an allocated empty memory block from the lock-free queue; if there is an available memory block in the lock-free queue, the memory block is directly returned for use; if the lock-free queue is empty, it is necessary to request a new memory block from the operating system and mark the memory block; When the life cycle of a data packet ends, the thread puts the data packet back into the lock-free queue instead of releasing the memory directly. The marked memory block requested from the operating system will be released to the operating system.

5. The Rust-based embedded network protocol stack construction method according to claim 4 is characterized in that: The steps to implement dynamic adjustment of object pool memory blocks are as follows: By using the load monitoring function mentioned in the batch processing module, when the system is detected to be under high load, the strategy of incremental allocation of memory blocks is triggered, and additional memory blocks are added to the object pool by requesting additional memory blocks from the operating system; when the system load is lower than the set low load threshold, the memory block release strategy is triggered, and the system will gradually reduce the number of memory blocks in the object pool; A usage status mark is added to each memory block. When the system is under low load, the status information is used to determine which memory blocks can be released. When a memory block needs to be added, the new memory block is added to the lock-free queue. When a memory block is released, the memory block is removed from the object pool and returned to the operating system. Under low load conditions, an inert recycling strategy can be adopted, that is, gradually releasing memory blocks that are no longer used, rather than releasing a large number of memory blocks at one time, and setting minimum and maximum thresholds to ensure that the object pool can be used normally. The memory blocks in the object pool are automatically recycled through life cycle management: When the life cycle of a memory block ends, the system will automatically put the memory block back to the object pool module or return it to the operating system based on the source of the memory block to prevent memory leaks. The object pool module is used to manage resources and use the life cycle and ownership mechanism to ensure memory safety.

6. The Rust-based embedded network protocol stack construction method according to claim 1 is characterized in that: The data transmission process is as follows: Each layer of the network protocol stack divides the data packet into two parts, the header and the data, according to the protocol header specified by each protocol. Both parts exist in the form of slice references. Each layer verifies the data packet according to the header information in the data, and after the processing is completed, the data part of the data packet is passed to the subsequent protocol layer of the network protocol stack, and the subsequent protocol layer repeats the above process; in this process, since what is passed is the reference of the data packet, only a shallow copy of the pointer is made, and no deep copy of the data is involved; Rust's borrow checker automatically checks the references of data packets during compilation. Each data packet has its own boundaries to ensure that there will be no data contention or data out-of-bounds problems when using the data packet. Each data packet has its own owner, who is responsible for managing the life cycle of the data packet. When the network protocol stack has processed the data and the data packet life cycle ends, the memory of the data packet is automatically released and the memory block is returned to the object pool or released back to the operating system. For data that needs to be shared, it is managed with smart pointers of shared ownership. The data will be released only when there is no place to use the data anymore. When a data packet is requested, an owner is assigned to it, and each pointer clearly points to an allocated resource.

7. The Rust-based embedded network protocol stack construction method according to claim 1 is characterized in that: The implementation steps of the bottom interface module are: When receiving data, the bottom-level interface module will obtain a free memory block from the object pool, and then temporarily store the memory block in a cache, obtain a raw pointer pointing to the memory block, and write the data received by the network card driver directly into the memory block pointed to by the raw pointer. After writing is completed, the bottom-level interface module will pass the data to the upper layer of the network protocol stack through a callback function for data processing; When sending data, the bottom interface module receives the data packet from the upper layer of the network protocol stack, writes the data into the network card driver, and returns the data packet to the object pool after sending; The network protocol stack is responsible for managing the life cycle of raw pointers to ensure that the network card driver does not exceed the MTU limit when writing data.

8. The Rust-based embedded network protocol stack construction method according to claim 1, characterized in that: Includes user interface modules: Design basic interface: provide methods for connecting, sending, receiving and closing connections; establish connection through connect method, process data exchange through send and receive methods, and close connection through close method; Through Rust's trait mechanism, a general plug-in interface is defined. Users can customize processing logic and data processing methods by implementing this trait. Each plug-in can independently implement specific processing logic and is decoupled from the core logic of the system. Design a plugin manager to manage plugin registration, execution, and lifecycle; The plugin manager has a storage function that can store multiple plugins and call the corresponding methods of these plugins at the appropriate time; Through generic design of network programming interface, it can support different types of data structures and processing logic; Ensuring data is always secure during processing in the network programming interface through ownership and lifecycle management of the plugin manager; The plugin manager is responsible for the registration, execution, and lifecycle management of plugins. Users can customize processing logic and data processing methods according to specific needs, and develop their own plugins by implementing specific traits. Each plugin interacts through a unified interface, and users can freely combine and replace different plugins to meet different business needs.

9. The Rust-based embedded network protocol stack construction method according to any one of claims 1-8, characterized in that: The data transmission process is as follows: After receiving data from the network card driver, the receiving thread puts the data into the batch buffer through the underlying interface module. The batch module then manages the data. The memory space allocation for these data comes from the object pool module, and a data packet is directly obtained from the object pool module. The bottom interface module decides when to pass data to the protocol stack processing thread based on preset conditions, and communicates between threads through message queues; When receiving data from the underlying interface module, the protocol stack processing thread first passes through the data link layer to determine the legitimacy of the data packet and MAC address. After a successful determination, it decides which layer of protocol to hand it over to based on the protocol type, and then passes it to the transport layer protocol. The transport layer will also perform a header check on the data it receives, and perform related operations based on the characteristics of the protocol. Finally, it parses the application layer data and hands the data over to the application thread for user use. When the user needs to send data, the user interface module obtains a data packet from the object pool, puts the user data into the data packet, and then transmits the data packet to the application layer. The application layer adds the protocol header and passes it downward, passing through the transport layer, network layer and data link layer. Finally, the underlying interface module hands the data to the network card driver to send it out. Data transmission does not necessarily pass through the application layer. It may also be sent through protocols such as ARP or ICMP. The process of sending data will also obtain a data packet from the object pool, put the data into the data packet and send it out through the underlying interface module. After the data is sent out, the data packet is returned to the object pool; During the data transmission process, the data transmission module is used to enable data packets to be transmitted between the layers of the protocol stack in a zero-copy manner.

10. A device for implementing the Rust-based embedded network protocol stack construction method according to any one of claims 1 to 8, characterized in that: Port the network protocol stack implemented in rust language to the seL4 microkernel and hardware development board to realize data interaction with the operating system; Configure the virtual memory and device management of the seL4 microkernel to ensure that the network protocol stack and network card driver can correctly access the required memory areas and device registers; Configure network card related parameters in the seL4 microkernel configuration file to ensure that the microkernel can identify and correctly configure the network card device when it starts, and map the network card address to the user space for the protocol stack to read and write data.

Images