A method and system for doorbell optimization processing of an intelligent network card sending channel
By introducing a dual-threshold triggering mechanism and hardware timeout management into the smart network interface card, the doorbell operation under the multi-channel architecture was optimized, the bandwidth occupation problem caused by frequent MMIO writing to the doorbell was solved, and the small packet sending performance and overall sending efficiency were improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- THE FIFTH RES INST OF TELECOMM SCI & TECH CO LTD
- Filing Date
- 2026-05-18
- Publication Date
- 2026-07-21
Smart Images
Figure CN122437861A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of smart network card technology, and in particular to a doorbell optimization processing method and system for a smart network card transmission channel. Background Technology
[0002] For PCIe network cards, the transmission process can generally be summarized as follows: First, the kernel calls the transmission function registered by the network card driver; then, the transmission function maps the memory address of the data packet to be transmitted to the direct memory access space and fills the DMA address into the transmission descriptor; subsequently, the driver checks the xmit_more attribute in the sk_buff structure passed by the kernel to determine whether there are more data packets to be transmitted; if it is determined that there are no more data packets, it immediately notifies the network card to transmit by performing a memory-mapped input / output write operation to the network card's PCIe base address register space. This operation is called "writing the doorbell".
[0003] To enhance data processing capabilities in multi-core CPU environments, modern smart network interface cards (NICs) have widely adopted a multi-transmit channel architecture. In this architecture, the number of transmit channels is typically equal to or greater than the number of CPU cores, with each channel corresponding to one CPU core. This aims to utilize CPU resources more effectively, avoid over-concentration of tasks on a single core, and prevent transmit channel congestion and increased transmission latency due to excessive CPU load.
[0004] However, current network interface card (NIC) designs mostly rely solely on the `xmit_more` flag passed by the kernel to determine whether to immediately write to the doorbell, leaving the flow control of data transmission to the kernel. This approach has significant shortcomings: applications need to call a dedicated application programming interface (API) for bulk data transmission to allow the kernel to set this flag during transmission. For many legacy applications that do not use such APIs, or those running on earlier kernel versions that do not support this flag optimization, each data packet transmission triggers a doorbell write operation. The doorbell write operation is essentially an MMIO operation; the CPU needs to directly access the PCIe BAR space to perform the write, a process that, unlike data packet transmission, cannot be independently completed by the NIC hardware and DMA controller to free up the CPU.
[0005] For smart network interface cards (NICs) employing a multi-channel architecture, although each channel operates independently, write operations to the BAR space still require sequential execution by the CPU and cannot be processed in parallel. In large packet scenarios, the doorbell write operation accounts for a small proportion of the total transaction layer packets transmitted through the PCIe channel, and its overhead has a limited impact on performance. However, in small packet scenarios, the number of doorbell operations surges, and the resulting TLP (Transaction Limiting Processing) occupies a significant proportion of the transmission channel, severely crowding out the actual data packet transmission bandwidth. This is one of the main reasons why the throughput performance of small packets is far lower than that of large packets.
[0006] In summary, there is an urgent need for a doorbell optimization method and system for the transmission channel of smart network cards to solve the technical problems of high PCIe bandwidth consumption and poor small packet transmission performance caused by frequent MMIO write doorbell operations under the current multi-channel architecture of smart network cards. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a new optimization technology solution from the CPU network card driver and network card hardware, which can reduce the frequent doorbell writing operations under the multi-channel architecture of smart network cards, and especially improve the performance of small packet transmission.
[0008] To achieve the above objectives, this application proposes a doorbell optimization processing method for a smart network card transmission channel, comprising: In the smart NIC driver layer, the data packets to be sent are written into the descriptor of the corresponding sending channel and temporarily stored in the sending queue; Check the dual threshold trigger conditions of the current transmission channel. The dual threshold trigger conditions include: the current depth of the transmission queue reaches a preset queue depth threshold, and the timeout transmission flag of the hardware timeout timer has been set. When one of the dual threshold triggering conditions is met, the smart network card hardware is notified to immediately send the data packets in the sending queue through the doorbell writing operation; otherwise, the doorbell writing operation is not performed and the process returns directly.
[0009] As a further solution, the management method for the hardware timeout timer specifically includes: The smart network card hardware generates an independent hardware timeout timer for each of its transmission channels; When a data packet is first written to an empty send queue, the driver checks and enables the hardware timeout timer for the corresponding channel. When the hardware timeout timer exceeds a preset time threshold, the smart network card hardware sets a timeout transmission flag via DMA to notify the driver to perform the doorbell write operation.
[0010] As a further solution, the method further includes the step of passively triggering the writing of the doorbell: If a data packet arrives at the receiving channel corresponding to a certain transmitting channel of the smart network card hardware, after processing the NAPI process corresponding to the receiving interrupt, the timeout transmitting flag is actively set for the associated transmitting channel.
[0011] As a further solution, the method further includes the step of dynamically disabling the hardware timeout timer: During the NAPI processing after sending is completed, it is checked whether there are still data packets to be sent in the corresponding sending queue; If no data packet to be sent is found in N consecutive detections, the enable flag of the hardware timeout timer is set to false to turn off the timer, where N is a configurable positive integer greater than or equal to 3.
[0012] As a further solution, the write doorbell operation is a sequential MMIO write operation to the PCIe BAR space of the smart network card hardware.
[0013] On the other hand, the present invention also provides a doorbell optimization processing system for a smart network card transmission channel, comprising: The smart network interface card hardware is used to generate and manage an independent hardware timeout timer for each transmit channel, and to set a timeout transmit flag via DMA after the timer expires. The doorbell optimization driver module executes a doorbell optimization processing method for the smart network card transmission channel as described in any of the above-mentioned methods during its operation.
[0014] Compared with related technologies, the doorbell optimization processing method and system for the intelligent network card transmission channel provided by this invention has the following advantages: 1. This invention is based on a dual-threshold batch sending mechanism, which abandons the traditional method of simply relying on the kernel xmit_more flag and creatively introduces a "dual-threshold" triggering mechanism composed of a hardware timeout timer and a sending queue depth. Only when the number of data packets accumulated in the driver layer reaches a certain number (reaching the queue threshold) or the waiting time exceeds a preset value (hardware timer timeout) will a doorbell write operation be performed, thereby realizing the "batch" notification of data packets and effectively reducing the number of doorbell operations.
[0015] 2. This invention is based on timeout management through hardware and driver collaboration, which transfers timeout control from the pure software level to the hardware level. The hardware generates and manages the timeout timer for each transmission channel and actively notifies the driver through DMA. This hardware collaborative design frees up the CPU and avoids the overhead of software polling or frequent timer checks, making timeout judgment more accurate and efficient.
[0016] 3. This invention is based on passive triggering and dynamic energy saving using receive interrupts. It utilizes receive interrupts to passively trigger the doorbell sending. When the transmission traffic is low, it no longer relies on active timer timeouts, but instead waits for received data packets to arrive and then triggers the sending of a small number of waiting data packets during the process of completing the reception. Simultaneously, a mechanism for determining the number of consecutive idle cycles (N) is introduced to dynamically disable the hardware timeout timer, achieving a balance between energy saving and low latency under low load. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of a doorbell optimization processing method for a smart network card transmission channel according to an embodiment of the present invention; Figure 2 This is a structural diagram of a doorbell optimization processing system for a smart network card transmission channel according to an embodiment of the present invention; Figure 3 A flowchart illustrating the processing of a modern network interface card (NIC) model; Figure 4 This is a flowchart illustrating the business processing of an embodiment of the present invention. The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0021] Example 1 Please see Figure 1 This embodiment provides a doorbell optimization processing method for a smart network card transmission channel, including: In the smart NIC driver layer, the data packets to be sent are written into the descriptor of the corresponding sending channel and temporarily stored in the sending queue; Check the dual threshold trigger conditions of the current transmission channel. The dual threshold trigger conditions include: the current depth of the transmission queue reaches a preset queue depth threshold, and the timeout transmission flag of the hardware timeout timer has been set. When one of the dual threshold triggering conditions is met, the smart network card hardware is notified to immediately send the data packets in the sending queue through the doorbell writing operation; otherwise, the doorbell writing operation is not performed and the process returns directly.
[0022] It should be noted that the core idea of this embodiment is to change the traditional approach of relying solely on the kernel to send a flag to determine whether to write to the doorbell, and instead use two conditions, "queue depth threshold" and "hardware timeout timer", to jointly control the triggering timing of the doorbell.
[0023] When the driver has data packets to send, it does not immediately notify the hardware. Instead, it temporarily stores the data packets in the sending queue. Only when the number of accumulated data packets reaches a preset threshold, or when the hardware timer times out and indicates that the waiting time has expired, does it perform a write doorbell operation to notify the network card hardware to take away and send multiple data packets in the queue in batches.
[0024] This embodiment transforms the single notification operation, which could occur for each data packet, into a centralized notification operation for a batch of data packets. The "queue depth threshold" condition ensures that when the upper-layer application continuously sends data packets, the sending channel can aggregate a sufficient number of data packets for batch sending, thereby reducing the overhead of each doorbell write. The "hardware timeout timer" condition acts as a fallback mechanism, ensuring that data is not excessively delayed due to indefinite waiting when the data packet arrival rate is low and the queue is difficult to fill naturally. This achieves a balance between reducing the number of doorbell operations and ensuring timely delivery.
[0025] This solution significantly reduces the frequency of CPU accessing the PCIe BAR space via MMIO to write doorbell data. Especially in small packet, low-volume scenarios, it effectively alleviates the PCIe channel bandwidth occupation caused by frequent doorbell writing, allowing more channel bandwidth to be used for real data packet transmission, thereby improving small packet throughput performance.
[0026] Furthermore, the management method for the hardware timeout timer specifically includes: The smart network card hardware generates an independent hardware timeout timer for each of its transmission channels; When a data packet is first written to an empty send queue, the driver checks and enables the hardware timeout timer for the corresponding channel. When the hardware timeout timer exceeds a preset time threshold, the smart network card hardware sets a timeout transmission flag via DMA to notify the driver to perform the doorbell write operation.
[0027] Specifically, this embodiment shifts the timeout control from the driver software to the smart network interface card (NIC) hardware level. The hardware independently generates and maintains a timeout timer for each transmission channel. When a transmission channel changes from an idle state to a state with data packets to be transmitted, the driver checks and activates the corresponding hardware timer after writing the first data packet to the queue, rather than starting a software timer. When the preset time threshold expires, the timer times out. This event is not detected by CPU polling, but rather by the NIC hardware actively writing a timeout flag to a specified location in system memory via DMA, thereby notifying the driver.
[0028] This embodiment frees the CPU from tedious tasks such as managing timeouts and status checks. Software-driven timers require the CPU to periodically query or respond to interrupts, and the overhead can be significant when there are many channels. In contrast, hardware timers operate independently on the network card, only transmitting information to the driver via DMA when a substantial conclusion is reached. This hardware-driver collaborative approach ensures the accuracy of timeout judgments and avoids unnecessary CPU overhead in scenarios involving high-frequency, multi-channel batch transmissions.
[0029] Therefore, the entire doorbell optimization solution receives reliable time protection. When packet reception is not frequent and the queue depth threshold is not reached, the driver does not need to worry about excessive delays caused by indefinite waiting; the hardware timeout notification automatically triggers batch transmission. Furthermore, since the timeout flag is set by hardware via DMA, the driver only needs to check the preset flag in the processing flow, resulting in simple and efficient logic that effectively complements the implementation of the dual-threshold doorbell triggering strategy.
[0030] Furthermore, the method further includes the step of passively triggering the writing of the doorbell: If a data packet arrives at the receiving channel corresponding to a certain transmitting channel of the smart network card hardware, after processing the NAPI process corresponding to the receiving interrupt, the timeout transmitting flag is actively set for the associated transmitting channel.
[0031] Specifically, when a data packet arrives at the receive channel corresponding to a transmit channel of the smart network card, the system will proactively set a timeout flag for the associated transmit channel after processing the NAPI procedure for that receive interrupt. This means that even if the hardware timeout timer of the transmit channel has not expired and the queue depth has not reached the preset threshold, the driver can still promptly detect the triggering timing and execute the write-to-the-doorbell operation through the external event of the receive interrupt, sending out the data packets waiting in the queue.
[0032] This passive triggering mechanism is primarily designed for scenarios with low or sparse transmission traffic. When the number of packets to be sent is small, relying solely on the queue depth threshold may cause significant delays due to insufficient packet accumulation; similarly, relying solely on a hardware timeout timer requires waiting for the timer to expire before transmission, which also introduces waiting latency.
[0033] This embodiment cleverly utilizes the data arrival event in the receiving direction. In many network applications, sending and receiving are simultaneous. When a data packet arrives from the other end, the local end may also have data waiting to be transmitted back. At this time, the sending can be triggered at the moment the receiving process is completed, which can effectively reduce the tail delay in unidirectional flow scenarios and avoid the additional timeout overhead caused by the sending end not having new data packets for a long time.
[0034] This design effectively complements the dual-threshold mechanism: during busy transmission periods, the queue depth threshold drives batch transmission; during sparser transmission periods, a hardware timeout timer ensures timely delivery; and in scenarios with extremely sparse transmission but intermittent reception traffic, reception interrupts passively trigger the process, avoiding unnecessary timer waiting. This allows the entire solution to balance low latency and low doorbell overhead across various traffic patterns, further enhancing overall performance in small packet scenarios.
[0035] Furthermore, the method further includes the step of dynamically disabling the hardware timeout timer: During the NAPI processing after sending is completed, it is checked whether there are still data packets to be sent in the corresponding sending queue; If no data packet to be sent is found in N consecutive detections, the enable flag of the hardware timeout timer is set to false to turn off the timer, where N is a configurable positive integer greater than or equal to 3.
[0036] Specifically, in this embodiment, the judgment logic is no longer based on a single detection, but instead introduces a mechanism of "N consecutive idle times". The driver records the number of consecutive times no data packets to be sent are detected. Only when this number reaches a preset configuration value N (N is a positive integer greater than or equal to 3, and its specific value can be optimized according to the actual scenario) is it determined that the sending channel has indeed entered a relatively idle state. At this time, the enable flag of the hardware timeout timer is set to false, and the timer is officially turned off. The method of determining multiple consecutive idle times can effectively avoid the repeated switching on and off jitter of the timer caused by the short interval of sending traffic, ensuring the stability of the mechanism.
[0037] Furthermore, the write doorbell operation is a sequential MMIO write operation to the PCIe BAR space of the smart network card hardware.
[0038] Specifically, in the PCIe network card architecture, the host CPU and the network card hardware communicate through the PCIe bus. The BAR space is a register area mapped by the network card hardware in the host memory address space. The CPU can send control signals to the network card hardware and notify it to start the transmission operation by executing MMIO write instructions to a specific address in this area.
[0039] The characteristic of this MMIO write operation is that it requires direct CPU participation and waiting for the write operation to complete, unlike ordinary data packet transmission which can be completed autonomously by the DMA controller and network card hardware. Each write operation generates a transaction layer packet on the PCIe bus, consuming valuable PCIe channel bandwidth. In traditional solutions, each data packet transmission may trigger an independent MMIO write operation. When the data packet size is small, a large number of small-sized TLPs will fill the PCIe channel, severely crowding out the effective bandwidth for data packet transmission. This is the fundamental reason why the performance of small packets is far lower than that of large packets, as mentioned in the background technology.
[0040] This embodiment employs a series of mechanisms, including dual threshold triggering, hardware timeout management, passive triggering, and dynamic shutdown, to minimize the frequency of MMIO writes to the BAR space. This consolidates multiple independently triggerable doorbell operations into a single operation, significantly reducing the proportion of TLP packets generated by MMIO write operations in the PCIe channel. This allows more channel bandwidth to be allocated to the transmission of actual data packets, ultimately resulting in a substantial improvement in small packet throughput performance.
[0041] Example 2 Please see Figure 2 This embodiment also provides a doorbell optimization processing system for a smart network card transmission channel, including: The smart network interface card hardware is used to generate and manage an independent hardware timeout timer for each transmit channel, and to set a timeout transmit flag via DMA after the timer expires. The doorbell optimization driver module executes a doorbell optimization processing method for the smart network card transmission channel as described in any one of Embodiment 1 during its operation.
[0042] As described in the background technology, the current criterion for whether a network card writes to the doorbell is based on whether the xmit_more attribute is true when the kernel passes sk_buff. The network card model can be abstracted as follows: Figure 3 As shown.
[0043] The specific steps are as follows: Figure 4 As shown, the real-time solution in this embodiment is as follows: The network interface card (NIC) hardware generates a corresponding timeout timer for each transmission channel.
[0044] When a packet is sent, the network card driver reads the timeout timer enable flag in the BAR space. If it is not enabled, it is set to true, and the timeout timer is recorded in the driver through the flag bit.
[0045] After the timeout timer is enabled, if the set time is exceeded, the network card hardware will use DMA to set a timeout sending flag to notify the CPU that it should write doorbell data.
[0046] The send queue is responsible for mapping data packets to the DMA space and writing them to the descriptor. If the send queue reaches the storage threshold (usually 64) or checks the timeout send flag, it writes a doorbell to notify the hardware to send the packet; otherwise, it returns directly without writing a doorbell.
[0047] If the receiving channel corresponding to the network card hardware's transmit channel receives a data packet, after receiving NAPI (NAPI receives the network card hardware's receive interrupt, it cyclically processes all data packets in its own queue, and can process up to NAPI_BUDGET data packets, with NAPI_BUDGET defaulting to 64) has finished processing, the timeout sending flag is set.
[0048] During the NAPI transmission process, if no packet is detected for transmission after the transmission is completed, the timeout timer enable flag is set to false to disable the function. At the same time, the flag bit is changed in the driver.
[0049] The following is a performance comparison before and after adopting the above optimization scheme. The test network card includes two 10G interfaces, and the test results are shown in Tables 1, 2, 3, and 4: Table 1. Single-channel performance comparison (unit: Gb / s) Table 2. Performance Comparison of 8 Channels (Unit: Gb / s) Table 3. Performance Comparison of Dual-Port Single-Channel (Unit: Gb / s) Table 4 Performance Comparison of Dual-Port 8-Channel Ports (Unit: Gb / s) Note: The test data uses kernel sending statistics and Layer 3 IP packet testing, without recording MAC headers and CRC checks.
[0050] As can be seen from the table above, the timeout scheme significantly optimizes small packets, especially in single-channel scenarios. The following is a latency comparison. The test method involves sending a single stream via the instrument, entering through port 1 of the network card under test, and returning to the instrument via port 2. The latency difference before and after using this optimization algorithm is then tested.
[0051] Table 5. Single-port latency statistics (unit: ms) As shown in Table 5, although the optimization scheme introduced some latency, the throughput of small packets was significantly improved. Overall, it is still within an acceptable range.
[0052] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A doorbell optimization processing method for a smart network card transmission channel, characterized in that, include: In the smart NIC driver layer, the data packets to be sent are written into the descriptor of the corresponding sending channel and temporarily stored in the sending queue; Check the dual threshold trigger conditions of the current transmission channel. The dual threshold trigger conditions include: the current depth of the transmission queue reaches a preset queue depth threshold, and the timeout transmission flag of the hardware timeout timer has been set. When one of the dual threshold triggering conditions is met, the smart network card hardware is notified to immediately send the data packets in the sending queue through the doorbell writing operation; otherwise, the doorbell writing operation is not performed and the process returns directly.
2. The doorbell optimization processing method for a smart network card transmission channel according to claim 1, characterized in that, The management method for the hardware timeout timer specifically includes: The smart network card hardware generates an independent hardware timeout timer for each of its transmission channels; When a data packet is first written to an empty send queue, the driver checks and enables the hardware timeout timer for the corresponding channel. When the hardware timeout timer exceeds a preset time threshold, the smart network card hardware sets a timeout transmission flag via DMA to notify the driver to perform the doorbell write operation.
3. The doorbell optimization processing method for a smart network card transmission channel according to claim 2, characterized in that, The method further includes a step of passively triggering the writing of the doorbell: If a data packet arrives at the receiving channel corresponding to a certain transmitting channel of the smart network card hardware, after processing the NAPI process corresponding to the receiving interrupt, the timeout transmitting flag is actively set for the associated transmitting channel.
4. The doorbell optimization processing method for a smart network card transmission channel according to claim 2, characterized in that, The method further includes the step of dynamically disabling the hardware timeout timer: During the NAPI processing after sending is completed, it is checked whether there are still data packets to be sent in the corresponding sending queue; If no data packet to be sent is found in N consecutive detections, the enable flag of the hardware timeout timer is set to false to turn off the timer, where N is a configurable positive integer greater than or equal to 3.
5. The doorbell optimization processing method for a smart network card transmission channel according to claim 1, characterized in that, The write doorbell operation is a sequential MMIO write operation to the PCIe BAR space of the smart network card hardware.
6. A doorbell optimization processing system for a smart network card transmission channel, characterized in that, include: The smart network interface card hardware is used to generate and manage an independent hardware timeout timer for each transmit channel, and to set a timeout transmit flag via DMA after the timer expires. The doorbell optimization driver module executes a doorbell optimization processing method for the smart network card transmission channel as described in any one of claims 1 to 5 during its operation.