RDMA network card data processing method, system and device based on local extension cache and storage medium

Abstract

This application discloses a method, system, device, and storage medium for RDMA network interface card (NIC) data processing based on a local extended cache. The method includes: determining a target connection identifier in response to a packet processing event; directly reading all metadata associated with the target connection from the local extended cache based on the target connection identifier; obtaining description information of the data to be processed from host memory based on connection attribute metadata, and performing permission verification and address translation on the virtual address based on memory protection metadata and address mapping metadata; and performing data transfer via DMA operation based on the translated physical address. The local extended cache stores all metadata entries required for RDMA protocol processing. By storing all metadata entries in the local extended cache and eliminating on-chip SRAM caching, this application can support the large capacity requirement of 16M QP while meeting the 800G line-rate processing requirement, reducing metadata access latency and chip resource overhead.

Description

Technical Field

[0001] This application relates to the field of RDMA network technology, and in particular to a data processing method, system, RDMA network card device and storage medium based on local extended cache for RDMA network cards. It can be widely used in scenarios with extreme requirements for RDMA concurrent connection number and throughput performance, such as AI training clusters, distributed storage, and high-performance computing. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology, as a high-bandwidth, low-latency network communication protocol, has been widely used in scenarios with extreme requirements for concurrent connections and throughput performance, such as Artificial Intelligence (AI) training clusters, distributed storage, and High Performance Computing (HPC). RDMA network interface cards (NICs) (i.e., smart NICs supporting RDMA functionality) are the core hardware carriers of this technology, and their performance directly determines the communication efficiency of the entire system.

[0003] Currently, mainstream RDMA network card architectures in the industry generally adopt a hierarchical storage scheme of "on-chip static random access memory (SRAM) cache + host double data rate (DDR) memory storage," represented by NVIDIA ConnectX series and Broadcom Thor series. The core working principle of this architecture is as follows: frequently accessed hot data in metadata table entries such as Queue Pair Context (QPC), Scatter-Gather Element (SGE), Memory Protection Table (MPT), Congestion Control Context (PCC), and Page Block List (PBL) are stored in the high-speed SRAM cache inside the chip to provide nanosecond-level access latency; at the same time, infrequently accessed cold data is stored in the host's DDR memory, which is accessed by the RDMA network card through the Peripheral Component Interconnect Express (PCIe) bus. To mask the access latency of up to approximately 1200ns for the PCIe bus and host DDR memory, the hardware needs to support a large number of outstanding concurrent requests (typically hundreds), maintaining line-rate processing capabilities through pipelining and out-of-order execution.

[0004] However, the above-mentioned tiered storage architecture has the following shortcomings in practical applications: First, the capacity of queue pairs (QPs) is severely insufficient to meet the needs of ultra-large-scale AI clusters. On-chip SRAM is extremely expensive in terms of area and power consumption, making it difficult to integrate large-capacity storage. For example, 16MB of on-chip SRAM can only support approximately 64K QPC entries of size 256B. This makes the QP capacity of existing network cards a serious bottleneck for system expansion, directly leading to decreased communication efficiency in large-scale clusters and limited system scalability. This is the most significant drawback of existing technology.

[0005] Secondly, cache misses cause a sharp performance degradation. When the actual number of QPs exceeds the on-chip SRAM cache capacity, the cache hit rate drops significantly, and the RDMA network card needs to frequently access the host DDR memory via the PCIe bus to obtain metadata. The PCIe access latency is as high as 1200ns, about 100 times the on-chip SRAM access latency. To hide this high latency, the hardware needs to maintain a large number of outstanding requests, significantly increasing hardware design complexity and power consumption.

[0006] Third, on-chip SRAM incurs huge resource overhead. The area of ​​an SRAM memory cell is more than 100 times that of a DRAM memory cell, and its power consumption is also much higher than that of DRAM. Increasing the on-chip SRAM capacity to support more QPs will directly lead to an increase in chip area and power consumption, which not only increases manufacturing costs but also brings more severe heat dissipation and reliability problems.

[0007] Fourth, it's difficult to balance high-capacity QP support with 800G line-rate processing. 800G line-rate processing places extremely high demands on metadata access bandwidth. For example, in a 1KB small packet scenario, 100 million packets need to be processed per second, and each packet requires accessing multiple metadata entries, resulting in a total metadata access bandwidth requirement exceeding 800Gb. Under the existing tiered storage architecture, PCIe bus bandwidth itself becomes a bottleneck, preventing the achievement of 800G line-rate in high-capacity QP scenarios.

[0008] In summary, the existing layered storage architecture of RDMA network cards has significant shortcomings in terms of QP capacity, access latency, chip resource overhead, and line-speed processing capability. There is an urgent need for an innovative architecture solution that can simultaneously meet the requirements of large-capacity QP support and 800G line-speed processing. Summary of the Invention

[0009] The technical problem to be solved by this application is to provide a data processing method, system, device and storage medium for RDMA network cards based on local extended cache. By storing all metadata entries in local extended cache, and the read operations for all metadata entries are completed directly through the local extended cache without going through the on-chip cache, it can meet the requirements of ultra-large-scale AI clusters for large capacity QP (supporting 16M QP) while also taking into account 800G line speed processing.

[0010] To address the aforementioned technical problems, as one aspect of this application, a data processing method for an RDMA network interface card (NIC) based on a local extended cache is provided. The RDMA NIC is equipped with a local extended cache, which stores all metadata entries required for RDMA protocol processing. Furthermore, all read operations performed by the RDMA NIC on these metadata entries are directly performed through the local extended cache, without going through the on-chip cache. The method includes: In response to a packet processing event, the target connection identifier is determined; Based on the target connection identifier, all metadata associated with the target connection is read directly from the local extended cache; the all metadata includes at least connection attribute metadata for locating host memory description information, memory protection metadata for authorization verification, and address mapping metadata for address translation. The description information of the data to be processed is obtained from the host memory according to the connection attribute metadata, and the virtual address in the description information is verified for permissions and the address is converted according to the memory protection metadata and the address mapping metadata to obtain the physical address; Based on the physical address, the data to be processed is moved between the host memory and the RDMA network card via DMA operation.

[0011] The local extended cache is an LPDDR6 extended cache, and the RDMA network card integrates a 4-channel LPDDR6 controller. The LPDDR6 extended cache provides 32GB of storage capacity and 1656Gb of total bandwidth.

[0012] The total number of metadata entries includes: Queue Pair Context (QPC) table, Distributed Aggregation (SGE) table, Memory Protection (MPT) table, Congestion Control (PCC) table, and Page Block (PBL) table; the number of entries in each of the QPC, SGE, MPT, PCC, and PBL tables is 16M; the size of each QPC entry is 256B, each SGE entry is 64B, each MPT entry is 64B, each PCC entry is 128B, and each PBL entry is 64B.

[0013] Wherein, the data packet processing event is a sending event, and the method further includes: dynamically adjusting the data packet sending interval and sending rate according to the congestion control metadata in all metadata.

[0014] The data packet processing event is a sending event: The determination of the target connection identifier includes: receiving a doorbell signal sent by the host through the PCIe bus, and parsing the doorbell signal to obtain the queue pair number QPN; The step of obtaining the description information of the data to be processed from the host memory includes: calculating the physical address of the work queue element WQE and reading the WQE based on the send queue base address and send queue consumer pointer in the connection attribute metadata; and reading the corresponding SGE table entry based on the SGE index in the WQE to obtain the data virtual address and local key L_KEY. The permission verification and address translation include: reading the corresponding MPT table entry with the L_KEY as the index to perform permission verification, and reading the PBL table entry according to the PBL base address and the page offset of the virtual address in the MPT to complete the translation from virtual address to physical address; The data transfer process involves reading user data from the converted physical address using a DMA engine, encapsulating it, and then sending it to the network.

[0015] The data packet processing event is a receive event: The determination of the target connection identifier includes: receiving a message from the network and extracting the queue pair number QPN from the message header; The description information of the data to be processed obtained from the host memory includes: calculating the physical address of the receiving WQE and reading the WQE according to the receiving queue base address and receiving queue consumer pointer in the connection attribute metadata; reading the corresponding SGE table entry according to the SGE index in the WQE to obtain the receiving buffer virtual address and local key L_KEY; The permission verification and address translation include: reading the corresponding MPT table entry with the L_KEY as the index to perform write permission verification, and reading the PBL table entry according to the PBL base address and the page offset of the virtual address in the MPT to complete the translation from virtual address to physical address; The data transfer is as follows: the payload data of the message is written to the converted physical address through the DMA engine.

[0016] After the data migration is completed, the process further includes: generating a completion queue element (CQE), writing it into the host memory via the PCIe bus, and triggering an interrupt to notify the host.

[0017] As another aspect of this application, an RDMA network card data processing system based on local extended cache is also provided, comprising: The identifier determination module is used to determine the target connection identifier in response to a packet processing event; The metadata reading module is used to directly read all metadata associated with the target connection from the local extended cache based on the target connection identifier. The local extended cache stores all metadata entries required for RDMA protocol processing. The RDMA network card performs all read operations on all metadata entries directly through the local extended cache, without going through the on-chip cache. The all metadata includes at least connection attribute metadata for locating host memory description information, memory protection metadata for authorization verification, and address mapping metadata for address translation. The description information processing module is used to obtain the description information of the data to be processed from the host memory based on the connection attribute metadata; The address translation module is used to perform permission verification and address translation on the virtual address in the description information according to the memory protection metadata and the address mapping metadata, so as to obtain the physical address; The data transfer module is used to transfer the data to be processed between the host memory and the RDMA network card through DMA operation based on the physical address.

[0018] The local extended cache is an LPDDR6 extended cache, and the RDMA network card integrates a 4-channel LPDDR6 controller. The LPDDR6 extended cache provides 32GB of storage capacity and 1656Gb of total bandwidth.

[0019] The total number of metadata entries includes: Queue Pair Context (QPC) table, Distributed Aggregation (SGE) table, Memory Protection (MPT) table, Congestion Control (PCC) table, and Page Block (PBL) table; the number of entries in each of the QPC, SGE, MPT, PCC, and PBL tables is 16M; the size of each QPC entry is 256B, each SGE entry is 64B, each MPT entry is 64B, each PCC entry is 128B, and each PBL entry is 64B.

[0020] The data packet processing event is a sending event. The system also includes a congestion control module, which is used to dynamically adjust the sending interval and sending rate of data packets based on the congestion control metadata in all metadata.

[0021] As another aspect of this application, an RDMA network card device is also provided, comprising: The local extended cache, as a physical storage medium, is used to store all metadata entries required for RDMA protocol processing; the RDMA network card device performs all read operations on the metadata entries directly through the local extended cache, without going through the on-chip cache. A cache controller, connected to the local extended cache, is used to manage the read and write operations of the local extended cache; An interface module is used to connect the host to an external network. The interface module includes a physical layer interface, a PCIe interface, and a MAC interface. The processing engine includes a send pipeline and a receive pipeline, which are respectively connected to the local extended cache through the physical layer interface and the cache controller, and are used to directly read all the metadata table entries from the local extended cache; the processing engine is used to implement the functions of the system as described above; And a DMA engine, connected to the processing engine, for transferring data between host memory and the local extended cache.

[0022] The local extended cache is an LPDDR6 extended cache, providing 32GB of storage capacity; the cache controller is a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb. The physical layer interface is a DDR PHY interface, which is connected to the cache controller and the processing engine respectively, and is used to realize physical layer data interaction between the processing engine and the local extended cache. The PCIe interface is connected to the processing engine and is used to communicate with the host. The MAC interface is connected to the processing engine and is used to communicate with external networks, supporting 800Gbps line-speed network throughput.

[0023] As another aspect of this application, a computing device is also provided, which includes host memory and the aforementioned RDMA network card device, wherein the host memory is used to provide the RDMA network card device with data for reading or writing.

[0024] As another aspect of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described above.

[0025] Implementing this embodiment has the following beneficial effects: First, this application provides a data processing method, system, device, and storage medium for RDMA network cards based on local extended cache. By storing all metadata entries (QPC, SGE, MPT, PCC, PBL) in a local LPDDR6 extended cache, all metadata entries are read directly through the local extended cache without going through the on-chip cache. In this application, the on-chip SRAM cache can be eliminated to maximize chip area savings. Compared to existing technologies where 16MB SRAM can only support approximately 64K QPC entries, this application achieves a large capacity support of 16M QP, which is 18 to 125 times that of existing mainstream network cards, and can meet the QP requirements of AI training clusters with hundreds of thousands of cards, significantly improving system scalability. Secondly, this application uses directly connected local LPDDR6 storage for metadata, eliminating the need for PCIe bus access and removing on-chip cache from the read path. This reduces metadata access latency from 1200ns in traditional solutions to approximately 200ns, a reduction of about 83%. This low-latency metadata access significantly improves RDMA communication performance, particularly in small message scenarios. Furthermore, this application directly points the read path of all metadata entries to the local extended cache, thereby avoiding reliance on the on-chip SRAM cache. In a preferred embodiment, the on-chip SRAM cache can be eliminated. The area of ​​an SRAM cell is approximately 100 times that of a DRAM, and its power consumption is also much higher. This solution significantly reduces chip area and power consumption, lowers manufacturing costs, and improves heat dissipation and reliability. In addition, this application integrates a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb, which can meet the metadata bandwidth requirements of 800G line-speed processing even in 1KB small packet scenarios. At the same time, low-latency local access eliminates the reliance on a large number of outstanding requests, reduces hardware complexity, simplifies hardware design, and reduces the bandwidth usage of the PCIe bus. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of this application.

[0027] Figure 1 A schematic diagram of the overall architecture of an application environment for a data processing method based on local extended cache in an RDMA network card provided in this application; Figure 2A schematic diagram of the main flow of an embodiment of an RDMA network card data processing method based on local extended cache provided in this application; Figure 3 A schematic diagram of the data flow for sending events provided in this application; Figure 4 A schematic diagram of the data flow for receiving event processing provided in this application; Figure 5 A schematic diagram of an embodiment of an RDMA network card data processing system based on local extended cache provided in this application; Figure 6 This is a schematic diagram of an embodiment of an RDMA network card device provided in this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, a further detailed description of this application will be provided below with reference to the accompanying drawings. It should be noted that the specific embodiments described below are for illustrative purposes only and do not constitute an undue limitation of this application.

[0029] I. System Overall Architecture like Figure 1 The diagram illustrates the overall architecture of an application environment for a data processing method based on local extended cache in an RDMA network card provided in this application. In this embodiment, the system architecture of the application environment mainly includes the following modules: The top-level module of the RDMA processing engine (RDMA_TOP), which includes the transmit pipeline (TX_PIPE) and the receive pipeline (RX_PIPE), is the core processing unit of the entire system. Among them, TX_PIPE is responsible for the entire process of RDMA transmit messages, and RX_PIPE is responsible for the entire process of RDMA receive messages.

[0030] The LPDDR6 controller integrates four independent channels and connects to the local LPDDR6 extended cache to uniformly manage the read and write operations of local LPDDR6 memory.

[0031] The physical layer interface (DDR PHY) connects the LPDDR6 controller to the TX_PIPE / RX_PIPE and is used to implement physical layer signal transmission and data interaction between the processing engine and the local LPDDR6 extended cache.

[0032] The DMA engine is used for high-speed data transfer between host memory, local LPDDR6 memory, and network interface.

[0033] The PCIe interface is used for high-speed communication between the host CPU and host memory.

[0034] The MAC interface is used for physical connection to external networks and supports 800Gbps line-speed network throughput.

[0035] In the above modules, RDMA_TOP is connected to the DDR PHY, PCIe interface, DMA engine and MAC interface respectively, forming a star topology with the processing engine as the core.

[0036] Table entry storage design: This solution stores all RDMA data entries in the local DDR (specifically LPDDR6) extended cache (see...). Figure 1 The upper part of the RDMA network card reads all metadata entries directly through the local extended cache, without going through the on-chip cache. Specifically, the read path for all metadata is as follows: TX_PIPE / RX_PIPE sends a read request to the LPDDR6 controller through the DDR PHY, and the LPDDR6 controller reads the corresponding metadata entry from the local LPDDR6 extended cache and returns it. This read path does not include the on-chip cache.

[0037] In one specific implementation, the on-chip SRAM cache can be eliminated, and all access to metadata entries can be handled entirely by the local LPDDR6 extended cache, thereby maximizing the savings in chip area and power consumption.

[0038] The specific design of each metadata table item is shown in Table 1 below: Table 1. Metadata entries in the local LPDDR6 extended cache

[0039] The QPC table, or queue context, is used to store connection status information and the address location of the send / receive queue in the host memory. The SGE table, or Distributed Aggregation Table, is used to record the address, length, and local key of the data to be transmitted in the host's virtual memory. The MPT table, or memory protection table, is used to verify whether the read / write / atomic operation permissions of the current operation on the target host's memory region are legal. The PCC table, or Congestion Control Context, is used to store flow control and congestion status parameters such as the current sending rate, congestion window, and RTT of a connection. The PBL table, or page table block, is used to perform the translation and mapping from host virtual address to physical address; As shown in Table 1, the total storage requirement is approximately 9GB, which is much smaller than the total capacity of 32GB, leaving ample room for expansion.

[0040] II. Examples of Data Processing Methods like Figure 2The diagram illustrates the main flow of an embodiment of a data processing method based on local extended cache in an RDMA network card provided in this application, along with... Figure 3 and Figure 4 As shown. In this embodiment, the method employs... Figure 1 The method, implemented using the system architecture involved, includes the following steps: Step S1: In response to the packet processing event, determine the target connection identifier.

[0041] Step S2: Based on the target connection identifier, directly read all metadata associated with the target connection from the local extended cache. All metadata includes at least connection attribute metadata (i.e., the QPC table) for locating host memory description information, memory protection metadata (i.e., the MPT table) for authorization verification, and address mapping metadata (i.e., the PBL table) for address translation.

[0042] Step S3: Obtain the description information of the data to be processed from the host memory according to the connection attribute metadata, and perform permission verification and address conversion on the virtual address in the description information according to the memory protection metadata and the address mapping metadata to obtain the physical address.

[0043] Step S4: Based on the physical address, perform data transfer of the data to be processed between the host memory and the RDMA network card via DMA operation.

[0044] The following combination Figure 3 and Figure 4 Taking sending and receiving events as examples, the steps involved in the above method will be explained in detail.

[0045] (a) Example of sending event handling When the data packet processing event is a send event, combined with Figure 3 The data flow diagram shown in this embodiment illustrates the following specific implementations of the method: Step S1 - Send: Determine the target connection identifier The host CPU runs the RDMA driver, writing the WQE (Work Queue Element) of the task to be sent into the host DDR's ROCE_SQ circular queue according to the RoCE v2 standard format. Simultaneously, it writes the corresponding SGE (Distributed Aggregate Table) entry for that task into a contiguous local LPDDR6 memory region. The driver updates the SQ producer pointer (SQ_PI), completing the host-side task submission. (Corresponding to...) Figure 3 Data flow in the middle ①) The host sends a DB (doorbell) signal to the smart network card via the PCIe Gen5 x16 bus, notifying the chip that a new WQE (Wireless QE) is pending. The doorbell signal first passes through the network card's PCIe interface, which parses the QPN (Queue Pair Number) and SQ (Signal Queue Count) producer pointer increment from the doorbell signal and forwards the task to the TX_PIPE transmit pipeline within the RDMA_TOP module. The TX_PIPE pipeline uses the parsed QPN as the target connection identifier. (Corresponding to...) Figure 3 Data flow in the middle ②) Step S2 - Send: Read all metadata The TX_PIPE pipeline uses the QPN as an index, sending read requests to the local LPDDR6 via the DDR PHY, and reading the complete context table entry of the corresponding QP from the local LPDDR6 QPC (Queue Pair Context) memory. Hardware automatically verifies the QP status, transfer type validity, and access permissions. (Corresponding to...) Figure 3 Data flow in the middle ③) In addition, TX_PIPE uses the QPN as an index to send a read request to the local LPDDR6 via the DDR PHY, retrieving the congestion status entry for the corresponding QP from the local LPDDR6's PCC (Congestion Control Context) memory. The PCC entry contains the current transmission rate, congestion window size, ECN tag count, RTT measurement, and DCQCN algorithm parameters. The hardware dynamically adjusts the packet interval and transmission rate based on the congestion status. (Corresponding to...) Figure 3 Data flow in (⑦) Step S3 - Send: Obtain description information and address translation TX_PIPE calculates the physical address of the WQE in the host DDR based on the SQ base address and the current SQ consumer pointer (SQ_CI) in the QPC table entry, and sends a DMA read request to the host DDR via the PCIe interface to read the complete WQE table entry (standard size 64B). Hardware parses the WQE opcode (SEND / WRITE / READ / ATOMIC), total data length, number of SGEs, MR key, and other core parameters. (Corresponding to...) Figure 3 Data flow in (④) Subsequently, TX_PIPE sends a bulk DMA read request to the local LPDDR6 based on the SGE offset and quantity in the WQE, reading SGE entries from the local LPDDR6. Each SGE entry contains a data virtual address (VA), data length (LEN), and a local key (L_KEY). The hardware verifies whether the total SGE length matches the length specified in the WQE and checks whether the number of SGEs exceeds the hardware maximum limit (16). (Corresponding to...) Figure 3 Data flow in (⑤) TX_PIPE uses L_KEY in SGE as an index to send a read request to the local LPDDR6 via the DDR PHY, reading the corresponding memory region's protection table entry from the local LPDDR6's MPT (Memory Protection Table) storage area. The MPT entry contains information such as the memory region's start address, length, access permissions (read / write / atomic), page size, and PBL base address. Hardware checks whether the SGE virtual address is within the address range specified by the MPT, verifying the legality of read / write / atomic operation permissions. (Corresponding to...) Figure 3 Data flow in the middle (⑥) TX_PIPE sends a read request to the local LPDDR6 via DDRPHY based on the PBL (Page Table Block) base address and the page offset of the SGE virtual address in the MPT entry, and reads the physical page number mapping entry of the corresponding virtual page from the local LPDDR6 PBL memory area. The hardware completes the translation from virtual address (VA) to physical address (PA), and automatically reads multiple consecutive PBL entries for cross-page data. (Corresponding to...) Figure 3 Data flow in (⑧) Step S4 - Send: Data Transfer TX_PIPE sends the converted physical address and data length to the DMA engine within RDMA_TOP. The DMA engine sends a DMA read request to the host DDR via the PCIe interface to read the raw data to be sent by the user. After the data returns via the PCIe bus, it is directly written to the transmit data buffer inside TX_PIPE. The hardware performs CRC32 verification on the read data to ensure data integrity. (Corresponding to...) Figure 3 Data flow in (9) Subsequently, TX_PIPE automatically encapsulates the Ethernet header, IP header, UDP header, BTH header, DETH header, and ICRC checksum according to the QPC and WQE parameters. The encapsulated complete message is then sent to the 800G MAC layer. After PCS encoding and SerDes serialization, it is transmitted to the network via an 8×112Gbps physical link. (Corresponding to...) Figure 3 Data flow in 10a loop After the message is successfully sent to the MAC layer, the TX_PIPE hardware generates a CQE (Completion Queue Element), containing information such as operation status, number of bytes transmitted, and completion sequence number. The CQE is written to the CQ (Completion Queue) in the host DDR via the PCIe interface, updating the CQ producer pointer and optionally triggering a host interrupt to notify the software of task completion. (Corresponding to...) Figure 3 Data flow in 10b loops

[0046] (II) Example of receiving event handling When the data packet processing event is a receive event, combined with Figure 4The data flow diagram shown in this embodiment illustrates the following specific implementations of the method: Step S1 - Receive: Determine the target connection identifier After being received via the 800G MAC interface and physical layer, packets from the network are forwarded to the RX_PIPE receive pipeline within the RDMA_TOP module. The RX_PIPE first enters the DPE (Data Path Engine) for preliminary processing, where the hardware extracts the Ethernet header, IP header, UDP header, and BTH header of the packet, separating the RDMA control information and payload data. (Corresponding to...) Figure 4 Data flow in the middle ①) The RPE (Receive Path Engine) within RX_PIPE performs deep parsing of the packet header, performs field validity checks and ICRC verification, and extracts the QPN (Queue Pair Number) from the BTH header as the target connection identifier. (Corresponding to...) Figure 4 Data flow ② First half: RPE preprocessing) Step S2 - Receive: Read all metadata The RPE within RX_PIPE is indexed by QPN. It sends a read request to the local LPDDR6 via the DDR PHY, retrieving the complete context table entry for the corresponding QP from the local LPDDR6's QPC (Queue Pair Context) memory. (Corresponding to...) Figure 4 Data flow ② (second half: QPC context reading) Step S3 - Receiving: Obtain description information and address translation RX_PIPE calculates the physical address of the receive WQE in the host DDR based on the RQ (Receive Queue) base address in the QPC entry and the current RQ consumer pointer (RQ_CI). It then sends a DMA read request to the host DDR via the PCIe interface to read the complete receive WQE entry. Hardware parses the WQE opcode, distinguishing between local operations (Read RSP / Atomic RSP) and remote operations (Write / Send / Atomic). (Corresponding to...) Figure 4 Data flow in the middle ③) The RX_PIPE function uses the SGE index in the receive WQE as an index to send a read request to the local LPDDR6, reading the corresponding SGE entry from the local LPDDR6 SGE memory area. Each SGE entry contains the receive buffer virtual address (VA), buffer length (LEN), and local key (L_KEY). The hardware verifies whether the total SGE length can accommodate the received payload data. Figure 4 Data flow in (④) RX_PIPE uses L_KEY in SGE as an index to send a read request to the local LPDDR6 via the DDR PHY, reading the corresponding memory region's protection table entry from the local LPDDR6's MPT (Memory Protection Table) storage area. The MPT entry contains information such as the memory region's start address, length, access permissions (read / write / atomic), page size, and PBL base address. Hardware checks whether the SGE virtual address is within the address range specified by the MPT and verifies the write operation permissions. (Corresponding to...) Figure 4 Data flow in (⑤) RX_PIPE sends a read request to the local LPDDR6 memory via DDRPHY based on the PBL (Page Table Block) base address and the page offset of the SGE virtual address in the MPT entry. It then reads the physical page number mapping entry for the corresponding virtual page from the local LPDDR6 PBL memory area. The hardware completes the translation from virtual address (VA) to physical address (PA). For cross-page payload data, it automatically reads multiple consecutive PBL entries to generate the final host memory physical write address. (Corresponding to...) Figure 4 Data flow in the middle (⑥) Step S4 - Receiving: Data Transfer RX_PIPE sends the converted physical address, data length, and payload data to the DMA engine within RDMA_TOP. Regardless of whether it's a local operation (Read RSP / Atomic RSP) or a remote operation (Write / Send / Atomic), the DMA engine sends a DMA write request to the host DDR via the PCIe interface, writing the payload data temporarily stored in RX_PIPE into the corresponding receive buffer of the host DDR. The hardware performs a CRC32 check on the written data to ensure data integrity. (Corresponding to...) Figure 4 Data flow in (⑦) After the DMA write operation is complete, the RX_PIPE hardware generates a CQE (Completion Queue Element), containing information such as operation status, number of bytes received, completion sequence number, and error code. The CQE is written to the CQ (Completion Queue) in the host DDR via the PCIe interface, updating the CQ producer pointer. An interrupt merging mechanism is triggered according to the configuration, sending an interrupt notification to the host at an appropriate time. The hardware automatically updates the RQ consumer pointer (RQ_CI) to prepare for processing the next message. (Corresponding to...) Figure 4 Data flow in (⑧)

[0047] III. Examples of Data Processing Systems like Figure 5 The diagram illustrates a structural schematic of an embodiment of a data processing system based on a local extended cache in an RDMA network card provided in this application. In this embodiment, system 1 includes: The identifier determination module 10 is used to determine the target connection identifier in response to a packet processing event.

[0048] The metadata reading module 11 is used to directly read all metadata associated with the target connection from the local extended cache based on the target connection identifier. The local extended cache stores all metadata entries required for RDMA protocol processing. The RDMA network card performs all read operations on all metadata entries directly through the local extended cache, without going through the on-chip cache. The all metadata includes at least connection attribute metadata for locating host memory description information, memory protection metadata for authorization verification, and address mapping metadata for address translation.

[0049] The description information processing module 12 is used to obtain the description information of the data to be processed from the host memory according to the connection attribute metadata.

[0050] Address translation module 13 is used to perform permission verification and address translation on the virtual address in the description information according to the memory protection metadata and the address mapping metadata to obtain the physical address.

[0051] The data transfer module 14 is used to transfer the data to be processed between the host memory and the RDMA network card through DMA operation according to the physical address.

[0052] In a specific example, the local extended cache is an LPDDR6 extended cache, the RDMA network card integrates a 4-channel LPDDR6 controller, and the LPDDR6 extended cache provides 32GB of storage capacity and 1656Gb of total bandwidth.

[0053] In this specific example, all metadata entries include: the QPC table, SGE table, MPT table, PCC table, and PBL table. Each of the QPC, SGE, MPT, PCC, and PBL tables has 16MB of entries. The size of each QPC entry is 256 bytes, each SGE entry is 64 bytes, each MPT entry is 64 bytes, each PCC entry is 128 bytes, and each PBL entry is 64 bytes.

[0054] In a specific example, when the data packet processing event is a sending event, the system further includes a congestion control module, which is used to dynamically adjust the sending interval and sending rate of data packets based on the congestion control metadata in all metadata.

[0055] The specific functional implementation of each module corresponds to the steps in the aforementioned method embodiments. For details, please refer to and combine with the aforementioned... Figures 1 to 4 The description will not be repeated here.

[0056] IV. Examples of RDMA Network Card Devices like Figure 6 The diagram shows a structural schematic of an embodiment of an RDMA network card device provided in this application. In this embodiment, the RDMA network card device 2 includes: The local extended cache 20, serving as a physical storage medium, stores all metadata entries required for RDMA protocol processing. All read operations of the RDMA network interface card (NIC) on these metadata entries are performed directly through the local extended cache, without going through the on-chip cache. In a specific example, the local extended cache is an LPDDR6 extended cache, providing 32GB of storage capacity.

[0057] Cache controller 21, connected to the local extended cache, is used to manage the read and write operations of the local extended cache. In a specific example, the cache controller is a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb.

[0058] Processing engine 22 includes a transmit pipeline (TX_PIPE) and a receive pipeline (RX_PIPE). The transmit and receive pipelines are connected to the local extended cache via the physical layer interface and the cache controller, respectively, for directly reading all metadata entries from the local extended cache. The processing engine implements the functions of the aforementioned data processing system, i.e., it is implemented by the hardware logic or firmware within processing engine 22. Figure 5 The functions of the identifier determination module 10, metadata reading module 11, description information processing module 12, address conversion module 13, and data transfer module 14 described herein.

[0059] DMA engine 23, connected to the processing engine 22, is used for data transfer between host memory and the local extended cache 20.

[0060] Interface module 24 is used to connect the host to an external network. Interface module 24 includes a physical layer interface, a PCIe interface and a MAC interface.

[0061] In some specific examples, the processing engine 22 and the DMA engine 23 can also be implemented in a single component, such as... Figure 1 The RDMA_TOP module shown.

[0062] In a specific example, the local extended cache 20 is an LPDDR6 extended cache, providing 32GB of storage capacity; the cache controller 21 is a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb. The physical layer interface is a DDR PHY interface, which is connected to the cache controller 21 and the processing engine 22 respectively, and is used to realize physical layer data interaction between the processing engine 22 and the local extended cache 20. The PCIe interface is connected to the processing engine 22 and is used to communicate with the host. The MAC interface is connected to the processing engine 22 and is used to communicate with external networks, supporting 800Gbps line-speed network throughput.

[0063] The specific functional implementation of each module mentioned above corresponds to each step / module in the aforementioned method and system embodiments. For details, please refer to and combine with the aforementioned... Figures 1 to 5 The description will not be repeated here.

[0064] Accordingly, as another aspect of this application, a computing device is also provided. In this embodiment, the computing device includes host memory, and as described above. Figure 6 The described RDMA network interface card (NIC) device 2. The host memory is used to provide data for reading or writing to the RDMA NIC device 2. For more details, please refer to and combine with the foregoing description. Figure 6 The description of that will not be repeated here.

[0065] Accordingly, as another aspect of this application, a computer-readable storage medium is also provided, storing a computer program, wherein the program, when executed by a processor, implements as follows: Figures 2 to 4 The steps of the described method. For more details, please refer to and combine with the foregoing descriptions. Figures 2 to 4 The description of that will not be repeated here.

[0066] V. Performance Analysis

[0067] (I) Bandwidth Analysis In an embodiment of the present invention, a single one-way RDMA message requires reading: QPC (256B) + SGE (64B) + MPT (64B) + PCC (128B) + PBL (64B) = 576B. A two-way message requires a total of: 576B × 2 = 1.12KB = 9Kb.

[0068] Under 800G bidirectional traffic, the bandwidth requirements for different packet sizes are as follows: 4KB packet: 800Gb / (4KB×8) = 25M pps, 25M × 9Kb = 225Gb 2KB packet: 800Gb / (2KB×8) = 50M pps, 50M × 9Kb = 450Gb 1KB packet: 800Gb / (1KB×8) = 100M pps, 100M × 9Kb = 900Gb Therefore, in the embodiments of this application, using 4-channel LPDDR6 can provide: 14.4G × 48 × 0.6 × 4 = 1656Gb bandwidth. Even in the most demanding scenario of 1KB small packets (bandwidth requirement of 900Gb), the embodiments of this application can still meet the 800G line speed requirement and leave sufficient margin.

[0069] (II) Delay Analysis In the embodiments of this application, the local LPDDR6 read latency is approximately 200ns, while the latency for traditional PCIe access to host DDR is approximately 1200ns. Implementing the method of this application can reduce metadata access latency by 83%, significantly reducing the number of outstanding requests required to hide latency.

[0070] VI. Overview of Core Innovations In summary, the core innovations of this application can be summarized as follows: Full-entry local LPDDR6 memory architecture: All RDMA data entries (QPC, SGE, MPT, PCC, PBL) are stored in the local LPDDR6 extended cache of the smart network interface card (NIC). All metadata entries are read directly through this local extended cache, bypassing the on-chip cache. At the system level, this read operation is implemented through a metadata read module; at the device level, it is performed via the processing engine's transmit and receive pipelines through the DDR PHY and LPDDR6 controller. In a specific example, the on-chip SRAM cache can be eliminated to maximize chip area and power savings.

[0071] High-capacity QP line-speed processing design: It provides a high bandwidth of 1656Gb and a large capacity of 32GB through 4-channel LPDDR6, realizing 800G line-speed processing capability of 16M QP.

[0072] Low-latency metadata access mechanism: By utilizing directly connected local LPDDR6, the metadata access latency is reduced from 1200ns for traditional PCIe access to host DDR to 200ns, reducing the need for outstanding requests.

[0073] Resource-optimized hardware architecture: By directly pointing the read path of all metadata entries to the local extended cache and bypassing the on-chip cache, the use of on-chip SRAM is avoided, which greatly saves chip area and power consumption, while simplifying hardware design and reducing complexity.

[0074] Reduced number of pending requests and reduced hardware complexity: Low-latency local LPDDR6 access eliminates the need for a large number of pending requests to hide latency, simplifying hardware design, reducing hardware complexity and power consumption, and also reducing bandwidth usage on the PCIe bus.

[0075] Implementing this embodiment has the following beneficial effects: First, this application provides a data processing method, system, device, and storage medium for RDMA network cards based on local extended cache. By storing all metadata entries (QPC, SGE, MPT, PCC, PBL) in a local LPDDR6 extended cache, all metadata entries are read directly through the local extended cache without going through the on-chip cache. In this application, the on-chip SRAM cache can be eliminated to maximize chip area savings. Compared to existing technologies where 16MB SRAM can only support approximately 64K QPC entries, this application achieves a large capacity support of 16M QP, which is 18 to 125 times that of existing mainstream network cards, and can meet the QP requirements of AI training clusters with hundreds of thousands of cards, significantly improving system scalability. Secondly, this application uses directly connected local LPDDR6 storage for metadata, eliminating the need for PCIe bus access and removing on-chip cache from the read path. This reduces metadata access latency from 1200ns in traditional solutions to approximately 200ns, a reduction of about 83%. This low-latency metadata access significantly improves RDMA communication performance, particularly in small message scenarios. Furthermore, this application directly points the read path of all metadata entries to the local extended cache, thereby avoiding reliance on the on-chip SRAM cache. In a preferred embodiment, the on-chip SRAM cache can be eliminated. The area of ​​an SRAM cell is approximately 100 times that of a DRAM, and its power consumption is also much higher. This solution significantly reduces chip area and power consumption, lowers manufacturing costs, and improves heat dissipation and reliability. In addition, this application integrates a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb, which can meet the metadata bandwidth requirements of 800G line-speed processing even in 1KB small packet scenarios. At the same time, low-latency local access eliminates the reliance on a large number of incomplete requests, reduces hardware complexity, simplifies hardware design, and reduces bandwidth consumption on the PCIe bus.

[0076] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method for an RDMA network card based on local extended cache, characterized in that, The RDMA network card is equipped with a local extended cache, which stores all metadata entries required for RDMA protocol processing. All read operations of the RDMA network card for all metadata entries are completed directly through the local extended cache, without going through the on-chip cache. The method includes: In response to a packet processing event, the target connection identifier is determined; Based on the target connection identifier, all metadata associated with the target connection is read directly from the local extended cache; the all metadata includes at least connection attribute metadata for locating host memory description information, memory protection metadata for authorization verification, and address mapping metadata for address translation. The description information of the data to be processed is obtained from the host memory according to the connection attribute metadata, and the virtual address in the description information is verified for permissions and the address is converted according to the memory protection metadata and the address mapping metadata to obtain the physical address; Based on the physical address, the data to be processed is moved between the host memory and the RDMA network card via DMA operation.

2. The method according to claim 1, characterized in that, The local extended cache is an LPDDR6 extended cache, and the RDMA network card integrates a 4-channel LPDDR6 controller. The LPDDR6 extended cache provides 32GB of storage capacity and 1656Gb of total bandwidth.

3. The method according to claim 2, characterized in that, The total metadata entries include: Queue Pair Context (QPC) table, Distributed Aggregate (SGE) table, Memory Protection (MPT) table, Congestion Control (PCC) table, and Page Block (PBL) table; the number of entries in each of the QPC, SGE, MPT, PCC, and PBL tables is 16M; the size of each QPC entry is 256B, each SGE entry is 64B, each MPT entry is 64B, each PCC entry is 128B, and each PBL entry is 64B.

4. The method according to claim 3, characterized in that, The data packet processing event is a sending event, and the method further includes: dynamically adjusting the data packet sending interval and sending rate according to the congestion control metadata in all metadata.

5. The method according to claim 4, characterized in that, The data packet processing event is a send event: The determination of the target connection identifier includes: receiving a doorbell signal sent by the host through the PCIe bus, and parsing the doorbell signal to obtain the queue pair number QPN; The step of obtaining the description information of the data to be processed from the host memory includes: calculating the physical address of the work queue element WQE and reading the WQE based on the send queue base address and send queue consumer pointer in the connection attribute metadata; and reading the corresponding SGE table entry based on the SGE index in the WQE to obtain the data virtual address and local key L_KEY. The permission verification and address translation include: reading the corresponding MPT table entry with the L_KEY as the index to perform permission verification, and reading the PBL table entry according to the PBL base address and the page offset of the virtual address in the MPT to complete the translation from virtual address to physical address; The data transfer process involves reading user data from the converted physical address using a DMA engine, encapsulating it, and then sending it to the network.

6. The method according to claim 4, characterized in that, The data packet processing event is a receive event: The determination of the target connection identifier includes: receiving a message from the network and extracting the queue pair number QPN from the message header; The description information of the data to be processed obtained from the host memory includes: calculating the physical address of the receiving WQE and reading the WQE according to the receiving queue base address and receiving queue consumer pointer in the connection attribute metadata; reading the corresponding SGE table entry according to the SGE index in the WQE to obtain the receiving buffer virtual address and local key L_KEY; The permission verification and address translation include: reading the corresponding MPT table entry with the L_KEY as the index to perform write permission verification, and reading the PBL table entry according to the PBL base address and the page offset of the virtual address in the MPT to complete the translation from virtual address to physical address; The data transfer is as follows: the payload data of the message is written to the converted physical address through the DMA engine.

7. The method according to any one of claims 1 to 6, characterized in that, After the data transfer is completed, the process also includes: generating a completion queue element (CQE), writing it into the host memory via the PCIe bus, and triggering an interrupt to notify the host.

8. A data processing system for an RDMA network card based on a local extended cache, characterized in that, include: The identifier determination module is used to determine the target connection identifier in response to a packet processing event; The metadata reading module is used to directly read all metadata associated with the target connection from the local extended cache according to the target connection identifier; the local extended cache stores all metadata entries required for RDMA protocol processing, and the RDMA network card performs all read operations on all metadata entries directly through the local extended cache without going through the on-chip cache; The total metadata includes at least connection attribute metadata for locating host memory description information, memory protection metadata for authorization verification, and address mapping metadata for address translation. The description information processing module is used to obtain the description information of the data to be processed from the host memory based on the connection attribute metadata; The address translation module is used to perform permission verification and address translation on the virtual address in the description information according to the memory protection metadata and the address mapping metadata, so as to obtain the physical address; The data transfer module is used to transfer the data to be processed between the host memory and the RDMA network card through DMA operation based on the physical address.

9. The system according to claim 8, characterized in that, The local extended cache is an LPDDR6 extended cache, and the RDMA network card integrates a 4-channel LPDDR6 controller. The LPDDR6 extended cache provides 32GB of storage capacity and 1656Gb of total bandwidth.

10. The system according to claim 9, characterized in that, The total metadata entries include: Queue Pair Context (QPC) table, Distributed Aggregate (SGE) table, Memory Protection (MPT) table, Congestion Control (PCC) table, and Page Block (PBL) table; the number of entries in each of the QPC, SGE, MPT, PCC, and PBL tables is 16M; the size of each QPC entry is 256B, each SGE entry is 64B, each MPT entry is 64B, each PCC entry is 128B, and each PBL entry is 64B.

11. The system according to any one of claims 8 to 10, characterized in that, The data packet processing event is a sending event. The system also includes a congestion control module, which is used to dynamically adjust the sending interval and sending rate of data packets based on the congestion control metadata in all metadata.

12. An RDMA network card device, characterized in that, include: Local extended cache, as physical storage medium, is used to store all metadata entries required for RDMA protocol processing; The RDMA network card device performs all metadata table entries read operations directly through the local extended cache, without going through the on-chip cache; A cache controller, connected to the local extended cache, is used to manage the read and write operations of the local extended cache; An interface module is used to connect the host to an external network. The interface module includes a physical layer interface, a PCIe interface, and a MAC interface. The processing engine includes a send pipeline and a receive pipeline, which are respectively connected to the local extended cache through the physical layer interface and the cache controller, and are used to directly read all the metadata table entries from the local extended cache; The processing engine is used to implement the functions of the system as described in any one of claims 8 to 11; And a DMA engine, connected to the processing engine, for transferring data between host memory and the local extended cache.

13. The RDMA network card device according to claim 12, characterized in that, The local extended cache is an LPDDR6 extended cache, providing 32GB of storage capacity; the cache controller is a 4-channel LPDDR6 controller, providing a total bandwidth of 1656Gb. The physical layer interface is a DDR PHY interface, which is connected to the cache controller and the processing engine respectively, and is used to realize physical layer data interaction between the processing engine and the local extended cache. The PCIe interface is connected to the processing engine and is used to communicate with the host. The MAC interface is connected to the processing engine and is used to communicate with external networks, supporting 800Gbps line-speed network throughput.

14. A computing device, characterized in that, Includes host memory and the RDMA network card device as described in claim 12 or 13, wherein the host memory is used to provide the RDMA network card device with data for reading or writing.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images